Warning: Contains programmer content

Today I had the opportunity to review PHP facilities to track file upload progress for a client project. It turned out to be rather difficult.

Native PHP file upload progress has been available since PHP 5.4 I tried this option first. There are quite a number of gotchas which I found listed in a Stackoverflow Answer:

  • Output buffering needs to be off
  • session.upload_progress.cleanup needs to be off (on by default in Ubuntu 16.04)
  • session.upload_progress.name form field needs to be before file upload field
  • Doesn't work with FastCGI or PHP-FPM.

The last item turned out to be complicating matters even further. According to another comment contribution it doesn't work with apache2 mpm-prefork module either, it needs mpm-worker. This on the other hand is multi-threaded and the PHP module shipping with Ubuntu doesn't appear to be compiled thread-safe.

So native file upload progress turned out to be a dead end.

Next I tried libapache2-upload-progress, which is apparently based on https://github.com/drogus/apache-upload-progress-module.

The tutorial links in this repository are all outdated, so I found it hard to figure out how this would be configured and how to pass the upload ID to the backend process. After figuring it out with the C source code of the module, I'm pretty sure this is the correct way to use this upload with jQuery:

<Virtualhost *:80>
	ServerAdmin webmaster@localhost
	ServerName site.example.com
	
	DocumentRoot /var/www/site

	<location /upload>
	      TrackUploads On
	</location>

        <location /progress>
 		ReportUploads On
	</location>

        LogLevel debug

</Virtualhost>

 

jQuery(document).ready(function() {
    var progressInterval;
    var randomId;
    jQuery('#upload').change(function uploadFiles(){
        var formData = new FormData(jQuery('#uploadform')[0]);
        for(var i=0; i<this.files.length; i++) {
            formData.append('upload[]', this.files[i]);
        }
        progressInterval = setInterval(checkProgress, 2000);
        randomId = Math.random().toString(36).substr(2, 10);
        jQuery.ajax({
        url: '/upload',
        type: 'POST',
        success: function(data, textStatus, jqXHR)
        {
            removeProgress();
        },
        headers: {
            "X-Progress-ID": randomId
        },
        // Form data
        data: formData,
        //Options to tell JQuery not to process data or worry about content-type
        processData: false,
        contentType: false
    });
    jQuery(this).val('');
});

function checkProgress()
{
    jQuery.post('progress?X-Progress-ID='+randomId,
    {
        
    },
    function uploadProgressResult(data) {
        if (parseInt(data.bytes_processed) && parseInt(data.content_length)) {
            var progress = data.bytes_processed / data.content_length * 100;
            jQuery('#progress').show().html(progress.toFixed(2) + '%';
        }
    }
}

 This attempt failed due to the module not detecting the file upload as a POST request, for some reason the request is not correctly identified with this error message: 

mod_upload_progress.c(175): Upload Progress: Non-POST request in trackable location: /upload

So, for my final and successful attempt at this, I tried a PECL extension: php-uploadprogress

PECL extensions have been removed from PHP7, so this wouldn't work out of the box either. However I found a PPA archive, which provides this package for Ubuntu 16.04

This Stackoverflow Answer explains the install steps:

    sudo add-apt-repository ppa:ondrej/php
    sudo apt-get update
    
    sudo apt-get install php-uploadprogress

And voila it works with this final frontend code

<input type="hidden" name="UPLOAD_IDENTIFIER" value="randomphpid" />
<input type="file" name="upload" id="upload" multiple="multiple" />

jQuery(document).ready(function() {
    var progressInterval;
    var randomId;
    jQuery('#upload').change(function uploadFiles(){
        var formData = new FormData(jQuery('#uploadform')[0]);
        for(var i=0; i<this.files.length; i++) {
            formData.append('upload[]', this.files[i]);
        }
        progressInterval = setInterval(checkProgress, 2000);
        randomId = Math.random().toString(36).substr(2, 10);
        jQuery.ajax({
            url: '/upload',
            type: 'POST',
            success: function(data, textStatus, jqXHR)
            {
                removeProgress();
            },
            headers: {
                "X-Progress-ID": randomId
            },
            // Form data
            data: formData,
            //Options to tell JQuery not to process data or worry about content-type
            processData: false,
            contentType: false
        });
        jQuery(this).val('');
    });

    function checkProgress()
    {
        jQuery.get('progress?id=randomphpid',
                    {

                    },
                    function uploadProgressResult(data) {
                        if (parseInt(data.bytes_uploaded) && parseInt(data.bytes_total)) {
                            var progress = data.bytes_uploaded / data.bytes_total * 100;
                            jQuery('#progress').show().html(progress.toFixed(2) + '% (' + data.est_sec + ' sec remaining)');
                            if (progress > 99.9999) {
                                setTimeout(removeProgress, 2000);
                            }
                        }
                    }
                );
    }

We've been promoting secure passwords for a long time, that is: Passwords which are at least 16 characters long and randomly generated.

Recently we had the opportunity to see what we are protecting ourselves against.

A large customers overseas IT security department asked us to protect the login page against brute force attempts to guess the password. This means that after 10 attempts the system would stop any further login attempts from the same IP address.

We knew these attempts are quite common. We didn't know how common. In the first month, we recorded over 47000 attempts to login, thats about one every 55 seconds. Imagine using an easier guessable password. You would hope not to have chosen one of the top 50000 passwords in use!

Are you a Web design, Web Development company or otherwise working in the digital agency space? Over the past 13 years I have identified area after area that has become so specialised, sometimes commonplace or sometimes crucially critical that it becomes futile to deliver these tasks in addition to your core work. These services are either so hard to deliver well or so cheap that they become a cost centre in your business, rather than help you thrive. Typically they are also a major distraction to your core business.  Get rid of these services as quick as you can. 

Of course good on you I'd any of these services are your core business!

See my list here:

Read more: Are you trying to deliver a "futile service"?

SSL ensures that content cannot be listened to between browser and server and as such it helps overcoming a number of security attacks including session fixation attacks. It also increases trust and the subjective quality of a site.

Further, Google has made Full Site SSL a weak ranking signal, so we would be advised to join Google’s SSL everywhere (video link) movement.

We recommend every website owner invest in a SSL certificate and to make the website and all development sites SSL only. The performance effect of this was negligible in 2010 and the overall security will be improved both objectively and subjectively. From an SEO perspective there can be some tools that do not work, but this can be checked beforehand.

SSL certificates have become very cost effective now, which makes SSL security a mass market tool that should be applied to every website.

Special Offer

Read more: Why every website should use SSL security

 Our security model is based on four pillars which interact at various points with each other:

  1. Attackers are automated programs (Botnets)

  2. Defense in depth

  3. Early detection

  4. Good chance of fast recovery

Here is how we are applying this model to a large Joomla website on a Linx/ Apache/ MySQL/ PHP (LAMP) server

Read more: The 4 pillars of security and reliability

I'm locked out of my Xero account. On invoicing day.

What happened? I've been billed through my accountant as a reseller and am moving to a different accountant. My old accountant "removed me from his ledger", ie. his billing and now Xero has deassociated my company data from my login. No malice intended, no open bills.

Both my accountant and myself are making efforts with Xero to get my access back and Xero support is responsive to that.

But I don't like the feeling.

Read more: Taking my own risk management advice

I've been reviewing and subsequently fixing a small business website and realised that the things done wrong on this Joomla website could happen to anyone. But how would a lay person even notice these things are wrong. And even if they've noticed, how could they make their case?

Read more: 5 things a Joomla dev can stuff up on even the simplest website

Many businesses now own enterprise Software as a service(SaaS) applications, even if they never have dreamed of it. Many consulting services that you would have sold on an hourly or day-rate basis in 2007 now need to be delivered with software. And most products that are sold are now at least traced - if not bought -  in a customer portal and possibly also a supplier portal. This means many organisations now own software, but are underprepared to build and maintain it.

One problem that we commonly find is that such web applications are not enterprise ready because they are not using the right URL architecture.

Read more: Why your enterprise web app needs the right URL architecture

Has a web developer ever vanished on you? It can happen in a number of ways:

  • A single developer may get busy on a big project, become employed somewhere or go oversees
  • They have a heart attack or get divorced
  • They simplybecome unresponsive and ignore you

While there are many ways a developer may vanish on you, there is a common theme to this pattern.

Read more: Why web developers vanish

1. Banners/ Slideshows/ Ad placements throughout the site

When Academic Colleges Group had just launched their new Joomla website, covering their 10 domestic and overseas private schools, they immediately faced a problem:

Each school had banners showing key messages as well as call to actions to visit the schools open days. These banners were associated to over 500 Joomla menu items through a common Joomla Banner management extension. While this is the default way of dealing with this problem, it quickly became apparent that making small changes to banners was nearly impossible. First of all it was nearly impossible to find existing entries. Further to this, the backend user interface was not build to handle so many entries.

Read more: 10 challenges with large Joomla websites and how to solve them

Recently we reviewed a web application from a security perspective. The system contained many urls which contained a SHA1 hash like this:

7288EDD0FC3FFCBE93A0CF06E3568E28521687BC

It was always different for every record.

After reviewing the whole application, I came to the conclusion that such hashes are an alarming signs in terms of security for a web application. Why?

Read more: If you see this "hash" in your web application - be afraid

Events like websites being hacked, passwords and user data being stolen have become a daily feature in our news. While we hear mainly about major companies being hacked, there is a multitude of such incidents going on for any size organisation that has a website or public facing web application.

Why do people hack websites?

I'm often asked this question by business owners. Usually it silently assumes that hacking is the activity of a lone, young person in their free time. Nothing could be further from the truth. The vast majority of hack attacks are carried out by highly specialised participants in a "crimical value chain" of hacking, fraud, money laundering and associated activities. These activities are how people in this value chain make a living. Your website is a target for these people, because it can be used in a wide range of activities "promoting their business". If hacked, your customers loose trust, you loose money and everyone looses - apart from the criminals.

Read more: 7 steps for agencies to secure their clients Joomla sites

Extensions are one of the major strengths of the Joomla Content Management System platform. The extension directory - the officially endorsed place to get Joomla extensions has recently been revamped and now sports over 8700 extensions.

However, we still see bespoke Joomla extensions being built day by day, both by us and other Joomla web development companies. Why is that? Surely you would expect to find a component for every use case online? There is a variety of reasons why it still makes sense to create custom/ bespoke components, modules and plugins for Joomla, on top of templates which is a whole other story.

Read more: Why create bespoke Joomla extensions with 8700+ on the JED

Your website or web based software is suppose to run 24/7, all year round. Of course the reality is that there are unexpected issues happening at all sorts of levels, so you have to expect that there will be outages. What are realistic outage times, how high is realistic, how low starts getting bad?

99,99%

99.99% is often shown by hosting providers and it is clearly an unrealistic measure.  Over a 1 year period, this would mean a maximum outage of about 315 seconds, which would be very hard to achieve. You would have to have at least 1 staff member on call for the outage alone, this staff member would need to be able to identify an outage in minimal amount of time and fix it within 5 minutes. This is not realistic.

Read more: What uptime should you expect from your hosting provider

Many business with medium to large Joomla websites use small bespoke components to solve specific problems. Joomla 2.5 has hit its end of live on 31 Dec 2014, so now will be the time to upgrade such websites to the current version, Joomla 3.3.

Some bespoke components may simply not work anymore after the upgrade, even though you may not see any error messages.

The reason is a simple change in Joomla 3.x: Components now need to have a database entry to work, whereas in Joomla versions up and including 2.5, they also work without having any database record set up in the website database. 

To solve this problem, Joomla 3.3 comes with a function to create this database entry, called "Discover". There is a Joomla Help page on how to use this function

Case Studies

  • Automated Quote and Import System - Streat Control

    Streat Control - an importer of electrical engineering goods - has a distributed operation in Auckland, Wellington and Christchurch, serving the local waterworks, refineries, breweries and others with instruments for liquid and gas monitoring and control. With these clients, everything is about accuracy. Their goods are five figures plus per unit and are custom manufactured to order and shipped to New Zealand. As this process is lengthy and costly, mistakes are very expensive.

    streatcontrol-case-study-smallWe provided a custom build and web based importing and tracking system, which also covers some areas of the Customer Relationship Management and the complete process of quoting, ordering and delivery. Through our development process we have replaced 7 MS Word templates, several disparate Outlook address books and thousands of uncoordinated emails between staff.

    The new sales management system leads the sales and support staff through a well defined process, which ensures the necessary information is gathered and stored consistently. The main focus in this project was on minimising the need for double data entry. This is achieved by storing all customer details in a flexible information model that even allows to track the same person with several roles with different companies.

    Quotes and sales documents are produced automatically by the system as PDF documents – this format has been chosen due to its compatibility with a large number of computer systems. The system also keeps track of all past quote information, which is accessible for reports. Reports, which have previously been done by copying information into shared Excel sheets is now available to staff in real time.

    The equipment that Streat Control imports is extremely complex and requires hours to days for the construction of quotes. They further deal with heavy weights such as NZ Steel, Auckland City Council or DB Breweries, who will tender out every job, but sometimes it is known in advance that Streat Control will not get a job. This situation allowed for the system to provide return of investment by reducing the amount and scale of repetitive unnecessary work.

    Once ordered, the equipment is then manufactured overseas and usually ready for shipping within 6 weeks. A mistake in this period will lead to a delay of a further 6 weeks, as manufacturers will “re queue” the order. Once shipped, a mistake can mean that you have a 500kg/ $30000 instrument in New Zealand that is hard to get rid of and unsellable.

    Copy and Paste mistakes were wide spread and frustrated sales staff, causing unnecessary follow up communication and searching of email archives.

  • Mental Health Service Database - Mind and Body Consultants

    Mind and Body provide Mental Health Peer Support - a support service that they have pioneered themselves and that is currently delivered in Central and West Auckland as well as Christchurch. They were looking for an integrated management system for their peer support workers. Initially this was triggered through the government requirements to deliver billing data electronically under the PRIMHD standard. At the same time, Mind and Body wanted to "own and control" the solution, so they are not dependent on a specific vendor.

    mindandbody-casestudy-smallWe provided a custom built database management system that tracks patients/peers through their year long engagement and collects all relevant time sheet and costing information along the way. This was built as a web based secure system, so that it could be easier extended with mobile phone travel expense tracking at a later date.

    We were also involved in training and delivery of the software, which was based on a "train the trainer" model. The choice for this was made to create maximum engagement of staff with the introduction of the software, so as to minimise any resistance that might otherwise to be expected.

    Finally with our help Mind and Body became PRIMHD compliant within 1 month after launch - the accepted minimum time frame typical for this was 6 month, with implementations ranging up to 2 years in terms of turnaround time.

    "Eileen informed me today that we are now officially PRIMHD compliant. That is a huge feat in the time that we have been sending off reports to the ministry. Quite frankly I am astounded (in a very good way) that we have done this in such short order. The expectation out there with other organisations is that it takes at least 6 months. Jochen and Eileen have managed it in about 1 month. Well done to the both of you and thanks."

    Rodger Jack - Mind and Body Consultants

     

     

  • Multi-National Multi-Lingual Website - Philips Selecon

    Philips Selecon - formerly Selecon New Zealand - designs, manufactures and exports theatre and architectural lighting systems to the world. Having been their Content Management System (CMS) provider for the past 9 years, we were charged in 2008 with building a 2nd generation corporate website, which included dealing with over 5000 products, over 1000 detail pages such as news, case studies and support material as well as E-Commerce facilities.

    Philips Selecon's products are market leaders or well presented in many major world wide markets, because of this the website Content Management System (CMS) had to manage multiple translations of the content as well as content that is only available in some markets or is hidden from some markets. We achieved this by

    • providing a language translation platform that can be selectively applied to content with an automatic fall back to British English as the default language. This includes facilities to detect changes to the original, so that translations can be refreshed. Languages in use are British English, US English, German, French, Russian and Spanish.
    • providing a facility to hide a complete US supplier portfolio from markets outside New Zealand, as this would otherwise lead to conflicts in the supplier relationship
    • providing a facility to easily hide key products from markets with 110V power current, as some products are technically not compatible with it. This has proven much more successful that marking these products with additional information. Our approach also allows to mark such products once, even though there may be references at 8+ areas in the website.
    • providing a facility to show country specific welcome messages in some areas of the website.

    To showcase Philips Selecon's work, a large number of case studies were reworked and produced. We extended the CMS to provide a simple keyword manager, so that these case studies can be easily categorised by country, type of use and various other keywords. Furthermore, the products use are linked to the case study.
    Based on this powerful information model, we are able to randomly retrieve select case studies in nearly all areas of the website - by relating the products used and keywords to the content of the core functionality of the page.

    To allow easy shopping for repeat users of the website, it also features a store area, where all key products and accessories are listed in 5 sections. While the core products have various support documentation and feature lists associated with them, we also had the challenge to present 5500 imported colour filters - basically coloured plastic sheets and gobos - metal frames to create different patterns on stage.

    We firstly proceeded to create a screen scraping program as to retrieve the information from the suppliers website, because they were not able and motivated to provide this information in a structured format.

    As a second step we created structured product pages for these products. As a third and final step, we created categorised shopping pages for groups of these products. This was done to present the products in a more concise fashion and also because a user would typically buy a number of colour sheets or related patterns in different sizes for a stage production. For this reason our presentation saves time for the user as well.

    Selecon New Zealand was acquired by Philips corporation on 1 Apr 2009. Since then www.seleconlight.com has continued to be the by far most sophisticated, detailed and extensive website in the stable of Philips lighting technology subsidiaries.

    As a result of this, we have also proceeded to integrate the products, case studies and support material of a US based sister operation of Philips into the website system. The fully integrated manner in how these 2 websites operate allows Philips to position themselves as a full service operator. At the same time, we are able to retain all existing links and domain names, as these remain unchanged.

  • Large Joomla Website - The Parenting Place

    The Parenting Place is a non-profit organisation specialising in parenting education. Having educated 15000+ parents in courses and sold over 20000 books and education resources, all largely through their website they were faced with a number of challenges.

    • The website needed a facelift to keep up with branding changes and an expansion of the services and approach of the organisation to parenting education. At the same time they wanted to use their excellent in-house graphic design resources to maximise brand cohesion and cost effectiveness.

    • It became difficult to extend the existing website and provide security updates. This was because it contained a lot of custom developed modules and complex extensions to existing modules.

    • Email marketing is a major activity for The Parenting Place, but it was hard to assess how many emails reached recipients due to emails being sent directly from the web server and being likely a target of spam filters.

    Facelift

    theparentingplace-case-study-2-smallFor the changes to look and feel, we worked closely with the Parenting Place in-house team to develop a new brand expression for the website. Once this was complete, we took over and developed website code and the associated Joomla templates.

    One of the key drivers was to implement a solution that would work well on many mobile browsers and as a result responsive web design solution techniques were chosen to allow for a cost effective coverage of desktop, IPad, IPhone and other mobile browsers.

    Website Upgrade

    To upgrade this website to a current Joomla version we used a combination of techniques:

    • Complex extensions, such as product and country specific promotional codes in Joomla's Virtuemart E-Commerce module were upgraded to the newest version using best practice software development techniques. This allowed us to re-use previously invested effort and kept specification changes to a minimum

    • Simpler extensions, such as webinar management were simply upgraded and audited for security and concerns for code complexity.

    • Fairly simple modules, based on complex off-the-shelf Joomla components, such as the event calendar were replaced with newly written components based on the popular Symfony framework.

    Email Marketing

    To increase effectiveness of the email marketing, the delivery platform had to be moved away from the web server and onto a dedicated platform.

    We chose Mailchimp for this, due to its ease of use, ability to target emails to different audiences and options to automate integration of email list management.

    As a result, email subscription preferences are still tightly integrated between The Parenting Place website and the email marketing platform and at the same time, we enjoy excellent delivery rates and the features and statistics of a major email marketing platform.

Welcome

jochen-daum-automatem

Welcome to Automatem Ltd.

I'm the owner/ operator, Jochen Daum with over 15 years of experience in Website and Web Application Development. Please contact me if there is anything I can help with.