07 281 1289
Screen scraping is the process of extracting information from a website to store it in a database, convert it or otherwise use it.
Screen scraping, otherwise known as data scraping is often used when there is an “asynchronous” power relationship between the business using it and a supplier or customer the information is retrieved from. Example of such relationships include:
Tenders for telecommunications, power and other utilities can involve accessing data on a website in an inefficient way, however the tender may not allow for the data format to be changed.
Similarly government departments may not have resource or assign priority to a data provision request, even though the appreciate the value it may bring to the outcome to be achieved.
NZ buyers of European and American suppliers may be faced with poorly formatted product data, but the supplier may not see improved quality data as a priority due to the small market share a New Zealand business would have worldwide.
Screen scraping can be done once off, or as a continuous process. It can be in one direction or feed data back to the website it was retrieved off.
If you see the data in a browser, then it can be scraped!
Streat Control - an importer of electrical engineering goods - has a distributed operation in Auckland, Wellington and Christchurch, serving the local waterworks, refineries, breweries and others with instruments for liquid and gas monitoring and control. With these clients, everything is about accuracy. Their goods are five figures plus per unit and are custom manufactured to order and shipped to New Zealand. As this process is lengthy and costly, mistakes are very expensive.
We provided a custom build and web based importing and tracking system, which also covers some areas of the Customer Relationship Management and the complete process of quoting, ordering and delivery. Through our development process we have replaced 7 MS Word templates, several disparate Outlook address books and thousands of uncoordinated emails between staff.
The new sales management system leads the sales and support staff through a well defined process, which ensures the necessary information is gathered and stored consistently. The main focus in this project was on minimising the need for double data entry. This is achieved by storing all customer details in a flexible information model that even allows to track the same person with several roles with different companies.
Quotes and sales documents are produced automatically by the system as PDF documents – this format has been chosen due to its compatibility with a large number of computer systems. The system also keeps track of all past quote information, which is accessible for reports. Reports, which have previously been done by copying information into shared Excel sheets is now available to staff in real time.
The equipment that Streat Control imports is extremely complex and requires hours to days for the construction of quotes. They further deal with heavy weights such as NZ Steel, Auckland City Council or DB Breweries, who will tender out every job, but sometimes it is known in advance that Streat Control will not get a job. This situation allowed for the system to provide return of investment by reducing the amount and scale of repetitive unnecessary work.
Once ordered, the equipment is then manufactured overseas and usually ready for shipping within 6 weeks. A mistake in this period will lead to a delay of a further 6 weeks, as manufacturers will “re queue” the order. Once shipped, a mistake can mean that you have a 500kg/ $30000 instrument in New Zealand that is hard to get rid of and unsellable.
Copy and Paste mistakes were wide spread and frustrated sales staff, causing unnecessary follow up communication and searching of email archives.
Mind and Body provide Mental Health Peer Support - a support service that they have pioneered themselves and that is currently delivered in Central and West Auckland as well as Christchurch. They were looking for an integrated management system for their peer support workers. Initially this was triggered through the government requirements to deliver billing data electronically under the PRIMHD standard. At the same time, Mind and Body wanted to "own and control" the solution, so they are not dependent on a specific vendor.
We provided a custom built database management system that tracks patients/peers through their year long engagement and collects all relevant time sheet and costing information along the way. This was built as a web based secure system, so that it could be easier extended with mobile phone travel expense tracking at a later date.
We were also involved in training and delivery of the software, which was based on a "train the trainer" model. The choice for this was made to create maximum engagement of staff with the introduction of the software, so as to minimise any resistance that might otherwise to be expected.
Finally with our help Mind and Body became PRIMHD compliant within 1 month after launch - the accepted minimum time frame typical for this was 6 month, with implementations ranging up to 2 years in terms of turnaround time.
"Eileen informed me today that we are now officially PRIMHD compliant. That is a huge feat in the time that we have been sending off reports to the ministry. Quite frankly I am astounded (in a very good way) that we have done this in such short order. The expectation out there with other organisations is that it takes at least 6 months. Jochen and Eileen have managed it in about 1 month. Well done to the both of you and thanks."
Rodger Jack - Mind and Body Consultants
Philips Selecon - formerly Selecon New Zealand - designs, manufactures and exports theatre and architectural lighting systems to the world. Having been their Content Management System (CMS) provider for the past 9 years, we were charged in 2008 with building a 2nd generation corporate website, which included dealing with over 5000 products, over 1000 detail pages such as news, case studies and support material as well as E-Commerce facilities.
Philips Selecon's products are market leaders or well presented in many major world wide markets, because of this the website Content Management System (CMS) had to manage multiple translations of the content as well as content that is only available in some markets or is hidden from some markets. We achieved this by
To showcase Philips Selecon's work, a large number of case studies were reworked and produced. We extended the CMS to provide a simple keyword manager, so that these case studies can be easily categorised by country, type of use and various other keywords. Furthermore, the products use are linked to the case study.
Based on this powerful information model, we are able to randomly retrieve select case studies in nearly all areas of the website - by relating the products used and keywords to the content of the core functionality of the page.
To allow easy shopping for repeat users of the website, it also features a store area, where all key products and accessories are listed in 5 sections. While the core products have various support documentation and feature lists associated with them, we also had the challenge to present 5500 imported colour filters - basically coloured plastic sheets and gobos - metal frames to create different patterns on stage.
We firstly proceeded to create a screen scraping program as to retrieve the information from the suppliers website, because they were not able and motivated to provide this information in a structured format.
As a second step we created structured product pages for these products. As a third and final step, we created categorised shopping pages for groups of these products. This was done to present the products in a more concise fashion and also because a user would typically buy a number of colour sheets or related patterns in different sizes for a stage production. For this reason our presentation saves time for the user as well.
Selecon New Zealand was acquired by Philips corporation on 1 Apr 2009. Since then www.seleconlight.com has continued to be the by far most sophisticated, detailed and extensive website in the stable of Philips lighting technology subsidiaries.
As a result of this, we have also proceeded to integrate the products, case studies and support material of a US based sister operation of Philips into the website system. The fully integrated manner in how these 2 websites operate allows Philips to position themselves as a full service operator. At the same time, we are able to retain all existing links and domain names, as these remain unchanged.
The Parenting Place is a non-profit organisation specialising in parenting education. Having educated 15000+ parents in courses and sold over 20000 books and education resources, all largely through their website they were faced with a number of challenges.
The website needed a facelift to keep up with branding changes and an expansion of the services and approach of the organisation to parenting education. At the same time they wanted to use their excellent in-house graphic design resources to maximise brand cohesion and cost effectiveness.
It became difficult to extend the existing website and provide security updates. This was because it contained a lot of custom developed modules and complex extensions to existing modules.
Email marketing is a major activity for The Parenting Place, but it was hard to assess how many emails reached recipients due to emails being sent directly from the web server and being likely a target of spam filters.
For the changes to look and feel, we worked closely with the Parenting Place in-house team to develop a new brand expression for the website. Once this was complete, we took over and developed website code and the associated Joomla templates.
One of the key drivers was to implement a solution that would work well on many mobile browsers and as a result responsive web design solution techniques were chosen to allow for a cost effective coverage of desktop, IPad, IPhone and other mobile browsers.
To upgrade this website to a current Joomla version we used a combination of techniques:
Complex extensions, such as product and country specific promotional codes in Joomla's Virtuemart E-Commerce module were upgraded to the newest version using best practice software development techniques. This allowed us to re-use previously invested effort and kept specification changes to a minimum
Simpler extensions, such as webinar management were simply upgraded and audited for security and concerns for code complexity.
Fairly simple modules, based on complex off-the-shelf Joomla components, such as the event calendar were replaced with newly written components based on the popular Symfony framework.
To increase effectiveness of the email marketing, the delivery platform had to be moved away from the web server and onto a dedicated platform.
We chose Mailchimp for this, due to its ease of use, ability to target emails to different audiences and options to automate integration of email list management.
As a result, email subscription preferences are still tightly integrated between The Parenting Place website and the email marketing platform and at the same time, we enjoy excellent delivery rates and the features and statistics of a major email marketing platform.