Memoirs of a Roadie

Points of Authority

Posted on 27th May 2011

Back in February I did a presentation for the Birmingham Perl Mongers, regarding a chunk of code I had been using to test websites. The code was originally based on simple XHTML validation, using the DTD headers found on each page. I then expanded the code to include pattern matching so I could verify key phrases existed in the pages being tested. After the presentation I received several hints and suggestions, which I've now implemented and have set up a GitHub repository.

Since the talk, I have now started to add some WAI compliance testing. I got frustrated with finding online sites that claimed to be able to validate full websites, but either didn't or charged for the service. There are some downloadable applications, but most require you to have Microsoft Windows installed or again charge for the service. As I already had the bulk of the DTD validation code, it seemed a reasonable step to add the WAI compliance code. There is a considerable way to go before I get all the compliance tests that can be automated written into the distribution, but some of the more immediate tests are now there.

As mentioned in my presentation to Birmingham.pm, I still have not decided on a name. Part of the problem being that the front-end wrapper, Test::XHTML, is written using Test::Builder so you can use it within a standard Perl test suite, while the underlying package, Test::XHTML::Valid uses a rather different approach and does provides a wider API than just validating single pages against a DTD specification. Originally, I had considered these two packages should be two separate releases, but now that I've added the WAI test package, I plan to expose more of the functionality of Test::XHTML::Valid within Test::XHTML. If you have namespace suggestions, please let me know, as I'm not sure Test-XHTML is necessarily suitable.

Ultimately I'm hoping this distribution can provide a more complete validation utility for web developers, which will be free to use and will work cross-platform. For those familiar with the Perl test suite structure, they can use it as such, but as it already has a basic stand-alone script to perform the DTD validation checks, it should be usable from the command-line too.

If this sounds interesting to you, please feel free to fork the GitHub repo and try it out. If you have suggestions for fixes and more tests, you are very welcome to send me pull requests. I'd be most interested in anyone who has the time to add more WAI compliance tests and can provide a better reporting structure, particularly when testing complete websites.

The Sanity Assassin

Posted on 12th May 2011

An update to my recent post.

With thanks to a fellow Perler, Smylers informs me that a Flash Cookie refers to the cookie used by Flash content on a site, which saves state on the users machines, by-passing browsers preferences. Odd that the advice singles out this type of cookie by name though, and not the others.

In an article on the Wall Street Journal I found after posting my article, I found it interesting to discover that the ICO themselves use Google Analytics. So after 25th May, if you visit the ICO website and see no pop-up, I guess that means Google Analytics are good to go. Failing that they'll see a deluge of complaints that their own website fails to follow the EU directive.

I also recommend reading the StatCounter's response too. They also note the problem with the way hosting locations are (not) covered by the directive, and the fact that the protection from behavioural advertising has got lost along the way.

After a discussion about this at the Birmingham.pm Social meeting last night, we came to the considered opinion that this would likely just be a wait and see game. Until the ICO bring a test case to court, we really won't know how much impact this will have. Which brings us back to the motives for the directives. If you're going to take someone to court, only big business is worth fining. Bankrupting an individual or a small business (ICO now have powers to fine up to £500,000) is going to give the ICO, the government and the EU a lot of really negative press.

Having tackled the problem in the wrong way, those the directives sort to bring into line are only going to use other technologies to retrieve and store the data they want. It may even effect EU hoisting companies, if a sizeable portion of their market decide to register and host their websites in non-EU countries.

In the end the only losers will be EU businesses, and thus the EU economy. Did anyone seriously think these directives through?

The Planner's Dream Goes Wrong

Posted on 11th May 2011

On May 26th 2011, UK websites must adhere to a EU directive regarding cookies, that still hasn't been finalised. Other member states of the EU are also required to have laws in place that enforce the directive.

Within the web developer world this has caused a considerable amount of confusion and annoyance, for a variety of reasons, and has enabled media outlets to scaremonger the doom and gloom that could befall developers, businesses and users. It wouldn't be so bad if there was a clear piece of legislation that could be read, understood and followed, but there isn't. Even the original EU directives are vague in the presentation of their requirements.

If you have the time and/or inclination the documents to read are Article 2 of Directive 2009/136/EC (the Directive), which amends the E-Privacy Directive 2002/58/EC (the E-Privacy Directive), with both part of the EU Electronic Communications Framework (ECF).

Aside from the ludicrous situation of trying to enforce a law with no actual documentation to abide by (George Orwell would have a field day), and questioning why we are paying polictians for this shambolic situation, I have to question the motives behind the creation of this directive.

The basic Data Protection premise for tightening up the directive is a reasonable one, however the way it has been presented is potentially detremental to the way developers, businesses and users, particularly in the EU, are going to browse and use the internet. The directive needed tightening due to the way advertisers use cookies to track users as they browse the web and target adverts. There has been much to complain about in this regard, and far beyond the use of cookies with companies such as Phorm trying to track information at the server level too. However, the directive has ended up being too vague and covers too wide a perspective to tackle the problem effectively.

Others have already questioned whether it could push users to use non-EU websites to do their business because they get put off using EU based sites. Continually being asked whether you want to have information stored in a cookie every time you visit a website is going to get pretty tiresome pretty quickly. You see, if you do not consent to the use of cookies, that information cannot be saved in a cookie, and so when revisiting the site, the site doesn't know you said no, and will ask you all over again. For those happy to save simple preferences and settings stored in cookies, then you'll be asked once and never again. If you need an example of how bad it could get, Paul Carpenter took a sartirical look at a possible implementation.

On Monday 9th May 2011, the Information Commissioner's Office (ICO) issued an advice notice to UK businesses and organisation on how to comply with the new law. However even their own advice states the document "is a starting point for getting compliant rather than a definitive guide." They even invent cookie types that don't exist! Apparently "Flash Cookies" is a commonly used term, except in the web technology world there are just two types of cookie, Persistent Cookies and Session Cookies. They even reference the website AllAboutCookies, which makes no mention of "Flash Cookies". Still not convinced this is a complete shambolic mess?

The directives currently state that only cookies that are "strictly necessary" to the consumer are exempt from the ruling. In most cases shopping carts have been used as an example of cookie usage which would be exempt. However, it doesn't exempt all 1st party cookies (those that come from the originating domain), and especially targets 3rd party cookies (from other domains). The advice states "The exception would not apply, for example, just because you have decided that your website is more attractive if you remember users' preferences or if you decide to use a cookie to collect statistical information about the use of your website." Both of which have significant disruption potential for both websites and their visitors.

Many of the 1st party cookies I use are Session Cookies, which either store an encrypted key to keep you logged into the site, or store preferences to hide/show elements of the site. You could argue both are strictly necessary or not depending on your view. Of the 3rd party cookies, like many people these days, I use Google Analytics to study the use of my websites. Of particular interest to me is how people find the site, and the search words used that brough the visitor to the site. It could be argued that these are strictly necessary to help allow the site visitor find the site in the first place. Okay its a weak argument, but the point remains that people use these types of analysis to improve their sites and make the visitor experience more worthwhile.

Understandly many people have questioned the implications of using Google Analytics, and on one Google forum thread, the Google approved answer seems to imply that it will only mean websites make it clearer that they use Google Analtyics. However this is at odds with the ICO advice, which says that that isn't enough to comply with the law.

If the ruling had been more explicit about consent for the storing of personal data in cookies, such as a name or e-mail address, or the use of cookies to create a personal profile, such as with advertisier tracking cookies, it would have been much more reasonable and obvious what is permissible. Instead it feels like the politicians are using a wrecking ball to take out a few bricks, but then aiming at the wrong wall.

For a site like CPAN Testers Reports, it is quite likely that I will have to block anyone using the site, unless they explictly allow me to use cookies. The current plan is to redirect people to the static site, which will have Google Analytics switched off, and has no other cookies to require consent. It also doesn't have the full dynamic driven content of the main site. In Germany, which already has much stricter requirements for data protection, several personal bloggers have choosen to not use Google Analytics at all in case they are prosecuted. I'm undecided at the moment whether I will remove GA from my websites, but will watch with interest whether other bloggers use pop-ups or remove GA from their sites.

Perhaps the most frustrating aspect of the directives and the advice is that it discusses only website compliance. It doesn't acknowledge that the websites and services may be hosted on servers outside the EU, although the organisation or domain may have been registered within the EU. It also doesn't differentiate between commercial businesses, voluntary organisations or individuals. Personal bloggers are just as at risk to prosecution as multinational, multibillion [currency of choice] businesses. The ICO is planning to issue a separate guidance on how they intend to enforce these Regulations, but no timescale is given. I hope that they make it absolutely clear that commercial businesses, voluntary organisations or individuals will all be treated differently from each other.

In their eagerness to appear to be doing something, the politicians, in their ignorance, have crafted a very misguided ruling that will largely fail to prevent the tracking of information and creation of personal profiles, which was the original intent of the changes. When companies, such as Phorm, can create all this personal information on their servers, using the same techology to capture the data, but sending it back to a server, rather than saving a cookie, have these directives actually protected us? By and large this will be a resounding No. Have they put in place a mission to disrupt EU business and web usage, and deter some from using EU based websites? Definitely. How much this truly affects web usage remains to be seen, but I suspect initially there will be an increase in pop-ups appearing on websites asking to use cookies.

It will also be interesting to see how many government websites adhere to the rulings too.

Into The Blue

Posted on 7th May 2011

I haven't been posting recently about the Perl projects I'm currently working on, so over the next few posts I hope to remedy that.

To begin with, one of the major projects I've been involved with for the past 8 years has been CPAN Testers, although you can find out more of my work there on the CPAN Testers Blog. This year I've been releasing the code that runs some of the websites, specifically those that are based on my other major project, Labyrinth. Spearheading these releases have been the CPAN Testers Wiki and CPAN Testers Blog, with further releases for the Reports, Preferences and Admin sites also planned. The releases have taken time to put together mostly because of the major dependency they all have, which is Labyrinth.

Labyrinth is the website management framework I started writing back in 2002. Since then it has grown and become a stable platform on which to build websites. With both the CPAN Testers Wiki and the CPAN Testers Blog, three key plugins for Labyrinth have also been released which hopefully others can make use of.

The Wiki plugin, was intended to be written for the YAPC::Europe 2006 Wiki, but with pressures of organising the conference and setting up the main conference site (which also used Labyrinth), I didn't get it finished in time. Once a CPAN Testers Wiki was mooted, I began finishing off the plugin and integrating into Labyrinth. The plugin has been very stable for the last few years, and as a consequence was the first non-core plugin to be released. It's a fairly basic Wiki plugin, not too many bells and whistles, although there are a couple of Perlish shortcuts, but for the most part you don't need them. The CPAN Testers Wiki codebase release was also the first complete working site for Labyrinth, which was quite a milestone for me.

Following that success, the next release was for the CPAN Testers Blog. Again the underlying plugin, the Blog Plugin, has been stable for a few years, so was fairly quick to package and release, however the secondary plugin, the Event Plugin, has been evolving for quite some time and took a little more time. As I use both these plugin for several other sites, it was a good opportunity to bring together any minor bug fixes and layout changes. Some of these have seen slight modifications to the core Labyrinth codebase and the core set of plugins. In addition it has prompted me to start working on the documentation. It is still a long way from being complete, but at least the current documentation might provide some guidance to other users.

One of my major goals for Labyrinth was for it to be a 'website in a box'. Essentially this means that I wanted anyone to take a pre-packaged Labyrinth base (similar to the Demo site), drop it on a hosting service and be able to run a simple installation script to instantiate the database and configuration. The installation would then also be able to load requested plugins, and amend the database and configuration files appropriately. I haven't got to that stage yet, but it is still a goal.

With this goal in mind, I have read with interest the recent postings regarding the fact that DotCloud are now able to run Perl apps. This is definitely great news, and is exactly the kind of setup I had wanted to make best use of for the 'website in a box' idea. However, with several other frameworks now racing to have the coolest instance, it isn't something I'm going to concentrate on right now for Labyrinth. Plus there is the fact that Labyrinth isn't a PSGI framework, which others have eagerly added to their favourite framework. Labyrinth came from a very different mindset than other now more well known frameworks, and tries to solve some slightly different problems. With just me currently working on Labyrinth, as opposed to the teams of developers working on other frameworks, Labyrinth is never going to be the first choice for many reasons. I shall watch with interest the successes (and lessons learned from any hiccups) of the other frameworks as it is something I would like to get working with Labyrinth. If anyone who has the time and knows PGSI/Plack well enough, and would like to add those capabilities to Labyrinth, please get in touch.

The next notable plugins I'll be working on are the Survey, Music and Gallery Plugins. The former has its own post coming shortly. The next notable CPAN Testers site released planned is the Reports site. With it being considerably more involved, it might take a little longer to package and document, but it will likely be the most complex site release for Labyrinth, which will give anyone interested in the framework a good idea of how it can be used to drive several sites all at once.

Addicted to Chaos

Posted on 31st March 2011

Sometime ago, a website I was working on needed the ability to view images on the current page from a thumbnail. Many websites now feature this functionality, but at the time only a few seemed to offer this, and the assumption was that the javascript required was rather complex. As such, I did a search of the viewer libraries available, either as Open Source or for free download, that I could use for a commercial website.

The initial search revealed a rather more limited result than I expected, and seemed to imply that the complexity had put people off from developing such a library. However, in retrospect it seems that a market leader has become so popular, stable and robust, that others have choosen to provide different or limited presentations based on similar designs.

Back last year I began writing a review of some of the viewers, but never got around to finishing it. Having some time recently, I decided to both complete the review and revisit the viewers to see what improvements have been made since I first investigated them.

Before I begin the individual reviews, I should note the requirements I was looking for in a viewer. Firstly, the viewer needed to be self contained, both with files and directory structure, so that the feature could be added or removed with minimal changes to other website files. The viewer needed to be run completely on the client side, no AJAX or slow loading of large images would be acceptable. However, the most significant requirement was that all code needed to work in IE6. Unfortunately this latter requirement was non-negotiable.

I was quite surprised by the results of the solutions I could find around the web, and although there are likely to be others now, the following is a brief review of each of the four immediate solutions I found, and my experiences with them.

Lightbox

Possibly the best know thumbnail viewer library available, and now a clear market leader. The original review was with v2.04, which had been the stable release from 2008. This month (March 2011) has seen a version 2.05 release with added IE9 support. Lightbox is licensed under the Creative Commons Attribution 2.5 License, and is free to use for commercial projects, although a donation would be very much appreciated.

While this viewer works in most browsers, and the features of images sets and loading effects looked great, it proved unworkable in many of the IE6 browsers I tried across multiple platforms. Despite searching in forums and in some howtos, there didn't seem to be an obvious fix to the problem. The viewer would either not load at all, load with a black layer over the whole web page, or begin to load and crash the browser. I know there are many problems and faults with IE6 and the javascript rendering engine, but these were supposedly stable releases.

As Lightbox makes use of the Prototype Framework and Scriptaculous Effects Library, which was already being used within the website the viewer was for, the library initially seemed to be the best fit. Failing IE6 so dramatically and consistently, disappointingly meant it couldn't be pursued further.

• Lightbox website

Slimbox

Slimbox is a Lightbox clone written for the JQuery Javascript Library. v2.04 is the last stable release, and the release that was originally reviewed. Slimbox is free software released under MIT License.

Slimbox is based on Lightbox 2, but utilises more of the JQuery framework and is thus slightly less bulky. While working well in the browsers I tried, it flickered several times in IE6 when loading the image. Anyone viewing the effect with eplipsy might well have felt ill. Even for someone not affected by eplisey this strobing effect was extremely off putting. I suspect this problem may well be an alternative side-effect to those seen with the original Lightbox, but again forums and howtos didn't provide a suitable fix in order to remedy this problem.

• Slimbox website

Dynamic Drive Thumbnail Viewer

This is the first thumbnail viewer that Dynamic Drive have available, as the second is an inline viewer rather than an overlay, which is what I was after, and is the version made available on July 7th, 2008. Scripts by Dynamic Drive are made available under their Terms of Use, and are free to use for commercial projects.

This a very basic viewer, relying on basic functionality rather than flashy effects. As such, it is simple in design and presentation. Rather than create a full browser window overlay, as both Lightbox and Slimbox do, the Dynamic Drive viewer simply contains the viewing image within a simple DIV layer tag. There is the possibility to add visual effects, but these can be easily turned off.

This seemed to work in most of the browser tried, except when clicking the image in IE6. The image appeared, but then immediately a javascript error popped up. After quickly reviewing the configuration and turning off the animation, the viewer opened and worked seamlessly across all the browsers tested.

• Dynamic Drive Thumbnail Viewer website

Highslide JS

Highslide JS is a very feature rich library, which provides much more than an image viewer. Highslide JS is licensed under a Creative Commons Attribution-NonCommercial 2.5 License, which means you are free to use the library for non-commercial projects. For commercial projects two payment methods are available, $29 for a single website, and $179 for unlimted use.

The feature set for displaying images includes the style of animation to open images, the positioning of text, and the linking of image sets. In addition, it also provides many features for regular content, which can then be used for tooltip type pop-ups, using embedded HTML, IFrames and AJAX. Another standard feature is the ability to allow the user to move the pop-up around the screen, to wherever might be convienent.

However, there is a downside. While this works well in most browsers, even just loading the Highslide JS website in IE6 throws up several errors. With the library being so feature rich, it is a considerably larger codebase, although removing comments can remove this down to just over 8KB, and I suspect some of the older browsers may not be able to handle some of the complexity. Their compatibility table suggests that it works all the way back to IE 5.5, but in the tests performed for IE6, when the site did open without crashing the browser, the viewer itself felt rather clunky when an image was opened and several of the visibility settings just didn't work. You also frequently get an 'Unterminated string constant' error pop-up, which just feels disconcerting considering they are asking you to pay for commercial usage.

If IE6 wasn't a factor, this may have been a contender, as the cost is very reasonable for a commercial project that would utilise all its features.

• Highslide JS website
  

Conclusion

These are just the four viewers that were prominent in searches for a "thumbnail viewer". They all seem to have the same, or at least a similar, style of presentation of images, which is likely due to the limited way images can be displayed as an overlay. However, the basic functionality of displaying an image seems to have been overshadowed by how many cool shiny features some can fit into their library, with configuration seeming to be an after thought.

With the ease of configuration to disable the IE6 error, the basic functionality and the freedom to use for commercial projects, the Dynamic Drive solution was utimately chosen for the project I was working on. If IE6 wasn't a consideration, I would have gone with Lightbox, as we already use Prototype and Scriptaculous. With IE6 usage dwindling on the website in question (Jun 2010: 38.8%, down to Mar 2011: 13.2%), it is quite possible that we may upgrade to a more feature and effect rich viewer in the future, and Lightbox does seem to be a prime candidate.

Consider this post a point of reference, rather than a definitie suggestion of what image viewer library to use. There may be other choices that suit your needs better than these, but these four are worth initial consideration at the very least.

Browsers & Operating Systems

For reference these were the browsers I tried, and the respective operating systems. And yes, I did test IE6 on Linux, where it occasionally stood up better than the version on Windows! Though this may be due to the lack of ActiveX support.

• IE6 (WinXP, Windows7, Linux)
• IE7 (Windows7)
• IE8 (Windows7)
• Firefox 3.6 (WinXP, Windows7, Linux)
• Opera 9.8 (Linux)
• Opera 10.52 (Linux)
• Chrome 5 (Windows7, Darwin)
• Chromium 6 (Linux)
• Safari 4 (Darwin, iOS)