Memoirs of a Roadie

Grand Designs

Posted on 31st December 2013

Over the last year I've made several releases for Labyrinth and its various plugins. Some have been minor improvements, while others have been major improvements as I've reviewed the code for various projects. I originally wrote Labyrinth after being made redundant back in December 2002, and after realising all mistakes I made with the design of its predecessor, Mephisto. In the last 11 years has helped me secure jobs, enabled me to implement numerous OpenSource projects (CPAN Testers and YAPC Conference Surveys to name just two) and provided the foundation to create several websites for friends and family. It has been a great project to work on, as I've learnt alot about Perl, AJAX/JSON, Payment APIs, Security, Selenium and many other aspects of web development.

I did a talk about Labyrinth in Frankfurt for YAPC::Europe 2011, and one question I was asked, was about comparing Labyrinth to Catalyst. When I created Labyrinth, Catalyst and its predecessor Maypole, were 2 years (and 1 year) away from release. Back then I no idea about an MVC, but I was pleased that in later years when I was introduced to the design concept, that it had seemed an obvious and natural way to design a web framework. Aside from this and both being written in Perl, Labyrinth and Catalyst are very different beasts. If you're looking for a web framework to design a mojor system for your company, then Catalyst is perhaps the better choice. Catalyst also has a much bigger community, whereas Labyrinth is essentially just me. I'd love for Labyrinth to get more usage and exposure, but for the time being, I'm quite comfortable with it being the quiet machine behind CPAN Testers, YAPC Surveys, and all the other commercial and non-commercial sites I've worked on over the years.

This year I finally released the code to enable Labyrinth to run under PSGI and Plack. It was much easier than I thought, and enabled me to better understand the concepts behind the PSGI protocol. There are several other concepts in web development that are emerging, and I'm hoping to allow Labyrinth to teach me some of them. However, I suspect most of my major work with Labyrinth in 2014 is going to be centred on some of the projects I'm currently involved with.

The first is the CPAN Testers Admin site. This has been a long time coming, and is very close to release. There are some backend fixes that are still needed to join the different sites together, but the site itself is mostly done. It still needs testing, but it'll be another Labyrinth site to join the other 4 in the CPAN Testers family. The site has taken a long time to develop, not least because of various other changes to CPAN Testers that have happened over the few years, and the focus on getting the reports online sooner rather than later.

The next major Labyrinth project I plan to work on during 2014, is the YAPC Conference Surveys. Firstly to release the current code base and language packs, to enable others to develop their own survey sites, as that has been long over due. Secondly, I want to integrate the YAPC surveys into the Act software tool, so that promoting surveys for YAPCs and Perl Workshops will be much easier, and we won't have to rely on people remembering their keycode login. Many people have told me after various events that they never received the email to login to the surveys. Some have later been found in spam folders, but some have changed their email address and the one stored in Act is no longer valid. Allowing Act to request survey links will enable attendees to simply log into the conference site and click a link. Further to this, if the conference has surveys enabled, then I'd like the Act site to be able to provide links next to each talk, so that talk evaluations can be donme much more easily.

Lastly, I finally want to get all the raw data online as possible. I still have the archives of all the surveys that have been undertaken, and some time ago I wrote a script to create a data file, combining both the survey questions and the responses, appropriately anonymised, with related questions linked, so that others can evaluate the results and provide even more statistical analysis than I currently provide.

In the meantime the next notable release from Labyrinth will be a redesign of the permissions system. From the very beginning Labyrinth had a permissions system, which for many of the websites was adequate. However, the original Mephisto project encompassed a permissions system for the tools it used, which for Labyrinth were redesigned as plugins. Currently a user has a level of permission; Reader, Editor, Publisher, Admin and Master. Each level grants more access than the previous one as you might expect. Users can also be assigned to groups, which also have permissions. It is quite simplistic, but as most of the sites I've developed only have a few users, granting these permissions across the whole site has been perfectly acceptable.

However, with a project I'm currently working on this isn't enough. Each plugin, and its level of functionality (View, Edit, Delete), need different permissions for different users and/or groups. The permissions system employed by Mephisto came close, but they aren't suitable for the current project. A brainwave over Christmas saw a better way to do this, and not just to implement for the current project, but to improve and simplify the current permission system, and enable to plugins to set their permissions in data or configuration rather than code, which is a key part of the design of Labyrinth.

This ability to control via data is a key element of how Labyrinth was designed, and it isn't just about your data model. In Catalyst and other web frameworks, the dispatch table is hardcoded. At the time we designed Mephisto, CGI::Application was the most prominent web framework, and this hardcoding was something that just seemed wrong. If you need to change the route through your request at short notice, you shouldn't have to recode your application and make another release. With Labyrinth switching templates, actions and code paths is done via configuration files. Changing can be dne in seconds. Admittedly it isn't something I've needed to do very often, but it has been necessary from time to time, such as disabling functionality due to broken 3rd party APIs, or switching templates for different promotions.

The permission system needs to be exactly the same. A set of permissions for one site may be entirely different for another. Taking this further, the brainwave encompassed the idea of profiles. Similar to groups, a profile can establish a set of generic permissions. Specific permissions can then be adjusted as required, and reset via a profile on a per user or per group basis. This then allows the site permissions to be tailored for a specific user. This then allows UserA and UserB to have generic Reader access, but for UserA to have Editor access to TaskA and UserB to be granted Editor access to TaskB. Previously the permission system would have meant both users be granted Editor access for the whole site. Now, or at least when the system is finished, a user's permissions can be set so they can be restricted to only the tasks they need access to.

Over Christmas there have been a few other fixes and enhancements to various Labyrinth sites, so expect to see those to also find their way back into the core code and plugins. I expect several Labyrinth related releases this year, and hopefully a few more talks at YAPCs, Workshops and technical events in the coming year about them all. Labyrinth has been a fun project to work on, and long may it continue.

Die Mensch-Maschine

Posted on 10th November 2011

German Perl Workshop 2011 - Speaker Evaluations

I have now sent out all the talk evaluations from this year's German Perl Workshop or more correctly Der 13. Deutsche Perl-Workshop. If you were a speaker and haven't received an email, please check your spam folders first, and let me know (barbie at cpan . org) if you don't find it. The mail will have come from barbie at birmingham . pm . org.

My thanks to all the organisers of GPW2011 and everyone who took the time to respond to the evaluations. From previous experience the speakers have very much appreciated your feedback. I would also like to extend extra special thanks to Max Maischein aka "Corion", who took the time to translate all the questions, templates and emails into German for me.

The results of the main survey will be published soon on the YAPC Conference Surveys site.

This is the first survey that I have undertaken in a non-English language, and for the most part it has been very successful. While there have been some slight problems due to byte vs character lengths (I'll save my 'why-oh-why did we ever start with ASCII and not UTF-8' rant for another day), the work Max has done to provide all the translations has started me on a path to be able to accommodate other languages.

At the moment the plan is to create a GitHub repository of all the necessary files, with language branches containing the appropriate translations. Then should anyone wish to request a survey instance in the future in a non-English language, their first step will then be to provide the necessary translations for me. It currently takes roughly a day to set-up an instance, so drop-in replacements for these files will ease the set-up process. It will also mean that as time goes on and questions get added, refined or deleted, we can replicate these changes across all languages.

I'd like to see the survey site get more use in the future, and although I'm happy to run the survey sites, with the support of Birmingham Perl Mongers, the longer term goal has always been to allow others to create their own instances. With the official release of Labyrinth this year, much of the tool set is now Open Source. I still need to release the Survey Plugin for Labyrinth and the additional command-line tools used, but getting the language translations moving will be a big step forward. Hopefully I'll have more news in the new year.

Questions & Answers

Posted on 9th May 2011

I mentioned in my last post that I was working on a Survey Plugin for Labyrinth. The plugin is used within the YAPC Conference Survey system, which has now been running for several YAPC events over the last 5 years. I had promised to try and release the complete survey site last year, but with it being a Labyrinth based site setup, I didn't want to release it without releasing Labyrinth first. Now that's done I can concentrate on getting the Survey Plugin and the complete survey system on CPAN.

This year I will be running the YAPC::NA and YAPC::Europe surveys as per usual. However, this year I am delighted to say I have also been asked to handle the survey for the Pittsburgh Perl Workshop too. Hopefully if all goes to plan, this will provide the test bed for many other workshops to provide surveys.

The Conference Surveys themselves started in 2006, and have provided some very interesting feedback for organisers. While event organisers and myself never expect to get 100% response from all attendees, the levels that we do get is absolutely phenomenal. With this kind of success, I would be very interested to see whether the same Survey system can be used by other non-Perl events. There is certainly nothing that prevents a non-Perl (or even a non-tech) event from using the system. Last year I did have a query from a non-Perl event, but the system wasn't ready for a stand-alone release, and I wasn't able to set anything up. However, this year, with a CPAN release coming soon, I am more hopeful that others might be able to use the system.

If you are an organiser for an event where you think a survey would be useful for feedback, please do get in touch. If I cannot host an instance for you, once I get a full release on CPAN, I can provide help and advice for getting your own hosting instance running.

Into The Blue

Posted on 7th May 2011

I haven't been posting recently about the Perl projects I'm currently working on, so over the next few posts I hope to remedy that.

To begin with, one of the major projects I've been involved with for the past 8 years has been CPAN Testers, although you can find out more of my work there on the CPAN Testers Blog. This year I've been releasing the code that runs some of the websites, specifically those that are based on my other major project, Labyrinth. Spearheading these releases have been the CPAN Testers Wiki and CPAN Testers Blog, with further releases for the Reports, Preferences and Admin sites also planned. The releases have taken time to put together mostly because of the major dependency they all have, which is Labyrinth.

Labyrinth is the website management framework I started writing back in 2002. Since then it has grown and become a stable platform on which to build websites. With both the CPAN Testers Wiki and the CPAN Testers Blog, three key plugins for Labyrinth have also been released which hopefully others can make use of.

The Wiki plugin, was intended to be written for the YAPC::Europe 2006 Wiki, but with pressures of organising the conference and setting up the main conference site (which also used Labyrinth), I didn't get it finished in time. Once a CPAN Testers Wiki was mooted, I began finishing off the plugin and integrating into Labyrinth. The plugin has been very stable for the last few years, and as a consequence was the first non-core plugin to be released. It's a fairly basic Wiki plugin, not too many bells and whistles, although there are a couple of Perlish shortcuts, but for the most part you don't need them. The CPAN Testers Wiki codebase release was also the first complete working site for Labyrinth, which was quite a milestone for me.

Following that success, the next release was for the CPAN Testers Blog. Again the underlying plugin, the Blog Plugin, has been stable for a few years, so was fairly quick to package and release, however the secondary plugin, the Event Plugin, has been evolving for quite some time and took a little more time. As I use both these plugin for several other sites, it was a good opportunity to bring together any minor bug fixes and layout changes. Some of these have seen slight modifications to the core Labyrinth codebase and the core set of plugins. In addition it has prompted me to start working on the documentation. It is still a long way from being complete, but at least the current documentation might provide some guidance to other users.

One of my major goals for Labyrinth was for it to be a 'website in a box'. Essentially this means that I wanted anyone to take a pre-packaged Labyrinth base (similar to the Demo site), drop it on a hosting service and be able to run a simple installation script to instantiate the database and configuration. The installation would then also be able to load requested plugins, and amend the database and configuration files appropriately. I haven't got to that stage yet, but it is still a goal.

With this goal in mind, I have read with interest the recent postings regarding the fact that DotCloud are now able to run Perl apps. This is definitely great news, and is exactly the kind of setup I had wanted to make best use of for the 'website in a box' idea. However, with several other frameworks now racing to have the coolest instance, it isn't something I'm going to concentrate on right now for Labyrinth. Plus there is the fact that Labyrinth isn't a PSGI framework, which others have eagerly added to their favourite framework. Labyrinth came from a very different mindset than other now more well known frameworks, and tries to solve some slightly different problems. With just me currently working on Labyrinth, as opposed to the teams of developers working on other frameworks, Labyrinth is never going to be the first choice for many reasons. I shall watch with interest the successes (and lessons learned from any hiccups) of the other frameworks as it is something I would like to get working with Labyrinth. If anyone who has the time and knows PGSI/Plack well enough, and would like to add those capabilities to Labyrinth, please get in touch.

The next notable plugins I'll be working on are the Survey, Music and Gallery Plugins. The former has its own post coming shortly. The next notable CPAN Testers site released planned is the Reports site. With it being considerably more involved, it might take a little longer to package and document, but it will likely be the most complex site release for Labyrinth, which will give anyone interested in the framework a good idea of how it can be used to drive several sites all at once.

Long Live Rock'n'Roll

Posted on 9th February 2011

On the 1st January 2011, I released the first Open Source version of Labyrinth, both to CPAN and GitHub. In additon I also released several plugins and a demo site to highlight some of the basic functionality of the system.

Labyrinth has been in the making since December 2002, although the true beginnings are from about mid-2001. The codebase has evolved over the years as I've developed more and more websites, and got a better understanding exactly what I would want from a Website Management System. Labyrinth had the intention of being a website in a box, and although it's not quite there yet, hopefully once I've released all the plugin code I can put a proper installation tool in place.

Labyrinth now is the backend to several Open Source websites, CPAN Testers using it for the Reports, Blog, Wiki and Preferences sites, as well as some personal, commercial and community projects. As a consequence Labyrinth has become stable enough to look at growing the plugins, rather than the core code. I'm sure there is plenty that could be done with the core code, but for the moment providing a good set of plugins, and some example sites are my next aims.

As mentioned, I see Labyrinth as a Website Management System. While many similar applications and frameworks provide the scaffolding for a Content Management System, Labyrinth extends that by not only providing the ability to manage your content, but also to provide a degree of structure around the functionality of the site, so the management of users and groups, menu options and access, as well as notification mechanisms, enable you to provide more control dynamically.

When writing the fore-runner to Labyrinth, one aspect required was the ability to turn on and off functionality instantly, which meant much of the logic flow was described in the data, not the code. Labyrinth has built on this idea, so that the dispatch tables and general functionality can be controlled by the user via administration screens, and not by uploading new code. When I started looking at this sort of application back in 2001, there was nothing available that could do that. Today there are several frameworks written in Perl that potentially could be tailored to process a website in this way, but all require the developer to design and code the functionality. Labyrinth aims to provide that pre-packaged.

I'm primarily releasing Labyrinth so that I can release all the code that drives the CPAN Testers websites. Giving others the ability to better suggest improvements and contribute. The system allows me the freedom to build websites quickly and easily, with the hardwork being put into the design and CSS layouts. With so many other frameworks available, all of which have bigger development teams and support mechanisms than I can offer, I'm not intending Labyrinth to be a competitor. It might interest some, which is great, but if you prefer to work on other frameworks that's great too. After all it's still Perl ;)

More news of plugins and sites being released coming soon.