Memoirs of a Roadie

The Time of the Turning

Posted on 7th May 2013

A few weeks ago I had the pleasure of attending the 6th annual QA Hackathon. The event has become THE event for developers of test modules, projects and toolchain applications to come together to discuss ideas and plan for the future, as well as release some great work while they are there too.

This year Shadowcat, the primary sponsors, took on the organisational duties. The event was originally to be in London, but due to personal circumstances the decision was made to move the location to Lancaster in the North West of England. Personally they made the right choice. The venue itself was the new InfoLab building at Lancaster University. The attendees came from far and wide once again, and it was great to catch-up with friends old and new, and even be introduced to some newer friends.

My plan for the weekend was mainly to look at CPAN Testers. With the servers for the Metabase coming soon, David Golden and myself had hoped to be able to set them up, and start looking at changing the backend code to work with the new Metabase database. Unfortunately, the servers weren't ready for us just yet, so I started looking at other things. For myself, one area of CPAN Testers, particularly the cpanstats database side of things, needed attention. Speed of processing reports.

My first task once settled in, was to look at the way that the reports are consumed from the Metabase. Due to the way SimpleDB has become very unreliable with the results it sends, in order to avoid missing reports the criteria for the date search has been altered slightly to be a little more thorough, and a smaller range is now used to retrieve a set of GUIDs. The results now appear to be a little more complete, although we still appear to be missing some every so often. There is also a tail of log.txt which also helps to catch up with the reports. This work saw a new release of CPAN-Testers-Data-Generator.

A big factor with the slowness of the CPAN Testers server is that it requires a lot of disk I/O, with the database updates being a key factor. The most intensive updates are surrounding the SQLite database that could be downloaded. This also includes creating the Gzip and Bzip2 archives. As only web crawlers seem to be downloading the files, I've suspended the update. This has now freed up a lot of resources and consequently some of the other tasks, particularly the builder has improved.

Next, the builder was the focus of my attention. Previously the builder has been building pages for both authors and distros all at once. Although the author pages are viewed slightly less, they were getting built more frequently, due to the way the requests are pushed into the queue for each report. Initially the logic for building pages was altered, which improved some of the higher requested pages, but the more optimal solution was to split the builder into two, one for authors and one for distros. With the reduction in processing elsewhere, this improved the builder performance considerably. Monitoring the way the author pages are built since the hackathon, has also allowed me to alter when the builder for authors runs. This has then allowed the builder for distros to take a higher priority. With more distro pages than authors, this now gives distro pages more opportunity to be built quicker. Currently reports are being built in less than 24 hours of being submitted. These updates saw a new release of CPAN-Testers-WWW-Reports.

Another release while at the event, related to the QA Hackathon itself, was the main QA Hackathon website. Before the event, BooK had asked if the files that make up the website that the main QA Hackathon uses could be added to GitHub. As such, I packaged up the site into a git repository and released it. If you wish to help contribute to the site, please do.

Although there was a lot of coding work involved in the weekend, one of the bigger uses of time was the Lancaster Consensus organised by David Golden. For a few hours each afternoon, a large group of key toolchain developers, secondary project developers and various interested parties, gathered to discussed various aspects associated with configuration, installation, testing and specification of Perl and CPAN. With so many developers in one room, it wasn't too surprising to have a few opposing views, but with a guiding hand from David, we did achieve a consensus. If you wish to read the outcome, please read David's write-ups of the discussion points. The Consensus meetings were perhaps the greatest achievement of the event. While there might not have been too much immediate coding output from them, the potential to improve Perl and CPAN is considerable. From a CPAN Testers perspective, Post-installation testing, Case insensitive package permissions and Rules for distribution naming were perhaps of most interest. Although it may be some time before Post-installation testing could be hooked into a CPAN Testers smoker, it will be a valuable addition to the testing reports against pre-installed environments.

During the event, I had several discussions with Garu regarding his work on the cpanminus smoker client, and the common smoker client. In the last minutes of the hackathon we were able to push through a very notable report submission. It is exactly this sort of collaborative effort that makes these hackathons worthwhile. I look forward to see everyone again in Lyon.

The QA Hackathons could not be the success they are with the support of all the sponsors. My personal thanks to them for helping to providing accommodation, food and a venue for us all to hack. A big thank you to cPanel, Dijkmat, Dyn, Eligo, Evozon, $foo, Shadowcat Systems Limited, Enlightened Perl Organisation and Mongueurs de Perl.

To Wish Impossible Things

Posted on 4th May 2013

The QA Hackathon website has had a bit of an update today. Primarily a new page and new photos have been added, but plenty of other updates have been included too.

The new page is a review page, to collect various blog and news posts relating to each year's event. Originally I listed all the reviews from previous years in the side panel, but now that we've just had the 6th annual event, the list was looking a little bit too cramped.

With the extra space, I've also been able to include the group shots that were taken at some of the events. Unfortunately there was no group shot taken in Birmingham, and I've not seen any during the 2010 and 2011 events, so if there are any, please let me know. Also if there is one of the Tokyo Satellite event this year I would love to include it on the site.

I've added some write-ups to the last few events in the About page. The biggest change though is likely only visible to those with screen readers, as I've made many changes to links and images to provide more accessibility. Several fixes to layout, spelling and wording have also been included too.

The site, particularly the list of reviews, is still incomplete. If a blog entry is missing that you think should be there, or you spot other items that could do with an update, feel free to email me with details, or fork the repo on GitHub and send me a pull request.

Enjoy The Silence

Posted on 4th May 2013

Unfortunately I've been having problems with my outbound mail from my personal mail server. I recently updated the server, and although the mail software was also updated, the problem seems to be regarding resolving IP addresses to the Virgin Media nameservers. I can resolve DNS lookups via my laptop, but the server gets stuck and times out :(

If I don't get it fixed soon, I'll have to look at using Gmail on a more permanent basis, so that I can at least reply to messages again and know they're going to get through. Although I don't mind using Gmail on occasions, storing all my mail with them does make me a little uncomfortable.

So if you haven't heard from me, and have been expecting a response, it's likely that I replied, but it never got through. By all means send me a reminder so I know to send a reply through Gmail to make sure you get the reply.

I've also been quiet on my blog for quite sometime. I have been rather busy with work, family, work, Perl, open source and work, and finding time to write here has been low on my list of things to do. However, I have a few posts coming soon to help redress the balance.