Memoirs of a Roadie

Growing Up

Posted on 13th August 2010

YAPC::Europe 2010 - Thoughts Pt 1/3 - Young Blood & The Old Guard

Last week I was in Pisa for YAPC::Europe 2010. Although I was doing a talk about CPAN Testers, my intention was to keep a low profile and observe more. Having run the conference surveys for the past few years, it has been noticeable that the attendance has been changing. While there are new people coming along to YAPCs, the general average age is getting older. Marketing Perl to companies to encourage its use is one thing, but attracting people in general to the language is also important. The fact that for a notable number of attendees this is their first YAPC, probably means we are getting something right.

There were several European Perl Mongers that were noticeably absent this year. While some had posted apologies (mostly due to imminent baby arrivals it would seem!), others perhaps have moved on to other jobs, projects or languages, or their life means that they cannot commit to something like YAPC any more. While we miss them, it is a natural way for the community to evolve. It does give a chance for newcomers to become involved and this year I wanted to see who we are potentially going to see more of.

It seems we have quite a few people who are giving us, the Perl community, a fresh look and I think that the Perl community is rather healthy at the moment thanks to them. At least from a European perspective. YAPCs are an ideal chance for people to meet and discuss projects, which otherwise can take days or weeks via email and even IRC. Those new to projects can better introduce themselves and forge better communication channels with other project members, both during the conference and at the evening social events. I think it was Dave Rolsky who observed that the Europeans seemed more accustomed to putting down laptops and talking, rather than sitting in silence hacking away. There certainly seemed to be lots of discussion in hallways this year at least.

With all the fresh faces around, it's crossed my mind on several occasions, as to who is the old guard these days. There are several I could name who kind of fit the bill, and many of us have been around working on projects for quite a few years. Not necessarily hacking on perl itself, but certainly helping to build the Perl community. We have quite a vibrant community, one that I think is quite inclusive, supportive and appreciative. We have disagreements at times, but it's a community that seems to easily span age and experience barriers and is willing to learn from each other.

Keeping a low profile initially seemed to be working for me, that is right up until the afternoon of the last day. During the day, José had asked if I would help with his lightning talk, but not wanting to be part of any more talks, I respectfully declined. Little did I realise it was just a ruse, so he could say thank you to me for organising and running the YAPC Surveys. So much for not drawing attention to myself! After the Lightning Talks, brian d foy took centre stage to present the White Camel Awards. I was very pleased to see both Paul Fenwick and José Castro receive awards, and in fact was laughing at José as he realised one of the awards was going to him. However, José was almost in hysterics when he saw my reaction when I realised I was also receiving an award.

As I mentioned in my acceptance speech, I've never wanted an award for what I do. I do it because I want to, and because I love being part of this community. I had been asked before whether I would accept a White Camel Award, and I'd said no. Although I don't think the awards themselves are a bad thing, its just that I think others have been more deserving of them. I've been involved in many Perl projects over the years, and have largely hid behind them, as I've always felt the projects themselves are far more important than me. The fact that several people felt I needed to be acknowledged this year, regardless of my reluctance to receive the award, I guess means that sometimes I just have to accept that people would like to say thank you for the work I do. If like José, there was one person I should thank for introducing me to the Perl community, it would be Richard Clamp. It was Richard who gave me my first proper Perl job and persuaded me to go to a London Perl Mongers social.

Which sort of brings me to one of the projects I helped with last year, and I'm very pleased to see continuing this year. Introducing people to the Perl community is one aspect of the Send-A-Newbie programme. Edmund instigated the programme last year, and we managed to bring 3 people to YAPC, giving them a chance to experience the conference and the community. The hope was that they would use and benefit from the experience, and hopefully feel more empowered to contribute to the community. Then maybe be in the future, they might be able to attend future YAPCs. I was delighted to see Alan Haggai Alavi at this year's YAPC, and surprised to see him so soon. I was then even more impressed to hear what he has been doing to promote Perl in India, as this is exactly the kind of enthusiasm the Send-A-Newbie programme can benefit from too. I spoke briefly with Leon Timmermans, who was this year's attendee via the Send-A-Newbie programme, and again it seems we've found another deserving recipient.

With programmes like Send-A-Newbie, the Perl marketing efforts and the community in general, I'm very hopeful that we'll be seeing more young blood in the community in the years to come. However, it still needs some effort from every one of us to ensure that happens. Which brings me to my next post in this short series, which I'll be posting soon.

I've now been in the community for over 10 years, with Birmingham Perl Mongers celebrating their 10th birthday in September. I'm guess that means I'm one of the old guard now, which isn't bad for a C programmer who had a lot to learn all those years ago. I feel I've come a long way in the last 10 years, and it's been a fantastic journey. Perl and the community have changed immensely in those years, and I'm looking forward to seeing how the young blood and fresh faces now, take us in new and interesting directions over the next 10 years and more.

All Over The World

Posted on 19th May 2010

Last year I went to 3 conferences, YAPC::NA, YAPC::Europe and LUGRadio Live. All very different in their own way, although all Open Source. Due to other projects, work and fanmily commitments, it has take quite a bit of time to review all the photos. After several months, I finally found some time to whittle them down to the selection I have uploaded here.

The first conference, YAPC::NA, took place in Pittsburgh, PA, USA. The team have been holding the Pittsburgh Perl Workshops for several years now, and by all accounts they had been very well received. With the YAPC set of conferences having started in Pittsburgh, at the Carnegie Mellon University where this conference also took place, the organisers were quite proud to promote a sort of home coming for the event. It was a good conference, though my first talk was somewhat problematic as we couldn't get a laptop to work with the projector. Thankfully my second talk went without a hitch. My thanks to confound for introducing me to 'xrandr', which solved all the problems I had getting Ubuntu talking to the projectors.

I also took the chance to visit the city of Pittsburgh and take a look around, with Abigail and myself taking an amphibious vehicle tour. The city has a lot of interesting places to see, and I'm glad I got to see the Heinz Building, the Stanley Theatre (where Bob Marley played his last ever gig), the venicular railways, and the site of the Three River Stadium (Yes, the car park! I'd seen a documentary about the building of Heinz Field and the demolishing of the Three River Stadium, so was even more intrigued to have a guided tour of the new stadium). If you're ever near the city, I recommend a visit, especially to see the flood levels of 1936 (the waters peaked at a rise of 46 feet above normal!).

The second conference, YAPC::Europe, was in Lisbon, Portugal. The conference itself was packed full of talks, though I think my lightning talk, which I'd been refining over the previous few months, generated the biggest reaction. Not surprising really, as it reminded people just how productive the Perl community was, particularly regarding CPAN.

I had originally thought about hiring a car and travelling along the Vasco da Gama Bridge (at 10.7 miles long, the longest road bridge in Europe), and do the circuit via the monument on the other side of the Tejo river, and back to Lisbon via the 25 de Abril Bridge (Lisbon's other bridge). I didn't in the end, but maybe I can save that for another time. Instead fellow Birmingham.pm'er Brian McCauley and myself walked around the city and took in some of the sights. When we got to the castle we managed to bump into a few other attendees (Paul Johnson, Aaron Crane and R Geffory Avery), who also had taken the advantage to do some sightseeing.

The last conference I attended was LUGRadio Live. For a number of reasons I didn't put forward a talk this year, but suggested JJ should give a talk instead. With the radio show no longer running, the conference had much more of a grassroots feel to it again. There ware some good talks, a couple of famous names, but mostly it felt like it was one big Linux User Group meeting, which to a degree it was, just a bit more global than your regular user group meeting ;) The conference was dubbed 'Back To Basic', but that really only applied to the extravagance. The quality of the conference was first rate. Being in Wolvehampton, just round the corner for me, I didn't take the opportunity to do any sightseeing, not that Wolverhampton is exactly the kind of place to do any sightseeing. As it happens I had taken Dan to the event, who loved it, especially building the lego models with all the other geeks. The following day was OggCamp, and although I would have liked to have attended, I had other commitments so had to pass. I think having the two events side by side though was a great idea, as it gives both events to feed off each other.

This year I'm currently only planning one conference, YAPC::Europe in Pisa, Italy. All being well I may get to see the tower, but as I'll be flying in and out just for the conference, I don't expect to see much more. I'm still undecided whether to submit a talk, as I'm trying to think of a suitable subject. I don't like repeating myself, but my two biggest profile Perl projects I've now covered for a couple of years (CPAN Testers and YAPC Surveys), so we'll see.

More photos to come, as I find time to get through the plethora of photos I've taken over the last year or so.

Long Time Gone

Posted on 4th May 2010

It has been quite a few months since I last posted here. Quite a few events and projects have happened and held my attention since I last wrote in my blog. And I still have a backlog of photos and videos from last year to get through too!

I did wonder whether anyone might think that after talking about Why The Lucky Stiff in one of my last posts, that I had done the same. Well for those who follow my CPAN Testers work, will know that CPAN Testers 2.0 has been a rather major project that finally got properly underway in December 2009. It's nearing completion, and I'll cover some of the highlights in a future post. Although it's been my most consuming project over the last 6 months or so, it hasn't been my only one. As mentioned in another of my last posts, I'm writing a book about how to host a YAPC. Due to other projects taking a higher priority, this has taken somewhat of a backseat for the time being, but I do plan on getting a second draft together within the next few months. I have looked into self-publishing the book and I'm now planning to have it formerly submitted with an ISBN (the internation book numbers) and supplied via print-on-demand print runs.

Another project that has been ongoing alongside my CPAN Testers work, has been my website management system, Labyrinth. This has been the website application I have been developing since 2002, and although several other Perl web frameworks have now been developed since, to lesser and greater degrees, Labyrinth has had the disadvantage of only having 1 core developer for the past 8 years. It's not an application that will revolutionise web development and deployment, but it has very successfully worked for a number of websites I have developed over the years. After having been relatively stable for the past year or two, I'm now cleaning up the code so I can properly release it as open source. This is mostly so that anyone wishing to contribute to CPAN Testers, or the YAPC Surveys, will then have all the code available to them. If anyone wants to use it and help develop it further, that would be a welcome bonus, but realistically other web frameworks have gained so much mindshare that I'm not expecting Labyrinth to make much of a dent any more. Not that that is a problem, as Labyrinth has made deploying websites so much easier for me, that I'll just be glad to let people help on CPAN Testers and the YAPC Surveys.

Speaking of the YAPC Surveys, YAPC::NA 2010 and YAPC::Europe 2010 are fast approaching. These will be next projects to get up and running. Thankfully the code base just needs a few upgrades to the latest version of Labyrinth, and some work on skinning the CSS to match the respective YAPC sites. All being well this should only take a few days. Then I'll be looking to release this version of the code base for anyone wishing to run similar surveys for themselves. I've already had one interested party contact me regarding a conference in October, so hopefully the code will be suitable, and only the questions need adapting. We shall see.

My other major project this year, also began back in December 2009. As some readers are well aware, I am an ex-roadie. From 1989-1994 I was a drum tech, lighting engineer and driver for Ark, one of the best Black Country bands ever. Not that I'm biased or anything ;) Last year the band got together for some rehearsals and planned a few reunion gigs. With interest gaining, an album was also planned. So this year, the band began recording and booking gigs. As a consequence the Ark Appreciation Pages desperately needed a makeover. I'll write more about what happened next in another post. Ark are back, and Mikey and I are delighted to be able to be involved with the band once again.

That's just a few of the projects that have taken up my time over the last 6-8 months. There are several others that I hope to post about, with family, time and work permitting. Expect to hear a little more from me than you have so far this year.

April Skies

Posted on 1st May 2009

For those that might not be aware, I got made redundant on 31st March (the day after the QA Hackathon had finished). Thankfully, I start a new job next week, so I've managed to land on my feet. However, this has meant that I've ended up having the whole of April off to do stuff. My plan was to work on some of the Open Source projects that I'm involved with to move them further along to where I wanted them to be. As it turned out two specific projects got my attention over the last 4 weeks, and I thought it worth giving a summary of what has been going on.

YAPC Conference Surveys

Since 2006, I've been running the conference surveys for YAPC::Europe. The results have been quite interesting and hopefully have help organisers improve the conferences each year. For 2009 I had already planned to run the survey for YAPC::Europe in Lisbon, but this year will also see YAPC::NA in Pittsburgh having a survey of their own.

The survey site for Copenhagen in 2008 added the ability to give feedback to Master Classes and talks. The Master Classes feedback was a little more involved, as I was able to get the attendee list, but the talks feedback was quite brief. As such, I wanted to try and expand on this aspect and generally improve the process of running the surveys. Part of this involved contacting Eric and BooK to see if ACT had an API I could use to automate some of the information. I was delighted to get an email back from Eric, who very quickly incorporated an API that I could use, to retrieve the necessary data to keep the survey site for a particular conference up to date, even during the conference.

With the API and updates done, it was time to focus on expanding the surveys and skinning the websites to match that of the now live conference sites. The latter was relatively easy, and only required a few minor edits to the CSS to get them to work with the survey site. The survey site now has 3 types of survey available, though only 2 are visible to anyone not taking a Master Class. Those that have taken one of the YAPC::Europe surveys will be aware I don't use logins, but a key code to access the survey. This has been extended so that it can now be used to access your portion of the survey website. This can now be automatically emailed to attendees before the conference, and during if they pay on the door, and will allow everyone to feedback on talks during the conference. On the last day of the conference the main survey will be put live, so you can then answer questions relating to your conference experience.

I'm hoping the slight change won't be too confusing, and that we'll see some ever greater returns for the main survey. Once it does go live, I'd be delighted to receive feedback on the survey site, so I can improve it for the future.

CPAN Testers Reports

Since taking over the CPAN Testers Reports site in June 2008, I have spent a great deal of time improving it's usability for users. However, it's come at a price. By using more and more Javascript to dynamically change the contents of the core pages, it's meant that I have received a number of complaints that the site doesn't work for those with Javascript disabled or who use a browser that doesn't implement Javascript. For this reason I had decided that I should create a dynamic site and static site. The problem with this is that the current system to create all the files takes several hours for each set of updates (currently about 16 hours per day). I needed a way to drive the site without worrying about how long everything was taking, but also add some form of prioritisation so that the more frequently requested pages would get updated more quickly than those rarely seen.

During April, JJ and I went along to the Milton Keynes Perl Mongers technical meeting. One of the talks was about memcached and it got me thinking as to whether I could use it for the Reports site. Discussing this with JJ on the way home, we threw a few ideas around and settled on a queuing system to decide what needed updating, and to better managed the current databases to add indexes to speed up some of the complex lookups. I was still planning to use caching, but as it turned out memcached wasn't really the right way forward.

The problem with caching is that when there is too much stuff in the cache, the older stuff gets dumped. But what if the oldest item to get dumped is extremely costly on the database, and although it might not get hit very often, it's frequent enough to be worth keeping in the cache permanently. It's possible this could be engineered with memcached if this was for a handful of pages, but for the Reports site it's true for quite a few pages. So I hit on a slightly different concept of caching. As the backend builder process is creating all these static files, part of the process involves grabbing the necessary data to display the basic page, with the reports then being read in via the now static Javascript file for that page. Before dropping all the information and going on to the next in the list, the backend can simply write the data to the database. The dynamic site can then simply grab that data and display the page pretty quickly, saving ALOT of database lookups. Add to the fact that the database tables have been made more accessible to each other, the connection overhead has also been reduced considerably.

The queuing system I've implemented is extremely simple. On grabbing the data from the cache, the dynamic site checks quickly to see if there is a more recent report in existence. If there is, then a entry is added to the queue, with a high weighting to indicate that a website user is actually interested in that data. Behind the scenes the regular update system simply adds an entry in the queue to indicate that a new entry is available, but at a low weighting. The backend builder process then looks to build the entries with the most and highest weightings and builds all the static files, both for the dynamic site and the static site, including all the RSS, YAML and JSON files. It seems to work well on the test system, but the live site will be where it really gets put through its paces.

So you could be forgiven in thinking that's it, the new site is ready to go. Well not quite. Another part of the plan had always been to redesign the website. Leon had designed the site based on the YUI layouts, and while it works for the most part, there are some pages which don't fit well in that style. It also has been pretty much the same kind of style since it was first launched, and I had been feeling for a while that it needed a lick of paint. Following Adam's blog post recently about the state of Perl websites, I decided that following the functional changes, the site would get a redesign. It's not perhaps as revolutionary as some would want, judging from some of the ideas for skins I've seen, but then the site just needs to look professional, not state of the art. I think I've managed that.

The work to fit all the pieces together and ensure all the templates are correct is still ongoing, but I'm hopeful that at some point during May, I'll be able to launch the new look websites on the world.

So that's what I've been up to. I had hoped to work on Maisha, my other CPAN distributions, the YAPC Conference Survey data, the videos from the QA Hackathon among several other things, but alas I've not been able to stop time. These two projects perhaps have the highest importance to the Perl community, so I'm glad I've been able to get on with them and get done what I have. It's unlikely I'll have this kind of time again to concentrate solely on Open Source/Perl for several years, which in some respects is a shame, as it would be so nice to be paid to do this as a day job :) So for now, sit tight, it's coming soon...

Back On Line

Posted on 16th February 2009

After the last few weeks of trying to access Twitter from the command line, I set about writing something that I could expand to micro-blog to any social networking site that supports many of the Twitter API type commands. At the moment it only works with Twitter and Identi.ca, but my plan is to look at creating plugins, or more likely to allow others to create plugins, that can enable the tool to interact with other micro-blogging sites.

After trying to think of a decent name, I finally settled on Maisha. It's a Swahili word meaning "life". You can grab the code from CPAN as App-Maisha.

Currently you'll need to use the standard Perl install toolset to install the application, but ultimately I'd like to have something that you can install just about anywhere without having to go through all the headache of installing dependencies. I'll have a go at doing an .rpm and a .deb package release, and will also try using PAR. It would be nice to have this as a standalone application that just about anyone can use, but for now CPAN will have to do.

My next immediate step is to look at writing something that interfaces to Facebook without requiring a developer key or any such nonsense. It will probably have to involve a bit of screen scraping, unless there is some more official API, but as yet I haven't found it. Everything regards Facebook applications seems to centre around the developer application that can do all sorts of dubious things, but mine is purely for the user to control from their desktop, not a 3rd party website/server. Thus giving them a developer API key assigned to me is wholly inappropriate. It would be nice if they had a restricted User API, which allows you to update your status and look at your friends' statuses, but I think I'll be in the minority wanting it.