Big data and data analytics Seminar

A photo of the interior of the Glazier's Hall

On Thursday, I was lucky enough to be invited to attend Big data and data analytics: commercial opportunities, privacy and effectiveness, one of several seminars offered by the Westminster eForum. It took place at the Glazier’s Hall, on London’s south bank next to London Bridge.

The four hour session, split into two halves, was chaired by two members of the House of Lords, Lord Inglewood and Lord Witty, and the speakers represented various users of big data in the UK. These included the ABPI, whose members carry out research and development into new medications, Dunnhumby, who worked with Tesco to launch the original Clubcard in the 1990s, academics and industry partners.

The talks given by the speakers were interesting, and focussed more on policy and high-level overviews, rather than technical details. For example, whilst Hadoop was passively mentioned on some slides, there wasn’t much about deployment and how it works. But there was some discussion about database design, as companies move away from traditional relational databases to big data capturing solutions.

Privacy implications came up several times as well, an irony not lost on one of the speakers who noted that the event coincided with Data Privacy Day. In particular, there was a focus on how to design systems with privacy in mind, but also that the UK’s and Europe’s more restrictive privacy laws may be part of the reason why the world’s biggest data users – GAFA (Google, Apple, Facebook & Amazon) – are all based in the US.

I came into the seminar essentially wearing two hats. My main reason for attending was as a blogger (or ‘freelance writer’ according to the attendance list), but data analysis is also one of my roles at work. However, we’re not yet at the stage where we’re using ‘big data’ – most of our data is all within standard relational databases and I can’t see that changing any time soon.

As always, such events also offer a chance to network and it was good to speak with some of the other attendees. As you’d expect from a more high-level seminar, this was an event for people with suits and ties, and not t-shirts and hoodies. Many were from government departments, regulators and other public sector bodies, as well as large organisations such as the BBC and Arqiva.

I came away with plenty of notes, and some action points to perhaps bring up at work. Channel 4’s Viewer Promise video was mentioned as great example of best practice for explaining their privacy policy – far better than pages and pages of legalese. Maybe universities could do something similar to explain the student contract at enrolment.

This day was made possible by Dell, but all thoughts are my own.

Open Tech 2005

I’m sat on the train back to Bradford (another direct one with wifi, woohoo!), ready to leave King’s Cross. The verdict on Open Tech 2005? Possibly the best £5 I’ve spent in a very long time.

There was wifi at the event (and indeed lots of geeks with Powerbooks reading their Bloglines subscriptions) however for some reason I couldn’t get it to work with this PDA. Damn Windows Mobile…

Anyway, what follows is my take on the days events.

Despite it being only two days after some wannabe terrorists tried to blow up the tube, the underground seemed surprisingly normal and I had no problems getting to where I needed to be. The event was being held in the Reynolds Building, part of the Imperial College London’s medical school in Charing Cross Hospital, and the talks were in a couple of the teaching rooms there. The first talk I went to was by Danny O’Brien on the issues surrounding privacy amongst those in the public eye, especially bloggers. It was very interesting (even if it did take a while to start due to internet access problems, eventually sorted with an impromptu SSH tunnel); Danny is a good speaker and kept the audience interested.

At 12:30 I moved over to the seminar room where I saw a Linux-powered iPod for the first time (and even watched a video on it), learned about scripting in iTunes and about MythTV, along with the BBC’s open source project Kamaelia. I also bumped into Sven from Blogwise and Gia who vaguely remembered me from the London Geek Dinner last month.

During a much needed lunch break Phil Wilson came to chat to me (it’s great to be recognised :) ), and we then headed back to the lecture room for the official launch of BBC Backstage – effectively the Beeb’s developer network, with APIs and syndication feeds galore (see BBC News Online article). There’s already some people doing cool stuff with it, like taking the BBC’s travel news feed and plotting the problems on a Google map, or a ‘changelog’ of the BBC News Online home page – Ben Metcalfe, who was presenting, used the archives from July 7th as a good example of why this was so cool. Ben Hammersley was the ‘master of ceremonies’ – he’s even more mad than his blog depicts him as. He was wearing a skirt

Ben H also MC’d the next session which was with Jeremy Zawodny from Yahoo!. Jeremy’s talk focused on openness in technology – RSS, APIs etc. and how it benefits companies (and the problems it can create). Obviously being from Yahoo! the focus was on Yahoo! and Flickr but it did show how Yahoo! is ‘getting it’. While it didn’t make me as excited as the BBC talk it’s still good to see big companies opening themselves up, as it were.

It was at this point that I found out I’d won the third prize in the prize draw, so I picked up an O’Reilly USB flash drive (256MB!), along with a 1-year subscription to Make magazine. I also bought a copy of Ben’s ‘Developing Feeds with RSS and Atom(sponsored link), which he signed.

The next session I went to was called Blogs and Social Software, and was chaired by Gia. Tom Reynolds, who works for the London Ambulance Service, did a talk about how to blog and not lose your job, which included lots of cat pictures, as any blogging talk should. He was followed by Paul Mutton, creator of the PieSpy IRC bot, about creating diagrams showing how individuals interact. He used data from Digital Spy’s Big Brother site to create an interaction map of the Big Brother housemates, which was pretty cool. Finally Paul Lenz talked about his Who Should You Vote For? site and his new project What Should I Read Next?, which lets you type in a book you have read and recieve suggestions for other titles to try. You can also add books that you like and help to extend the data set.

There was one final set of sessions after that but in tne interests of getting home at a reasonable time I left early so as to catch the 18:30 train from King’s Cross. All in all I had a great day and it was well worth the journey – I just wish I could have gone to all the sessions. That would have required some kind of cloning device though.

As well as those that I met (I also met Lloyd Davis in the last session), I saw but didn’t really have chance to meet Cory Doctorow, Tom Coates, Simon Willison and Matt Webb, amongst others.

Yikes, it’s taken me over an hour to write this. If I ever buy a PDA (this one is borrowed) I’ll be buying a keyboard for it. I’m in the Lincolnshire wilderness between Grantham and Newark now, by the way.

Update: Back home, so I’ve added links to some of the stuff I was talking about.