Kicking around the concept of sharding

I was really enjoyed tonight’s (27th july) Supermonday session, the session was split into just one speaker session, followed by a group discussion.

The speaker was John Colquhaun, a research assistant and graduate of computer science from Newcastle uni that has been working on a some software to help doctors communicate risk to patients with cardio-vascular problems. This involved firstly calculating risks based on some existing equations, integrating this into patient records, then expressing the result in smiley faces.

While i found this very interesting, and got the concept, the graphic designer in me simply wasn’t impressed with the graphical representations, the random placement of the smilies and the surrounding white space were very distracting. I think a nice clean representation of a heart with the appropriate percentage of risk ghosted out or something similar would have made it far clearer, far more compelling representation to get the message across, but what do i know!

Unfortunately the IP cameras we brought took a tumble, so there won’t be a speakers video this month.

The second half of the evening was the bit that i actually enjoyed the most, a group discussion, based the question “Do we need to encrypt our data or should we just shard it and scatter it to the four corners of the world? ” Before tonight i’d never heard of ‘sharding’ data, and in fact like many other foolishly assumed it was a typo and should have read ‘sharing’. While it was intended not to have a mediator, Ross introduced the subject and explained a bit about it. Apparently ‘sharding’ like the name suggest involved splitting a file up into a number of shards, each of which is then encrypted and stored on a separate file server in a separate datacentre, so that if any one server is compromised the stolen data would be worthless. Effectively this solution could a cheap, secure and resilient way to store data. The round the table discussion format worked well with various people contributing with their perspective on security , performance,legality, effectiveness and so on.

Ross, who runs an email spam filtering service explained that he received about a million small files a day that he had find a way to store securely but cost effectively store these files and that this was the solution we’d come up with. Kicking the concept around as a group we came up with lots of interesting ideas that we’d probably never have come up with otherwise.

I found the whole idea of thinking about something very new to be quite rewarding, and had plenty to consider on the train home.

Mike Parker of orangebus and Mike Lowenstein of gavurin both announced that they were hiring, so if your in the market for a design, development or sales position give them a shout.

Related links: Andrew Waite’s review of July’s Supermonday meeting on Infosanity.

This entry was posted in event, featured, hosting, internet, media, security, services, supermonday, web. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>