Archiving the web: a record of history

 Anyone visiting the Amazon website last week would have been struck by a letter on the home page from Jeff Bezos informing UK shoppers that they could now purchase Amazon's e-reader, the Kindle, for shipping to the UK.

Cue a flurry of commentary about the death of books and the demise of culture.

Inevitably we are moving towards a world of digital content. Blu-Ray may well be the last ever 'physical' entertainment media format. But as we move away from information being stored on physical formats (CDs/DVDs/paper/books) to data stored in the internet how, in the future, are we going to look back and understand the state of knowledge at a particular time in history?

The written word, stored on parchment and paper and filling libraries and archives has always provided historians with an evolving and largely permanent record of human history. Digital content, in contrast, is more fluid and more concerned with the state of information at the present time. New content trumps old content. Older information on the web is largely constrained to out of date blogs and websites rather than a systematic approach to archiving.

To put this into context the journal Science has found that 13% of Internet references in scholarly articles were inactive after only 27 months.

However, since 1996 a non-profit organisation called the Internet Archive has been archiving digital content with the goal of building an 'internet library'. It has archived over 150 billion web pages and hundreds of thousands of moving images, live music files, audio and document texts. Its "Way Back Machine" is a useful tool that provides a snapshot of selected major sites over the years. Here is Apple.com back in in 1997.

The Internet Archive has also partnered with eleven National Libraries (Australia, Canada, Denmark, Finland, France, Iceland, Italy, Norway, Sweden, The British Library & The US Library of Congress) to create the The International Internet Preservation Consortium (IIPC). The mission of the IIPC is to acquire, preserve and make accessible knowledge and information from the Internet for future generations.

However, no single project can ever hope to archive the entire web. The approach to preserving digital content may not be of major concern today because the web is simply too new. However as the internet becomes the de facto reference point for all human knowledge: it will become of critical importance.

Challenges for the ad-funded business model

 The 'ad funded' model has taken a bit of a beating this year. The big question is whether it can continue in a recession-hit media world.

Last month Blyk (the ad-funded mobile operator) withdrew their consumer offer and just this week Joost (the online video company) announced that market conditions meant that it too was moving away from a consumer focussed ad-funded model. These were both businesses heralded as at the frontier of the new media world and examples of how an ad-funded business model could be used to deliver new services to consumers on new platforms.

But that model is under severe threat.

Recessionary-driven declines in media spends have driven massive losses and cuts at ITV and focussed attention on the long-term viability of Channel 4. Newpapers too are suffering. And, to put this into a digital context, the first quarter of this year saw a 5% drop decline in online advertising.

So a new model has to be found as only the largest traffic sites can continue a purely ad-funded approach.

Surely it's got to focus on subscriptions. However, the print media industry let the genie out of the bottle with free content - and I think it's hard to see consumers willingness to pay for something that they have considered free for some time - unless there is an industry-wide approach. But who will blink first?

There are some interesting models out there. Spotify the online music service uses a blend of ad-based revenues and a subscription model. However, many people have asked how Spotify can actually make a profit and there is a good interview with Daniel Ek (the Spotify CEO) on TechRadar - although Ek does not directly answer the question as to how its business model really works. The core question for Spotify and others is what % of users will pay the subscription model when the free model is pretty good. The ads aren't really intrusive enough to make me switch to a premium version.

An alternative option is using micropayments for content. Here users buy premium content in return for just a couple of pence. However will users really want to make that 'purchase decision' every time they want to access/read a piece of content? And how will that work practically? The user would already have had to buy-in to the site producing that content and lodged some form of payment (in the same way that works for iTunes). The micropayment model has been mooted for over a decade - but is still not gained much traction. Perhaps the media downturn will see it dusted it off and reconsidered.

Breaking news

Seth Godin recently predicted that by 2012 "there will be no significant newspapers printed on newsprint in the US".

Now whilst that may be sensationalistic in terms of timescale - there is no doubt that the newspaper industry is in trouble. In the six months to March of this year the decline in US newspaper weekday circulation almost doubled (Source: Bloomberg) and a number of historic US newspaper titles have already hit financial trouble.

Yes the recession has hit media spend - but the core issue is that more & more consumers are getting their news from the internet.

But that's not really new news. What is new is how the internet is increasingly becoming the source for breaking news.

Last Thursday night I was doing some things around the house with BBC News 24 on in the background. But rather than seeing a journalist on the screen - there was a caption with an image of the TMZ.com site leading with the news of Michael Jackson's death. For the next couple of hours the BBC's news was simply reporting on the news from TMZ and the LA Times. Mainstream news media reporting breaking internet news.

In September last year, Robert Peston chose his blog to announce the proposed Lloyds/HBOS merger before breaking the news on BBC.

More recently, with reporting restrictions in place, the news from Iran has been lead by blogs, Tweets and videos from people on the streets.

In a 24-hour global news cycle no single news organisation is going to have the coverage to capture every breaking news story. So thanks to Twitter, blogs & the camera-phone the power of the consumer-journalist is massive. News can be broken by anywhere, anywhere, at any time.

Social media is increasingly driving the news agenda.

The challenge for the established media therefore becomes two-fold:

• How do they drive revenue from their online properties,
  
• How do they continue to verify the myriad of consumer-journalist sources?

Are decision engines the Emperor's new clothes?

It's been a big month for 'decision engines'.

First we had the build-up and launch of Wolfram-Alpha as mentioned a few weeks ago and now we have the launch of Microsoft's own decision engine called 'Bing'. Bing went live last week, a couple of days early. The earlier-than-planned-launch is a strong indication of the effort Microsoft has put behind it. Let's not forget how important Bing is to Microsoft. The majority of Microsoft's recent challenges to Google's search and online advertising dominance have failed to make an impact on their market share. Bing is the successor to Microsoft's Live Search. But what's different? And what exactly are 'decision engines', a term so new that there isn't a Wikipedia entry to define them yet?

Decision Engines are being positioned as the evolution of search. Whereas search engines 'simply' (and I use the word advisedly) return the closest match to a given search term, decision engines move us closer towards the 'semantic web' and artificial intelligence by using contextual information, together with the results of previous user's searches, to provide more accurate and more helpful results. Well that's the theory anyway.

But do either Wolfram Alpha or Bing live up to this?

Wolfram Alpha has hit the 100 million query mark, but getting an answer to your question is still quite hit and miss. It's actually quite difficult to think of a useful query that Wolfram Alpha has an answer for. Many questions still get a "Wolfram Alpha isn't sure what to do with your input".

Bing on the other hand might well be trying to position itself as a 'decision engine' but it's not as revolutionary as its pre-launch billing. It's more of an evolution of search, attempting to improve on Google's offering.

And there are some nice features: the image search allows you to filter images by size, style, layout etc. A neat feature. The video shortcuts allows you to play the video by just rolling over the image thumbnail in the search. But Bing has been criticised for delivering poorer results than Google. For a neat way of comparing Google and Bing, Blackdog have created a split screen site to search both sites at the same time.

Will it be enough to make inroads into Google's market share? Only time will tell. But from first impressions it's just not significantly different or better than Google to shift consumer behaviour.

There is a danger that the term Decision Engine is being used by Microsoft in order to create some new news: as an attempt to be viewed as creating something new rather than just going head-to-head with Google.  But that's exactly what Microsoft is doing.

The Wolfram Alpha product may not be quite there yet, but it takes us much more into the area of decision engines and its evolution will be much more interesting and important to watch.