Since you are reading this, you have survived the first working day of 2013. Well done. With this momentous occasion arrives a flurry of predictions for the coming year.

Google self-driving car 2011Witness Guardian technology editor Charles Arthur make his predictions for the year: self driving cars, augmented reality, superfast mobile broadband, flexible screens, continuous self measurement and floating skateboards. Ok, I made the last one up. But notice that four of Arthur’s five involve Big Data in some way or other. Only flexible screens do not.

Elsewhere, there is a defense of the Big Data. Forbes’ Edd Dumbill fears a backlash against the term from those who dislike its imprecision. It may be woolly, but this mammoth will not be extinct in 2013, he forecasts.

At the New York Times, Steve Lohr says the next important step in the Big Data will come from machines. Sensors can collect data from the Internet of Things.   He cites the IDC report  The Digital Universe in 2020, which forecasts that machine-generated data will increase to 42 percent of all data by 2020, up from 11 percent in 2005. Where will all these data go? No doubt some will go into systems made by EMC, which sponsored the report.

Big Data itself is in the business of prediction. Part of the point of collecting vast quantities of data, some of which is unstructured, is to understand how, say, your sales relate to your ad positioning, the weather, or web traffic to form an view about what may happen in the future and how you may affect it.

JK-FT-smaller1-150x150With this in mind I read John Kay’s piece in yesterday’s Financial Times. Professor Kay is not an IT guru or Big Data evangelist. He’s an economist. He points out that, “most of the problems we face in business and finance are ill-defined and open-ended. We often do not really know the answers even after the event”.

He concludes that “…managers who know the future are more often dangerous fools than great visionaries”.

Over Christmas I told my nephew, born 1997, about the TV drama Space 1999.  It confidently predicted that interplanetary space travel would be routine before the end of the twentieth century. He found this hilarious. The truth is humans haven’t been to the moon since 1972, 25 years before he was born. cb-space1999_2

So predictions should come with a health warning, though I am pretty confident people will continue to make them. The only things that are certain are death and taxes, if Benjamin Franklin is to be believed.

Now, if I can just get my floating skateboard to file my tax return for me, I’ll be happy.


Bookended by Julia Bradbury and Sir Ranulph Fiennes, Big Data burst into Britain’s living rooms on Wednesday night.

Around five million viewers of BBC TV’s The One Show, a light entertainment and magazine programme, saw an item by FT columnist and popular economics author Tim Harford which promised Big Data was the biggest information revolution in the history of mankind.

Witness the moment here, 38 minutes in.

Anyone familiar with the Big Data story will know with the spiel. Multiple devices connect us to the internet generating mountains of data.

Oxford astrophysics Dr Chris Lintott pops up to tell us that the next generation of radio telescopes will generate 20 terabytes of data every second.

IBM master inventor Dr Andy Stanford-Clark shows how you can control your household appliances with a reassuringly old smartphone.

And then comes the warning. Who is holding all this data about you? What will they do with it? Who controls it? How do you keep safe?

Oxford professor of cyber security Sadie Creese then reassures us. “It is about understanding the risks to which we are exposed,” she says.

So when online, don’t leave your front door open, so to speak.

That prime time TV has covered Big Data at all is an event to celebrate. The advent of the data explosion needs an opportunity to gain some public understanding of its importance.

But Creese’s reassurance got me thinking. How do we understand risk? The problem calls to mind work by MIT professor John Sterman.

Last year he published a paper on the public understanding of the risks of climate change. “The strong scientific consensus on the detection, attribution, and risks of climate change stands in stark contrast to widespread confusion, complacency and denial among policymakers and the public,” he says.

He then evokes mental models, a term originated by British psychologist PN Johnson-Laird to describe the structural analogies of the world people use to make decision and solve problems.

Sterman says: “Our mental models lead to persistent errors and biases in complex dynamic systems like the climate and economy. Where the consequences of our actions spill out across space and time, our mental models have narrow boundaries and focus on the short term.”

This makes me wonder whether such a weakness in our mental models could hold back our understanding of the risks involved in Big Data.

It also makes me question whether Big Data itself will offer the benefits promised by the computer industry. As Sterman points out, the limitations of mental models exist in “educated elites” (data scientists?) as well as laypeople. Nor are they remedied by more information alone.

Whatever the answer, I have an inkling that reaping the opportunities presented by Big Data will take more than terabytes and new technology.

Those wishing to read Sterman’s paper in full should see the attached PDF.

Communicating Climate Change Risks in a Skeptical World (1)

Anyone familiar with both the BBC’s One Show and Big Data may be surprised to see them mentioned in the same sentence. The first is a family friendly low-brow human interest show residing comfortably in the slot between daytime TV and an evening’s entertainment and news. The latter is a big business trend which every pundit around promises will be taking over an IT strategy near you some time very soon.

So when I saw Andy Stanford-Clark tweet that he was about to pontificate on Big Data to an unsuspecting nation, hitherto blissfully unaware of the concept and tucking into fish fingers, peas and mash, I questioned whether it was for real. Indeed it is, he tweeted back.

twitter screengrab andy stanford clark

Andy is IBM’s master inventor. Really he is. He’s one of the driving forces behind the UK’s Eco Island, a project designed to make the Isle of Wight, which lies off England’s south coast, self sufficient in energy by 2020.

If ever there was a good application for good data science, big or otherwise, it is in bringing some intelligence to the energy grid. Patterns of consumption are dislocated from production in a way that wastes money and resources while needlessly adding to carbon emissions at a time when the planet can least afford it. Smart metering combined with better use of data from social media, the environment and economy will form part of the solution. We’ll see how that comes across on the One Show at 7pm tonight, God willing.

I’ll post an iPlayer link and a review tomorrow, should the item go ahead as planned.

On a personal note, my blogging hiatus is explained at least in part by my move from near here…

west norwood pic for blog

…to near here…

carbis bay for blog

This enterprise involved more than one 5am start, a tail-lift transit van and more than a thousand miles driving west and east and west again across the English country side. At the same time I was struggling to keep freelance clients happy while everything was in boxes. Tomorrow I’ll endeavor to construct an office out of various things acquired from Croydon Ikea three weeks ago. Wish me luck.

“In many domains of life, Slovic said, people form opinions and make choices that directly express their feelings and their basic tendency to approach or avoid without knowing that they are doing so.”

This quote from Daniel Kahneman’s book Thinking, fast and slow, cites psychologist Paul Slovic’s findings on how emotions affect human decision making. It is one example of the biases and flaws that affect the choices we make. I’ve been thinking about these weaknesses for the last few months. I’ve even spoken to a few people in the know: professors and doctors from the London School of Economics, MIT and Warwick Business School.

Now I fly over the Atlantic, from London to Washington, to attend a conference with delegations from companies that will, one way or other, be investing millions and millions of dollars, pounds or euro in the idea that businesses need to collect and store more data, and analyse it better, in order to improve the decisions they make. Big Data has become the totemic theme for 2012 in the IT world. Like ecommerce and web 2.0 before it, the Big Data jamboree allows software vendors, hardware companies, consultants and analysts to gather around the same theme to create demand for the next wave of products and services their customers will buy for fear of being left behind in the technology arms race.

I don’t want to pour cold water on it. I was writing about IT when the dotcom hype was deafening. Since then, Amazon has indeed revolutionised retail and bitten painful chunks from markets previously dominated by companies such as Waterstones and HMV. But I also remember and how it spectacularly failed. The technology can be there, but success depends on many other factors.

In this blog I will look at some of the impressive technologies and processes companies are beginning to use to try to make better decisions and work in new ways. But I also want to discuss the flaws in human decision making. How they are managed or mitigated will help determine the success or failure of Big Data and analytics which have this years captured the IT market.