It's made up of everything from the photos we take on our smartphones to the sensors that track our weather. And it’s given rise to a whole new sector - businesses that use data in innovative ways, whether it’s working out which route we should drive or helping companies understand why a customer is browsing but not buying.
"In 2000, it cost ten dollars per gigabyte to store data," says Mike Upchurch, founder of analytics software company Fuzzy Logix. "By 2005 it was 50 cents a gigabyte. Now it's three cents a gigabyte. So people are just storing astronomical amounts of data.
"But it’s all completely worthless unless you can do two things: ask it a question and take actions on the result. We worked with a major UK supermarket, for example, using weather patterns to help determine what should be on their shelves. This resulted in millions of dollars’ worth of savings in perishable food."
Conversion optimisation agency PRWD, uses data to help companies improve the percentage of website customers who complete the desired goal – whether that’s filling in a survey or buying something.
"There is a big emphasis on investing in data tools. But a lot of companies aren’t investing in the people who will actually interrogate that data and provide the stories for the business to drive an actionable output," says PRWD’s optimisation strategist Chris McCormick.
"For example, we might look at 'success data' – what’s going well for an ecommerce website. Perhaps the data shows that presenting reviews to customers makes them more likely to buy. So let’s make more of those reviews, and bring them to the forefront."
Enabling app creation
Data’s also behind the innovation success story of the last ten years - apps. Transport apps have seen massive growth – Mapway’s London Tube Map, for example, has been downloaded more than 15 million times. But that data needs work before app developers can use it, and that’s where companies like Transport API come in.
In 2010, the UK government began releasing transport data. But this isn’t always easy to use. It’s around 60 different data feeds, including everything from bus times to tube delays.
"We saw a business opportunity there," says Emer Coleman, TransportAPI’s business development director. "If you’re a developer and you want to make an app, you have to do a lot of work to integrate those feeds together and clean up any anomalies. We do that for them, and put it out as a single source."
And it’s not just big business which benefits from Transport API’s source: the company’s three levels of pricing include a freemium model. "You can access 30,000 data hits a month for free," says Emer. "That means people can experiment, do some research and development and some crucial concept testing – and they can do it for nothing."
Making things better
Big data specialists Mastodon C’s Witan project is also aiming to use data to drive innovation. It’s building an open source platform that enables cities to use their data better, to help them plan for their future education, employment, housing and energy needs.
The UN estimates that three billion people will live in cities by 2050. So planning cities for the future is a big challenge.
"Many cities don’t have data by city: they have it by ward, or by national level," says delivery manager Elisabeth Weise. "Another problem is that they might have the data but work in silos: population projects are done by one team, then employment projections are done by another."
The first tool, allowing boroughs to run their own population projections, has just been launched across the 33 London boroughs. "It’s really freeing up resources and giving much more analytic power to the boroughs," says Weise.
"There’s been a lot of enthusiasm. This is really been missing: there are a lot of niche products out there but they are very expensive. There’s no big buy-in: you can start with demography, as this underlies so much city planning. Cities are such drivers of growth - but they need to have the proper tools to be able to do it."