Will Small Data be the new Big Data?

23 January 2013

By Tom Robinson, David Lancefield and Carlo Gagliardi

Data is a vital resource in the new digital economy. But at present, the private sector is a largely closed data environment. A consumer’s personal data (their ‘small data’) is either stored offline, separated into digital silos or managed by businesses, unseen by consumers. This will change as open data principles are adopted. We believe new ways of using ‘small data’ could cause a fundamental shift in the economics of doing business.

There are many reasons to support this view - the network externalities of open data in the private sector are better understood than before [1] [2], the stock and flow of digital data is on an explosive growth path [3],  social pressure for control over the collection and use of personal data is growing [4] and government is actively supporting open data initiatives [5] . We are moving towards a more open data future. But what might it look like, and what are the implications?

We think that it’s likely that individuals will control “vaults” of personal data, stored securely, with flexibility over who has access to what, for how long, and with what privileges. These sources of data will contribute to a personal ‘Digital Identity’, incorporating additional information held by businesses as a result of new technologies and innovations - like e-health devices, in-car telematics and smart meters.

We believe that these ‘Digital Identities’ - combined with new consumer tools for interacting with multiple vendors at once - could transform the economics of doing business, for three main reasons.

First, the business environment will become a four party system. An open data future creates a potential market for fourth party organizations working to help and empower consumers. These fourth parties will assist consumers with managing their data and - with permission - act on their behalf to save them money and improve or provide new services.

Second, trust between business and consumers will become a highly valued asset, as consumers will only share their personal data with organisations they trust. The result will be the emergence of  ‘buying brands’ that cultivate relationships with businesses in adjacent sectors, using their trusted positions to provide complete service and product solutions centred around activities like health, home and leisure.

Third, the ‘Intention Economy’ described by Doc Searls [6] - where buyers rather than sellers are in charge - could become a reality. Consumers backed by personal data and the right digital infrastructure will be free to signal to the market their specific personal requirements and intent to buy. Suppliers can then bid directly to individuals to provide the precise product or service they desire. Consumers have wants and needs beyond the simple choices offered by business; an open data future could allow them to articulate these needs. In economic terms, we’ll see a more efficient market in which there are clearer signals, lower transaction costs and better targeting of products and services.

This open data future could be closer than we think. Personal Data Store services like MyDex, Personal.com and the Locker Project already exist. Services like Intently that facilitate intention signalling and reverse auctions are being actively developed and programming languages like KRL for personal data control and interaction are being adopted.

Pressure originating from government initiatives like ‘midata’ and action to promote consumer privacy and autonomy are likely to be key drivers moving us in this direction. High performing businesses should be able to tap in to new opportunities to take their brand, trust and product to new customers. And it’s an opportunity for new entrants to take on the fourth party role at the centre of this new digital order.

[1] BIS, “ Better Choices: Better Deals”, Dec 2012
[2] World Economic Forum, “Unlocking the value of Personal Data”, Oct 2012
[3] “Data production will be 44 times greater in 2020-“ – SCS, Data rEvolution
[4] Emergence and uptake of technologies like DoNotTrack and browser add-ons like Mozilla’s Collusion demonstrate that internet users are concerned by how much data businesses already hold on them, how it is gathered and what is done with it.
[5] Through the data.gov.uk and midata programmes
[6] Doc Searls, Berkman Center for Internet & Society, Harvard University, ‘The Intention Economy’


Tom Robinson  |  + 44(0)20 721 34302

David Lancefield  |  +44 (0)20 7213 2263

Carlo Gagliardi  |  +44 (0)20 7213 5659



Thank you for sharing this very insightful article. Online pricing is a fascinating area for economists and we are seeing a lot of new commercial practices emerging which are attracting the attention of antitrust authorities. The main questions are whether vertical restraints on online pricing such as minimum price (resulting in best price guarantees for final customers) actually reduce competition and therefore harm consumers (the OECD will discuss this very issue in a roundtable due to take place in February). Last November the OFT launched a call for information to improve its understanding of how the use of data about consumers is affecting online markets and its effect, if any, on pricing. They are going to investigate whether firms use data to modify prices offered to consumers, so that each consumer can end up paying a different price for the same product (based on the personal data and therefore their expected willingness to pay), if this is harmful, where the boundaries of acceptable company conduct lie, and if these practices are in breach of consumer protection legislation.

The article has interesting thoughts, but perhaps overly bullish on the likelihood.

Whilst industry may well want to move to an open data environment, it's clear that today such an approach would, for good reasons, quickly fall foul of the Data Protection Act. And, this is something the OFT amongst others are already investigating.

At the same time consumers are becoming quite savvy in this area and, as recent experiences with Facebook have shown, now started to realise that their data is valuable and so they are much less inclined to give it away with the tick of a 'I agree' box.

The route to the open world is likely to be much more complex that the map suggests.

The key skill in Big Data is in determining how analysis is applied to which elements of available Small Data. As the article suggests, this is about much more than price. Effective targeting of product suitability and availability could significantly reduce costs such as underwriting, advice, marketing and distribution.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated and will not appear until the author has approved them.