What Big Data Needs to Do to Grow Up

We are in an Information Revolution — and have been for a while now. But it is entering a new stage. The arrival of the Internet of Things or the Industrial Internet is generating previously unimaginable quantities of data to measure, analyze and act on. These new data sources promise to transform our lives as much in the 21st century as the early stages of the Information Revolution reshaped the latter part of the 20th century. But for that to happen, we need to get much better at handling all that data we’re producing and collecting.

Consider the more than $44 billion projected by Gartner to be spent on big data in 2014. The vast majority of it — $37.4 billion — is going to IT services. Enterprise software only accounts for about a tenth.

The disproportionate spending on services is a sign of immaturity in how we manage data. In his seminal essay, “Why Software is Eating the World,” Marc Andreessen pointed out that for each new technology wave, the money eventually shifts to software. Software spending represents an industrialization and packaging of work that would otherwise happen manually, as one-off services, within each organization. As markets mature, more of the processes move to partners and other providers, so the industry leaders can spend their time and energy on high-value processes that contribute to their competitive differentiation. Software is part of a broader ecosystem that lets businesses focus on activities that are core rather than context. Core, in Geoffrey Moore’s definition, is what a business’s customers cannot get from anyone else; context is all the other stuff a business needs to get done to fulfill its commitments.

To understand what that path to maturity might look like for big data, it’s helpful to look at another, similar transformation. Data is the raw material that we attempt to turn into useful information. We can learn something from the manufacturers who turned raw materials into achievements as complex as automobiles.

The earliest automobile manufacturers were “vertically integrated,” which is to say they pretty much did everything themselves. Contrast that with today’s automobile manufacturers, which source parts from a global marketplace of independent suppliers. A manufacturer like Ford might have more than a thousand Tier 1 suppliers.

By calling on a rich ecosystem of industrialized products and services, automobile manufacturers can focus on the high-value, core activities that differentiate their products while driving down the total costs of production. This shift has led to a dramatic increase in automobile capabilities, without a corresponding increase in costs.

Over the years, the automobile industry has embraced numerous innovations to achieve this transformation. These include:

• Standardization: From parts to specifications and protocols, standardization is the essential first step in building a mature ecosystem.
• Quality testing and controls: Standardization also includes the concept of quality controls, testing protocols, and acceptance testing to enforce adherence to standards.
• Design for manufacturing/design for assembly: Integrating manufacturing and/or assembly processes into product design has reduced manufacturing and assembly costs at the source.

The shift from vertically integrated manufacturing (doing everything in-house) to integrated and collaborative design, manufacture, and assembly has enabled us to build larger, more advanced and complex goods than ever before. It has reshaped our communities and lives.

Can we make a similar shift with data?

When businesses try to manufacture real insight and value from raw data, most are like the early manufacturers, doing nearly everything in-house. Despite recent technological advances, the task of turning data into information is characterized by vast inefficiencies.

That’s why we see such a huge spending on services — we haven’t figured out how to automate and industrialize different parts of the data processes. We treat nearly every data-related task as a high-value, core process, for which we must spend on specialized services. It’s all core, no context.

Until we find ways to start treating some of the data tasks as contextual — to industrialize those processes — then we will be limited in what we can accomplish with data.

To industrialize these processes, we will need advances comparable to those that have occurred in manufacturing, including:

• Data standardization, particularly industry-specific standards and taxonomies.
• Data quality processes, such as advances in data integration/cleansing and quality control.
• Third-party data services (i.e., data clouds or “industrialized data services” in Accenture terminology) that enable data sharing and exchange at scale, between applications and organizations.
• Vertical data applications that understand all aspects of data relating to a specific task (e.g. IT Security).
• Data assembly, meaning dynamic assembly of raw data, processed data, and contextualized data.
• Industry cooperation in sharing the data that is “context” and common to all players.

With the ability to standardize data (akin to parts standardization in manufacturing) and use those standards in multiple applications, companies can begin to partner, outsource and collaborate on much of the work of involving in turning data into insight. The more that companies can share and repeat the context processes around different types of data, the more resources they have to invest in the higher-value, core data processes.

What Big Data Needs to Do to Grow Up
Mahesh S. Kumar