11436 SSO

Sharper Insights with Predictive Analytics and Unstructured Data

Jul 09, 2014

Businesses have traditionally relied exclusively on structured data—the type that fits neatly in rows and columns—to make business decisions. For example, in a traditional database, a company might look at  the first 12 cells of row 21 to calculate the average customer order value for each of the past 12 months. This analysis could provide valuable insights into order size trends.

But “unstructured data”—the kind that doesn’t fit within a traditional relational or row-column database—can be an equally important source of business insight. The catch, however, is that it’s far harder to plumb.

That’s because unstructured data is, by definition, unruly and difficult to organize. The term usually describes text: emails, documents, tweets, Facebook posts, notes, survey responses; it can also describe binary content including audio, video, and images. While the specific definitions might vary, there’s general agreement on one important statistic: unstructured data makes up 80-90% of all data in an organization, and it’s growing much faster than its structured counterpart.

Unfortunately, traditional computer programs weren’t built to handle this kind of data. Take the tweet, “The Godfather is an awesome movie!” It’s a simple sentence that’s easy for a person to read and analyze and know that it refers to a particular Francis Ford Coppola film. We humans can hypothesize that the person tweeting probably likes action and mafia movies, Marlon Brando, Al Pacino, and so on. A traditional computer program can’t.

In an enterprise context, when firms apply predictive analytics to improve business outcomes, they often don’t analyze unstructured data. This omission typically results from a lack of in-house expertise or access to systems or solutions that include text or sentiment analysis technology. When businesses do evaluate unstructured data, the social channels—Facebook and Twitter, for example—usually receive the most attention.

Social networking data can be useful, but they are also very noisy. Enterprises often overlook the inclusion of internal sources (emails, service call notes, and documents, for example) of unstructured data, and miss out on potentially rich and useful insights about customer behavior, concerns, intent, and needs. As a result, their analyses or predictions are less precise than they could have been, and often, they don’t know how many potential revenue opportunities they are leaving on the table.

Agent notes to the rescue

Let’s look at healthcare insurers (payers). For these enterprises, the call center represents a key avenue to manage customer relationships. The complexity and intricacies of health plans often require human cognition to resolve customer issues. In this scenario, the notes that call center agents record after completing a customer service call can provide valuable signals about that member’s needs and concerns, including predictors of future dissatisfaction and the likelihood of a member to churn.

This was exactly Apigee’s experience when we worked with a healthcare insurer with a large Medicare Advantage business that aimed to predict which members were likely to complain to Medicare (see our earlier blog post on The Business Value of Reducing Medicare Complaints with Big Data Predictive Analytics).

This customer, who used Insights, Apigee’s big data predictive analytics platform, included in the predictive analytics process the notes that their customer service representatives took after each call. They found that when call center agent notes were analyzed in addition to member profile and claims data in the predictive analytics process, the prediction accuracy was nearly twice that of when the notes weren’t included. They provided rich insights into likely customer dissatisfaction, and therefore directly impacted the business' bottom line.

Call center agent notes helped double prediction accuracy  for one Apigee customer

Got unstructured data?

As you increase your use of big data and predictive analytics, unstructured data deserves attention, especially internal sources that contain valuable clues about product usage and how customers perceive their relationship with you. Apigee Insights includes unstructured data processors, a collection of inference engines that extract powerful signals from text. Using Apigee Insights, businesses can dramatically improve the prediction accuracy by analyzing both structured and unstructured data.

To learn more about predictive analytics and unstructured data, be sure to register for our webcast “Predictive Analytics on Big Data: DIY or Buy?,” coming up on Thursday, July 10.

image: Peter Castleton/Flickr

Scaling Microservices