Insights from API data: The human factor
In the last post after our Webinar Visibility at the Edge - Deep Insights from your API, I talked about what it means to get 360 degrees of visibility from your API. This time, I'll talk about using that 360 view to get deep insights.
It is one thing to keep all the data, it is quite another to gain insights from it.
We've all seen examples of enterprises saying . . . "let's first store it, we'll worry about its uses later." Is that a good strategy? Is there a danger of collecting a lot of the data that is not very actionable or useful?
If you are generating 1000 TPS, you'll generate around 200 terabytes of data per year. We know that storage costs are going down but if you're storing 200 TBytes in a landfill fashion, you're looking at spending hundreds of thousands of dollars per month.
The basic question becomes: for the cost of collection and storage what value are you getting? You have to get incremental value for incremental investment. The only way to get this incremental value is to not "store and forget." Instead "store, analyze, synthesize, get insights" and then determine what else to store creating a tight loop between data, analysis, and business decisions.
Are there any easy on-ramps to doing this?
There are certain technological considerations that help get started.
Cloud model: Any enterprise looking at an API strategy should look at a cloud-based strategy, either with the help of people who know how to do it or with your own people. It is no secret that getting infrastructure set up is not easy in any sized enterprise. Enterprise IT infrastructure has a 6-12 month procurement cycle and you have to plan it carefully. But how to plan in this space? You cannot know what API traffic will be or what value you'll get from the data 6-12 months ahead of time.
In a cloud model, you can avoid showing up-front costs without showing the value.
Out-of-the-box API data. Consider the kind of out-of-the-box API data you want to expose to your business users, which can show instantaneous value. It may be primarily operational value, but still instantaneous data that shows value.
Contextual data. Don't boil the ocean. Start with critical data from the enterprise that is “close” to the API. Consider what’s your core context, and expand slightly around the API data, restricting the scope of what you take in. In other words, take a judicious and iteritive approach to incorporating context data, which should help avoid paying the cost of context without knowing the value it adds.
Is all that sufficient to affect the business?
You need data, you need analytics, and most importantly, for data and analytics to be relevant, you need to derive insights from these two.
Analytics is about running machine learning and computations on large amounts of data.
Insights is about how people can take the output of the machines and convert it into business value.
I believe that machines cannot replace human beings in deriving the insights that are critical to driving and pivoting your business. Deep insights come from API data, contextual data, machine analytics, and people augmenting the machine intelligence.
What are the organizational challenges to succeeding here?
It is important for those embarking on API journey to convince people to invest in APIs but also in analytics. I've made the case in the past that enterprises should make analytics and APIs peers in their strategy. This might mean an 80:20 effort for APIs:analytics. Making an investment in this culture change will pay dividends as you'll be able to show value far earlier than if you spend 100% on APIs up front and zero on analytics.
Another organizational challenge can be that not all API strategies are fully sanctioned arms of their enterprises. Many are nascent and need nurturing. People can still be wary of "opening up" back-end systems to let people interact with them.
Like many things in the corporate world, APIs and analytics need executive sponsorship. Of course, they have to deliver value, but without the executive sponsorship, the skeptics might kill it before it has a chance to prove itself.
Can an API provider do something in the design of an API to make it more "insight friendly?"
I assert it's not a good idea to try to design for insights or to think about insight as an end goal. In this world, you're either in the business of APIs, or in the business of running a business.
If you're in the former, you never want to optimize for insights. Instead, you want insights to help you optimize your APIs for developers, for building apps, for app end users, and so on.
if you're in the latter camp, where APIs are generating a large amount of traffic and transactions of your business, then insights into the business of APIs now become insights into your overall business. Again, you don't want to, and probably can't effectively design for insights. Rather let the insights optimize your business.
That said, you can definitely think ahead to collecting those out-of-the-box data that will give you a jump start.
Should you collect data yourself or gather it from systems already collecting it?
First work with people who have put APIs in front of data: figure out if you can use the data in this way. If not, and if you think you can derive a huge amount of value from the data, you want to be able to be able to slice and dice it so you might want a copy of it for yourself.