Insights from API data: 360 degrees of visibility
Thanks to all who participated in the Webinar: Visibility at the Edge - Deep Insights from your API. Check out the video and slides here.
In the last post, I talked about what we mean by visibilty at the edge of your business and the concept of 360 degrees of visibility. This time, I'll talk about the data that gets you the 360 degree view.
Q: What’s needed to make this 360 degree visibility happen?
There are a few things.
- First obviously, you need to be collecting data from the APIs.
- Second is collecting data that adds context to the API data.
- Thirdly, you need analytics that model and predict both the business and operational metrics.
Let's elaborate on data that adds context. Visibility into APIs is the best aperture into enterprises but just API data is insufficient - you need a mechanism to collect other "contextual" data.
Say a user is interacting with a Telco API. A cell phone number is flowing through that API. Other than that phone number, there's a huge amount of context that is not captured in that API:
- who's number is it?
- what do you know about that person?
- which app is that interaction happening through?
- Are they using only your APIs, or others; are they happy with your API or not? . . .
Knowing the mobile number and knowing the name, address, payment history of the person using that number, and so on, are two very different things.
Clearly visibility into APIs gives you valuable operations information, and even business impact information like the number of products purchased.
But wouldn't it be interesting and useful to know in what context that product was purchased, in which context the app was built, and so on? API data is great for operational and business visibility. Having contextual data greatly enriches the information for business visibility. Ideally you can collect enough contextual information around the APIs so that your analytics is no longer just looking at the bits and bytes of APIs.
Q: Does this combination of API data and context data dictate a need for a BIG DATA solution for companies?
Yes. Let's look at the two factors that I think are driving to a big data solution. The first is the volume and chattiness of API traffic, and the second is the new ways in which you need to interact with data.
Think about how your APIs are designed. Are they optimized first for back-end efficiency or for easy consumption?
They should be (and probably are) optimized for consumption with the understanding that the back-end optimization can come later. Optimizing for back-end systems first can lead you down a path where you build a system that's efficient from a back-end perspective, but not necessarily one that is attractive for developers to consume and adopt.
This necessity to optimize for consumption tends to lead to API traffic that is chatty and voluminous. Even a small sized enterprise that has 1000 TPS can easily end up with 200 terabytes of data every year. That's not huge, but it's also not small for most companies.
Just as important as the volume of data is how you interact with that data. Traditional data warehouses were built to answer questions you know you have and repeatedly.
The new big data solutions are about determining the questions to ask. In this scenario, it's not just the volume that is at issue - but the variety and nature of questions you need to ask. In an API, app, and customer-centric world you don't really know the patterns and questions at the outset.
Next time, we'll talk about analytics and how to get those insights.