11436 SSO

Insights from API data: 360 degrees of visibility

anant
Mar 07, 2012

Thanks to all who participated in the Webinar: Visibility at the Edge - Deep Insights from your API. Check out the video and slides here.

In the last post, I talked about what we mean by visibilty at the edge of your business and the concept of 360 degrees of visibility. This time, I'll talk about the data that gets you the 360 degree view.

What do you mean by visibility at the edge?
 
in the past, enterprises could focus on controlling their interactions with customers through physical presence or websites. Now people are interacting more and more with core businesses through apps. 
Although the interaction has moved farther away and businesses are no longer in control of the interaction in the same way, customers are more important to the business than ever before. More data is coming in from developers and end users through apps and APIs.
 
Clearly, to get such visibility, we will collect lots of data. Before we get into the data part, can you give examples of what such visibility might tell us?
 
Think about the number of different constituents around API interactions (even non-API interactions). There are the people transacting with your business, developers building apps, teams in enterprises responsible for the building and the success of APIs, teams responsible for the operations of APIs, and business users interested in the top and bottom lines. 
 
<<image>>
There are also lots of interactions through various channels with different systems, including billing, catalog, procurement, order management, and so on. The biggest challenge is how to collect the disparate perspectives you get through these channels and generate a 360 degree sense of it. If you believe in the apps economy, you believe this will all get standardized around the APIs "interface."
 
Your APIs and the management of APIs gives you visibility into a larger set of interactions than any one back-end system can. If you harness that visibility, regardless whether you run operations for the back-end systems, your business is to make the APIs successful, or you are a business person focused on the top and bottom line, you can get a lot of insights into your business. For example: 
 
- How well is the API being adopted?
- What is the response time per request?
- Does operations need to allocate more resources at peak times?
- What impact is an API having on my bottom line?
 
But there's something even more transformative one can get from API data. Instead of thinking only about what the enterprise learns from API data, think about what developers and the app end users can learn from the API data. For example, if you are a developer, you'd like to know what impact you're having on the business whose API you are using to build your app. 
 
Visibility into the APIs gives insights to all of the constituents, therefore providing a 360 and actionable view.
 
 
What’s needed to make this 360 degree visibility happen?
 
There are a few things. First obviously, you need to be collecting data from the APIs. 
Second is collecting data that adds context to the API data. 
Thirdly, you need analytics that model and predict both the business and operational metrics.
 
Let's elaborate on data that adds context.
 
<<360 image>
 
Visibility into APIs is the best aperture into enterprises but just API data is insufficient - you need a mechanism to collect other "contextual" data. 
 
Say a user is interacting with a Telco API. A cell phone number is flowing through that API. Other than that phone number, there's a huge amount of context that is not captured in that API:
- who's number is it?
- what do you know about that person? 
- which app is that interaction happening through?
- Are they using only your APIs, or others; are they happy with your API or not? . . .
Knowing the mobile number and knowing the name, address, payment history of the person using that number, and so on, are two very different things.
 
Clearly visibility into APIs is the best aperture into your enterprise. You get valuable operations information, and even business impact information like the number of products purchased. 
 
But wouldn't it be interesting to know in what context that product was purchased; in which context the app was built, and so on? Ideally you can collect enough contextual information around the APIs so that your analytics is no longer just looking at bits and bytes of APIs. 
 
In summary - API data is great for operational and business visibility. Having contextual data greatly enriches the information for business visibility.
 
Does this combination of API data and context data lead us to a need for a BIG DATA solution for companies?
 
Yes. But let's talk about some examples of what that means. 
 
Let's just think about how your APIs are designed. Are they optimized first for back-end efficiency or for easy consumption?
They should be and probably are optimized for consumption with the understanding that the back-end optimization can come later. Optimizing for back-end systems first can lead you down a path where you build a system that's efficient from a back-end perspective, but not necessarily one that is attractive for developers to consume and adopt. 
 
This necessity to optimize for consumption tends to lead to API traffic that is chatty and voluminous. Even a small sized enterprise that has 1000 TPS can easily end up with 200 terabytes of data every year. That's not huge, but it's also very small. So just thinking about the volume and chattiness of traffic, you are definitely driving towards a big data solution. 
 
Secondly, it's also not so much about the data volume but how you interact with data.
 
Traditional data warehouses have been built to answer questions you know you have and repeatedly.
 
The new big data solutions are about determining the questions to ask. In this scenario, it's not just the volume that is at issue - but the variety of questions you need to ask. In an API, app, and customer-centric world you don't really know the patterns and questions at the outset.
 
In summary, given volume and chattiness of API traffic, and the new ways in which you need to interact with that data, I believe that we are driving to a bid data solution. 
 

Next time, we'll talk about gaining insights from that big data and the analytics on that data

In this post, I'll write out some of concepts and ideas we explored on the Webinar.

Q: What’s needed to make this 360 degree visibility happen?

There are a few things.

- First obviously, you need to be collecting data from the APIs.
- Second is collecting data that adds context to the API data
- Thirdly, you need analytics that model and predict both the business and operational metrics.

Let's elaborate on data that adds context. Visibility into APIs is the best aperture into enterprises but just API data is insufficient - you need a mechanism to collect other "contextual" data. 

Say a user is interacting with a Telco API. A cell phone number is flowing through that API. Other than that phone number, there's a huge amount of context that is not captured in that API:

- who's number is it?
- what do you know about that person?
- which app is that interaction happening through?
- Are they using only your APIs, or others; are they happy with your API or not? . . .

Knowing the mobile number and knowing the name, address, payment history of the person using that number, and so on, are two very different things.

Clearly visibility into APIs gives you valuable operations information, and even business impact information like the number of products purchased. 

But wouldn't it be interesting and useful to know in what context that product was purchased, in which context the app was built, and so on? API data is great for operational and business visibility. Having contextual data greatly enriches the information for business visibility. Ideally you can collect enough contextual information around the APIs so that your analytics is no longer just looking at the bits and bytes of APIs.

Q: Does this combination of API data and context data dictate a need for a BIG DATA solution for companies?

Yes. Let's look at the two factors that I think are driving to a big data solution. The first is the volume and chattiness of API traffic, and the second is the new ways in which you need to interact with data. 

Think about how your APIs are designed. Are they optimized first for back-end efficiency or for easy consumption?

They should be (and probably are) optimized for consumption with the understanding that the back-end optimization can come later. Optimizing for back-end systems first can lead you down a path where you build a system that's efficient from a back-end perspective, but not necessarily one that is attractive for developers to consume and adopt. 

This necessity to optimize for consumption tends to lead to API traffic that is chatty and voluminous. Even a small sized enterprise that has 1000 TPS can easily end up with 200 terabytes of data every year. That's not huge, but it's also not small for most companies. 

Just as important as the volume of data is how you interact with that data. Traditional data warehouses were built to answer questions you know you have and repeatedly.

The new big data solutions are about determining the questions to ask. In this scenario, it's not just the volume that is at issue - but the variety and nature of questions you need to ask. In an API, app, and customer-centric world you don't really know the patterns and questions at the outset.

Next time, we'll talk about analytics and how to get those insights.

API Management Decision-Making Kit

Next Steps

 
 

Resources Gallery

News