Analyze API message content using custom analytics

Was this helpful?

In Build reports using API Analytics, you learned how to create different reports based on default data collected by the Apigee API Platform.

In this topic, you will populate Apigee Analytics with data specific to your API. This capability is beneficial when you need to analyze your data for patterns that are specific to your business. This topic uses Yahoo Weather API as a sample source of data.

Back to top

Creating policies

The Yahoo Weather API returns XML-formatted responses. For example, to get a weather forecast for Palo Alto:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=12797282

The content within the response that you are interested in analyzing is probably the specific weather report data and forecast:

<yweather:location city="Palo Alto" region="CA"   country="United States"/>
<yweather:units temperature="F" distance="mi" pressure="in" speed="mph"/>
<yweather:wind chill="48"   direction="0"   speed="6" />
<yweather:atmosphere humidity="82"  visibility=""  pressure="30.19"  rising="0" />
<yweather:astronomy sunrise="6:52 am"   sunset="4:54 pm"/>

<yweather:forecast day="Mon" date="19 Nov 2012" low="48" high="66" text="Partly Cloudy" code="30" />
<yweather:forecast day="Tue" date="20 Nov 2012" low="51" high="66" text="Partly Cloudy" code="30" />

For another location:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2520841
<yweather:location city="Williamsburg" region="NY"   country="United States"/>
<yweather:units temperature="F" distance="mi" pressure="in" speed="mph"/>
<yweather:wind chill="37"   direction="0"   speed="6" />
<yweather:atmosphere humidity="73"  visibility="10"  pressure="30.49"  rising="2" />
<yweather:astronomy sunrise="6:48 am"   sunset="4:32 pm"/>
. . . 
<yweather:forecast day="Mon" date="19 Nov 2012" low="38" high="50" text="Partly Cloudy" code="30" />
<yweather:forecast day="Tue" date="20 Nov 2012" low="40" high="50" text="Partly Cloudy" code="30" />

To extract the information of interest, you use XPath expressions. For example to extract the value of the city:

/rss/channel/yweather:location/@city

(There are many tools available online that you can use to construct XPath expressions for your XML documents.)

ExtractVariables can be used to extract variable message content using JSONPath (for JSON-formatted messages), URI path, HTTP headers, query parameters, and form parameters. Any variable that can be populated using the ExtractVariables policy is available for custom analytics. See Extract message content using ExtractVariables for instructions on parsing message content and populating variables. 

To create custom reports for XML-formatted messges, you define such XPath expressions in an ExtractVariables policy, and then use them to populate custom variables. Custom variables are like the predefined variables documented in the  Variables reference, except that, for custom variables, the prefix name and variable name are user-defined. The result is a new type of variable with a name of the format prefix.variable. In this example, four custom variables are defined:

  • weather.location
  • weather.condition
  • weather.forecast_today
  • weather.forecast_tomorrow

NOTE: The only special character permitted in custom variable names is underscore, '_'.

Under apiproxy/policies, create a file called ParseWeatherReport.xml with the following content:

<ExtractVariables name="ParseWeatherReport">
  <!-- Parse the XML weather report using XPath. -->
  <VariablePrefix>weather</VariablePrefix>
  <XMLPayload>
	<Namespaces>
		<Namespace prefix="yweather">http://xml.weather.yahoo.com/ns/rss/1.0</Namespace>
	</Namespaces>
    <Variable name="location" type="string">
      	<XPath>/rss/channel/yweather:location/@city</XPath>
    </Variable>
    <Variable name="condition" type="string">
      	<XPath>/rss/channel/item/yweather:condition/@text</XPath>
    </Variable>
    <Variable name="forecast_today" type="string">
      	<XPath>/rss/channel/item/yweather:forecast[1]/@text</XPath>
    </Variable>
	<Variable name="forecast_tomorrow" type="string">
      	<XPath>/rss/channel/item/yweather:forecast[2]/@text</XPath>
    </Variable>
  </XMLPayload>
</ExtractVariables>

These variables will be populated each time a response message is returned to an app by your API proxy. Once the variables are populated, the values are forwarded to Apigee Analytics by defining a 'statistic' for each variable. 

You define statistics in a StatisticsCollector policy, one statistic per custom variable.

Under apiproxy/policies, create a file called AnalyzeWeatherReport.xml with the following content:

<StatisticsCollector name="AnalyzeWeatherReport">
  <Statistics>
    <Statistic name="location" ref="weather.location" type="string">Earth</Statistic>
    <Statistic name="condition" ref="weather.condition" type="string">Sunny</Statistic>
    <Statistic name="forecast_today" ref="weather.forecast_today" type="string">Rainy</Statistic>
    <Statistic name="forecast_tomorrow" ref="weather.forecast_tomorrow" type="string">Balmy</Statistic>
  </Statistics>
</StatisticsCollector>
Back to top

Attaching policies to the ProxyEndpoint response flow

Policies attached to the ProxyEndpoint response flow are enforced on outbound response messages, before the response is returned to the calling client app.

The example below parses response, populates variables using the ParseWeatherReport policy, and then forwards those values to Apigee Analytics using the AnalyzeWeatherReport policy.

<ProxyEndpoint name="default">
<Flows>
    <Flow name="default">
    <Response>
      <Step><Name>ParseWeatherReport</Name></Step>
      <Step><Name>AnalyzeWeatherReport</Name></Step>
    </Response>
  </Flow>
</Flows>
    <HTTPProxyConnection>
        <!-- Base path used to route inbound requests to this API proxy -->
        <BasePath>/weather</BasePath>
        <!-- The named virtual host that defines the base URL for requests to this proxy -->
        <VirtualHost>default</VirtualHost>
    </HTTPProxyConnection>
  <RouteRule name="default">
    <!-- Connects the proxy to the target defined under /targets -->
    <TargetEndpoint>default</TargetEndpoint>
  </RouteRule>
</ProxyEndpoint>
Back to top

Importing and deploying the API proxy

$ python tools/deploy.py -n weatherapi -u myname:mypass -o myorg -e test -p / -d simpleProxy
Writing ./simpleProxy/apiproxy.zip to ./apiproxy.zip
Writing ./simpleProxy/apiproxy/weatherapi.xml to apiproxy/weatherapi.xml
Writing ./simpleProxy/apiproxy/policies/AnalyzeWeatherReport.xml to apiproxy/policies/AnalyzeWeatherReport.xml
Writing ./simpleProxy/apiproxy/policies/ParseWeatherReport.xml to apiproxy/policies/ParseWeatherReport.xml
Writing ./simpleProxy/apiproxy/proxies/default.xml to apiproxy/proxies/default.xml
Writing ./simpleProxy/apiproxy/targets/default.xml to apiproxy/targets/default.xml
Imported new proxy version 3
Environment: test
  Revision: 4 BasePath = /
  State: deployed
Back to top

Populating Analytics data for custom variables

Before reports can be be built against custom data, requests must be submitted to an API in order to populate analytics data for the custom variables defined in the policies.

For example:

Palo Alto:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=12797282

Shanghai:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2151849

London:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=44418

Wiliamsburg:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2520841
Back to top

Generating a custom report

Now generate a custom report based on the data that Apigee records from calls to the the Weather API. The report you generate here provides the number of requests submitted to the API by location. Here we generate the report using the Apigee API Platform API. You can generate the same custom report using the UI at enterprise.apigee.com.

Note that the timeRange parameter must be modified to include the time interval when data was collected.

In the example request below, the custom dimension is called location. This request builds a custom report for locations based on the sum of message counts submitted for each location.

$ curl -u myname:mypass https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?'select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day'
{
  "environments" : [ {
    "dimensions" : [ {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "4.0"
        } ]
      } ],
      "name" : "London"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "19.0"
        } ]
      } ],
      "name" : "Palo Alto"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "2.0"
        } ]
      } ],
      "name" : "Shanghai"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "14.0"
        } ]
      } ],
      "name" : "Williamsburg"
    } ],
    "name" : "test"
  } ],
  "metaData" : {
    "samplingRate" : "100"
  }
}

In some cases, there may be a large number of results. It may be useful to filter the list to report the top 2 locations by message volume. This is done by adding the topk query parameter and providing an integer value for the number to filter:

$ curl -u myname:mypass https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?'select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day&sortby=sum(message_count)&topk=2"

Results can also be filtered by specifying the values of the dimensions of interest. In the example below, the report is filtered by results for London and Shanghai :

$ curl -u myname:mypass https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?"select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day&filter=(location%20in%20'London','Shanghai')"
{
  "environments" : [ {
    "dimensions" : [ {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "4.0"
        } ]
      } ],
      "name" : "London"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "2.0"
        } ]
      } ],
      "name" : "Shanghai"
    } ],
    "name" : "test"
  } ],
  "metaData" : {
    "samplingRate" : "100"
  }
}

Post questions to the Apigee Developer Forum.

Back to API Platform Developer Guide.

 

Back to top