Was this helpful?

In Use the analytics API to measure API program performance, you learned how to use the RESTful API exposed by Analytics Services to get statistics on a variety of entities monitored by Apigee Edge.

In this topic, you will learn how to use the Analytics Services API combined with Policies to analyze data that is unique to your app and API traffic. Most of the data that is key to your business is found in the payload content moving back and forth from apps to your backend services. aUsing Analytics Services, you can define custom dimensions that Apigee Edge uses to collect, analyze, and provide reports on that data.

This topic demonstrates the usage of custom analytics against the Yahoo Weather API. The goal of the exercise is to create a custom dimension called location that enables you to collect statistics on the number of requests received for weather reports for different locations. Once you have defined the custom dimension called location, you can use the capabilities of the Analytics Services API to retrieve and filter statistics that the Apigee Edge collects. The custom dimension location will also be made available in the Management UI, enabling you to create reports for this dimension using the API Edge's UI-based tools.

Parsing payloads using Policies

The Yahoo Weather API returns XML-formatted responses. You request a weather report for particular location by providing a WOEID, which stand for "where on Earth ID". The WEOID for Palo Alto, CA is 12797282. To get a weather forecast for Palo Alto, you submit the following request to the Yahoo Weather API:

$ curl http://weather.yahooapis.com/forecastrss?w=12797282

To collect custom analytics, you need to call the API using an API proxy. The PAI proxy will inspect the request and response messages from your app to the Yahoo API. You are provided with a pre-configured API proxy in the test environment of your organization. The API proxy is called weatherapi. You can invoke that API proxy to obtain a proxied response from the the Yahoo Weather API.

If you don't have an account on Apigee Edge, see API reference getting started.

You can invoke the proxied version of the Yahoo Weather API using the following command. As usual, substitute your organization name on Apigee Edge for the variable {org_name}.

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=12797282

The interesting part of the response message, the weather report and forecast, is shown below. (Note that the response, except for specifics such as timestamps, is exactly the same as the response from the direct API call above.)

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<rss version="2.0" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
 <channel>
  <item>
   <!-- Some XML excluded here. . . for brevity -->
   <yweather:forecast day="Wed" date="1 Oct 2013" low="49" high="72" text="Sunny" code="30" />
   <yweather:forecast day="Thu" date="2 Oct 2013" low="48" high="73" text="Sunny" code="30" />
   <yweather:forecast day="Fri" date="3 Oct 2013" low="47" high="72" text="Sunny" code="32" />
   <yweather:forecast day="Sat" date="4 Oct 2013" low="48" high="75" text="Sunny" code="32" />
   <yweather:forecast day="Sun" date="5 Oct 2013" low="49" high="77" text="Sunny" code="32" />
   <guid isPermaLink="false">USCA1093_2013_1_13_7_00_PDT</guid>
   </item>
 </channel>
</rss>

To see the same data for a different location, submit the same request with a different WOEID, 2520841, for Williamsburg.

	
$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2520841
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<rss version="2.0" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#">
 <channel>
  <item>
   <!-- Some XML excluded here. . . for brevity -->
   <yweather:location city="Williamsburg" region="NY"   country="United States"/>
   <yweather:units temperature="F" distance="mi" pressure="in" speed="mph"/>
   <yweather:wind chill="37"   direction="0"   speed="6" />
   <yweather:atmosphere humidity="73"  visibility="10"  pressure="30.49"  rising="2" />
   <yweather:astronomy sunrise="6:48 am"   sunset="4:32 pm"/>
   <yweather:forecast day="Mon" date="1 Oct 2013" low="38" high="50" text="Cloudy" code="1" />
   <yweather:forecast day="Tue" date="2 Oct 2013" low="40" high="50" text="Cloudy" code="1" />
   <guid isPermaLink="false">USCA1093_2013_1_13_7_00_PDT</guid>
   </item>
 </channel>
</rss>

Both of these message contain potentially valuable information. However, Apigee Edge doesn't yet 'know' how to feed this message content into Analytics Services for processing. To enable this, Edge provides Policy type that parses message payloads and feeds the content into Analytics Services.

The Policy type that enables this functionality is called ExtractVariables. The ExtractVariables Policy type enables you to configure JSONPath or XPath expressions that the Apigee Edge used to extract pieces of data from message content.

There are many tools available online that you can use to construct XPath expressions for your XML documents. There also many tools available for JSONPath.

See Extract message content using ExtractVariables.

To extract the information of interest from the weather report, you use an XPath expression. For example, to extract the value of the city, the XPath expression is:

/rss/channel/yweather:location/@city

Note how this XPath expression reflects the structure of the XML nodes. also, note the prefixyweather is defined by a namespace:

xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0

To enable the XML message to be parsed properly, you use both the XPath and the namespace definition in the policy.

You will include this XPath expression in the ExtractVariables Policy. Once the XPath has been evaluated, the Policy needs a place to store the value that results from the evaluation. For this, the Policy uses variables. You can create custom variables whenever you need them by defining a variable prefix and variable name in the ExtractVariables Policy. The Policy can then store the value that it finds by parsing the message payload in the variable, and Policies and code can retrieve that value to do some additional processing on it.

In this example, you define four custom variables:

  • weather.location
  • weather.condition
  • weather.forecast_today
  • weather.forecast_today

In these variables that you define here, weather is a prefix. Location, condition, forecast_today, and forecast_today are each variable names, and each will have an associated XPath expression.

You can give custom variables any name you want. The only restrictions are that custom variables cannot share the same name as any Variables reference, and the only special character permitted in custom variable names is underscore, _.). However, you should always define semantically accurate names for your variables, especially when using variables for custom analytics.

The following ExtractVariables Policy configuration demonstrates how you can define these custom variables. Note that the VariablePrefix is weather, and that each Variable element has a name attribute that maps to the list of custom variables that you need to have populated. Each variable in turn has an associated XPath expression that will populate the variable with the appropriate value for each response message.

To create the Policy, under /apiproxy/policies, create a file called ParseWeatherReport.xml with the following content:

<ExtractVariables name="ParseWeatherReport">
 <!-- Parse the XML weather report using XPath. -->
 <VariablePrefix>weather</VariablePrefix>
  <XMLPayload>
   <Namespaces>
    <Namespace prefix="yweather">http://xml.weather.yahoo.com/ns/rss/1.0</Namespace>
   </Namespaces>
   <Variable name="location" type="string">
    <XPath>/rss/channel/yweather:location/@city</XPath>
   </Variable>
   <Variable name="condition" type="string">
    <XPath>/rss/channel/item/yweather:condition/@text</XPath>
   </Variable>
   <Variable name="forecast_today" type="string">
    <XPath>/rss/channel/item/yweather:forecast[1]/@text</XPath>
   </Variable>
   <Variable name="forecast_tomorrow" type="string">
    <XPath>/rss/channel/item/yweather:forecast[2]/@text</XPath>
   </Variable>
 </XMLPayload>
</ExtractVariables>

The next step is to create another Policy that reads these values and sends them to Analytics Services for processing. The StatisticsCollector Policy type is used to do this. In the StatisticsCollector Policy, you define a Statistic by providing a pointer to each of the custom variables defined in the ExtractVariables Policy. You can also provide a default value for custom variable, which will be forwarded to Analytics Services if the variables cannot be resolved on a response. (in the example below, the default values are Earth, Sunny, Rainy, and Balmy.)

Under /apiproxy/policies, create a file called AnalyzeWeatherReport.xml with the following content:

<StatisticsCollector name="AnalyzeWeatherReport">
 <Statistics>
  <Statistic name="location" ref="weather.location" type="string">Earth</Statistic>
  <Statistic name="condition" ref="weather.condition" type="string">Sunny</Statistic>
  <Statistic name="forecast_today" ref="weather.forecast_today" type="string">Rainy</Statistic>
  <Statistic name="forecast_tomorrow" ref="weather.forecast_tomorrow" type="string">Balmy</Statistic>
 </Statistics>
</StatisticsCollector>

Attaching policies to the ProxyEndpoint response Flow

To make things work properly, Policies must be attached to the API proxy Flow in the appropriate location. In this use case, the Policies must execute after the response has been received from the Yahoo Weather API and before the response is sent to the request client. To accomplish this, the Policies must be attached to the ProxyEndpoint response Flow, so that they will be enforced on outbound response messages, before the response is returned to the calling client app.

The example ProxyEndpoint configuration below first executes the Policy called 'ParseWeatherReport' to parse the response message. The ParseWeatherReport evaluates the XPath expressions and populates appropriate variables. The policy called 'AnalyzeWeatherReport' then forwards those values to Analytics Services.

 
<ProxyEndpoint name="default">
 <Flows>
  <Flow name="default">
   <Response>
    <Step><Name>ParseWeatherReport</Name></Step>
    <Step><Name>AnalyzeWeatherReport</Name></Step>
   </Response>
  </Flow>
 </Flows>
 <HTTPProxyConnection>
  <!-- Base path used to route inbound requests to this API proxy -->
  <BasePath>/weather</BasePath>
  <!-- The named virtual host that defines the base URL for requests to this proxy -->
  <VirtualHost>default</VirtualHost>
 </HTTPProxyConnection>
 <RouteRule name="default">
 <!-- Connects the proxy to the target defined under /targets -->
  <TargetEndpoint>default</TargetEndpoint>
 </RouteRule>
</ProxyEndpoint>

Importing and deploying the API proxy

After you have made these changes, you need to import and deploy the API proxy that you have configured.

If you are using the simpleProxy in the Using the samples on GitHub, run:

$ sh deploy.sh

For deployment instructions, see Configure and deploy an API proxy.

Populating Analytics data for custom variables

After you deploy your changes, you need to populate some data in Analytics Services. You can do this by running the following commands, each of which uses a WOEID for a different geographic location.

Palo Alto:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=12797282

Shanghai:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2151849

London:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=44418

Wiliamsburg:

$ curl http://{org_name}-test.apigee.net/weather/forecastrss?w=2520841

Generating a custom report

Now that you have populated some data, you can use the RESTful API exposed by Analytics Services to get statistics on your new custom dimensions, in the same way that you use the API to get statistics on the out-of-the-box dimensions.

You can also generate custom reports using the UI at enterprise.apigee.com. You will find that a new dimension called location is available in the UI for you to use in generating reports.

timeRange parameter must be modified to include the time interval when data was collected.

In the example request below, the custom dimension is called location. This request builds a custom report for locations based on the sum of message counts submitted for each location.

As usual, substitute your organization name for the variable {org_name}, and substitute the username and password for your account on Apigee Edge for myname:mypass:

$ curl https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?"select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day"
-u myname:mypass
{
  "environments" : [ {
    "dimensions" : [ {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "4.0"
        } ]
      } ],
      "name" : "London"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "19.0"
        } ]
      } ],
      "name" : "Palo Alto"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "2.0"
        } ]
      } ],
      "name" : "Shanghai"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "14.0"
        } ]
      } ],
      "name" : "Williamsburg"
    } ],
    "name" : "test"
  } ],
  "metaData" : {
    "samplingRate" : "100"
  }
}

In some cases, there may be a large number of results. It may be useful to filter the list to report the top 2 locations by message volume. This is done by adding the topk query parameter and providing an integer value for the number to filter:

$ curl https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?'select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day&sortby=sum(message_count)&topk=2" \
-u myname:mypass

Results can also be filtered by specifying the values of the dimensions of interest. In the example below, the report is filtered by results for London and Shanghai :

$ curl https://api.enterprise.apigee.com/v1/o/{org_name}/environments/test/stats/location?"select=sum(message_count)&timeRange=11/19/2012%2000:00~11/21/2012%2000:00&timeUnit=day&filter=(location%20in%20'London','Shanghai')" \
-u myname:mypass
{
  "environments" : [ {
    "dimensions" : [ {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "4.0"
        } ]
      } ],
      "name" : "London"
    }, {
      "metrics" : [ {
        "name" : "sum(message_count)",
        "values" : [ {
          "timestamp" : 1353369600000,
          "value" : "2.0"
        } ]
      } ],
      "name" : "Shanghai"
    } ],
    "name" : "test"
  } ],
  "metaData" : {
    "samplingRate" : "100"
  }
}

For complete API documentation, see the Analytics Services API reference.

 

Comments

This page doesn't seem to appear in the search results for "custom dimensions," which is a shame because it is exactly the right hit!

http://apigee.com/docs/search/site/%2522custom%2520dimensions%2522

Add new comment

Provide your email address if you wish to be contacted offline about your comment.
We will not display your email address as part of your comment.

We'd love your feedback and perspective! Please be as specific as possible.
Type the characters you see in this picture. (verify using audio)

Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.