Using policy composition

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation.
info

In this topic, you will learn how to create a mashup using policy composition. Policy composition is an Apigee proxy pattern that lets you combine results from multiple backend targets into a single response using policies.

For a general overview of policy composition, see "The policy composition pattern" in API Proxy Cookbook patterns.

Download and try out the sample code

About this cookbook example

This cookbook example illustrates an API proxy pattern called policy composition. This pattern provides one way (there are others) to mash up data from multiple backend sources. More generally, this topic demonstrates how policies can be combined and chained together to produce a desired result. For a general overview of this pattern and other related ones, see API Proxy Cookbook patterns.

The example discussed here uses policy composition to mash up data from these two separate public APIs:

  • The Google Geocoding API: This API converts addresses (like "1600 Amphitheatre Parkway, Mountain View, CA") into geographic coordinates (like latitude 37.423021 and longitude -122.083739).
  • The Google Elevation API This API provides a simple interface to query locations on the earth for elevation data. In this example, the coordinates returned from the Geocoding API will be used as input into this API.

App developers will call this API proxy with two query parameters, a postal code and a country ID:

$ curl "http://{myorg}-test.apigee.net/policy-mashup-cookbook?country=us&postalcode=08008"

The response is a JSON object that includes the geocoded location (latitude/longitude) for the center of the supplied postal code area combined with the elevation at that geocoded location.

{  
   "ElevationResponse":{  
      "status":"OK",
      "result":{  
         "location":{  
            "lat":"39.7500713",
            "lng":"-74.1357407"
         },
         "elevation":"0.5045232",
         "resolution":"76.3516159"
      }
   }
}

Before you begin

If you'd like to read a brief overview of the policy composition pattern, see "The policy composition pattern" in API Proxy Cookbook patterns.

Before you explore this cookbook example, you should also be familiar with these fundamental concepts:

  • What policies are and how to attach them to proxies. For a good introduction to policies, see What's a policy?.
  • The structure of an API proxy flow, as explained in Configuring flows. Flows let you specify the sequence in which policies are executed by an API proxy. In this example, several policies are created and added to the API proxy's flow.
  • How an API proxy project is organized on your filesystem, as explained in API proxy configuration reference. This cookbook topic demonstrates local development (file system-based) as opposed to cloud-based development where you could use the management UI to develop the API proxy.
  • Use of API key validation. This is the simplest form of app-based security that you can configure for an API. For more information see API keys. You can also walk through the Secure an API by requiring API keys tutorial.
  • A working knowledge of XML. In this example, we build out the API proxy and its policies with XML files that reside on the filesystem.

If you have downloaded the sample code, you can locate all of the files discussed in this topic in the mashup-policy-cookbook sample folder. The following sections discuss the sample code in detail.

Going with the flow

Before moving to the policies, let's take a look a the main flow of our example API proxy. The flow XML, shown below, tells us a lot about this proxy, the policies it uses, and where those policies are called.

In the sample download, you can find this XML in the file doc-samples/policy-mashup-cookbook/apiproxy/proxies/default.xml.

<ProxyEndpoint name="default">
  <Flows>
    <Flow name="default">
      <Request>
            <!-- Generate request message for the Google Geocoding API -->
            <Step><Name>GenerateGeocodingRequest</Name></Step>
            <!-- Call the Google Geocoding API -->
            <Step><Name>ExecuteGeocodingRequest</Name></Step>
            <!-- Parse the response and set variables -->
            <Step><Name>ParseGeocodingResponse</Name></Step>
            <!-- Generate request message for the Google Elevation API -->
            <Step><Name>AssignElevationParameters</Name></Step>
      </Request>
      <Response>
            <!-- Parse the response message from the Elevation API -->
            <Step><Name>ParseElevationResponse</Name></Step>
            <!-- Generate the final JSON-formatted response with JavaScript -->
            <Step><Name>GenerateResponse</Name></Step>
      </Response>
    </Flow>
  </Flows>

  <HTTPProxyConnection>
    <!-- Add a base path to the ProxyEndpoint for URI pattern matching-->
    <BasePath>/policy-mashup-cookbook</BasePath>
    <!-- Listen on both HTTP and HTTPS endpoints -->
    <VirtualHost>default</VirtualHost>
    <VirtualHost>secure</VirtualHost>
  </HTTPProxyConnection>
  <RouteRule name="default">
    <!-- Connect ProxyEndpoint to named TargetEndpoint under /targets -->
    <TargetEndpoint>default</TargetEndpoint>
  </RouteRule>
</ProxyEndpoint>

Here's a summary of the flow's elements.

  • <Request> - The <Request> element consists of several <Step> elements. Each step calls one of the policies that we'll create through the rest of this topic. These policies are concerned with creating a request message, sending it, and parsing the response. By the end of this topic, you'll understand the role of each of these policies.
  • <Response> - The <Response> element also includes <Steps>. These steps also call policies that are responsible for processing the final response from the target endpoint (the Google Elevation API).
  • <HttpProxyConnection> - This element specifies details about how apps will connect to this API proxy, including the <BasePath>, which specifies how this API will be called.
  • <RouteRule> - This element specifies what happens immediately after the inbound request messages are processed. In this case, the TargetEndpoint is called. We'll discuss more about this important step later in this topic.

Creating the policies

The following sections discuss each of the policies that make up this policy composition example.

Create the first AssignMessage policy

The first AssignMessage policy, listed below, creates a request message that will be sent to the Google Geocoding service.

Let's start with the policy code, and then we'll explain its elements in more detail. In the sample download, you can find this XML in the file doc-samples/policy-mashup-cookbook/apiproxy/policies/GenerateGeocodingRequest.xml.

<AssignMessage name="GenerateGeocodingRequest">
  <AssignTo createNew="true" type="request">GeocodingRequest</AssignTo>
  <Set>
    <QueryParams>
      <QueryParam name="address">{request.queryparam.postalcode}</QueryParam>
      <QueryParam name="region">{request.queryparam.country}</QueryParam>
      <QueryParam name="sensor">false</QueryParam>
    </QueryParams>
    <Verb>GET</Verb>
  </Set>
  <!-- Set variables for use in the final response -->
  <AssignVariable>
    <Name>PostalCode</Name>
    <Ref>request.queryparam.postalcode</Ref>
  </AssignVariable>
  <AssignVariable>
    <Name>Country</Name>
    <Ref>request.queryparam.country</Ref>
  </AssignVariable>
</AssignMessage>

Here's a brief description of the elements in this policy. You can read more about this policy in Assign Message policy.

  • <AssignMessage name> - Gives this policy a name. The name is used when the policy is referenced in a flow.
  • <AssignTo> - Creates a named variable called GeocodingRequest. This variable encapsulates the request object that will be sent to the backend by the ServiceCallout policy.
  • <QueryParams> - Sets the query parameters that are needed by the backend API call. In this case, the Geocoding API needs to know the location, which is expressed with a postal code, and a country ID. The app user supplies this information, and we simply extract it here. The sensor param is required by the API, and is either true or false, and we just hardcode it to false here.
  • <Verb> - In this case, we are making a simple GET request to the API.
  • <AssignVariable> - These variables store the values we are passing to the API. In this example, the variables will be accessed later in the response returned to the client.

Send the request with ServiceCallout

The next step in the policy composition sequence is to create a ServiceCallout policy. The ServiceCallout policy, listed below, sends the request object that we created in the previous AssignMessage policy to the Google Geocoding service, and saves the result in a variable called GeocodingResponse.

As before, let's take a look at the code first. A detailed explanation follows. You can read more about this policy in Service Callout policy. In the sample download, you can find this XML in the file doc-samples/policy-mashup-cookbook/apiproxy/policies/ExecuteGeocodingRequest.xml.

<ServiceCallout name="ExecuteGeocodingRequest">
  <Request variable="GeocodingRequest"/>
  <Response>GeocodingResponse</Response>
  <HTTPTargetConnection>
    <URL>http://maps.googleapis.com/maps/api/geocode/json</URL>
  </HTTPTargetConnection>
</ServiceCallout>

Here's a brief description of this policy's elements.

  • <ServiceCallout> - As with the previous policy, this one has a name.
  • <Request variable> - This is the variable that was created in the AssignMessage policy. It encapsulates the request going to the backend API.
  • <Response> - This element names a variable in which the response is stored. As you will see, this variable will be accessed later by the ExtractVariables policy.
  • <HTTPTargetConnection> - Specifies the target URL of the backend API. In this case, we specify that the API return a JSON response.

Now we have two policies, one that specifies the request information needed to use the backend API (Google's Geocoding API), and the second one that actually sends the request to the backend API. Next, we'll handle the response.

Parse the response with ExtractVariables

The ExtractVariables policy provides a simple mechanism for parsing content from the response message obtained by a ServiceCallout policy. ExtractVariables can be used to parse JSON or XML, or it can be used to extract content from URI paths, HTTP headers, query parameters, and form parameters.

Here's a listing of the ExtractVariables policy. You can read more about this policy in Extract Variables policy. In the sample download, you can find this XML in the file doc-samples/policy-mashup-cookbook/apiproxy/policies/ParseGeocodingResponse.xml.

<ExtractVariables name="ParseGeocodingResponse">
  <Source>GeocodingResponse</Source>
  <VariablePrefix>geocoderesponse</VariablePrefix>
  <JSONPayload>
    <Variable name="latitude">
       <JSONPath>$.results[0].geometry.location.lat</JSONPath>
    </Variable>
    <Variable name="longitude">
       <JSONPath>$.results[0].geometry.location.lng</JSONPath>
    </Variable>
  </JSONPayload>
</ExtractVariables>

The key elements of the ExtractVariable policy are:

  • <ExtractVariables name> - Again, the policy name is used to refer to the policy when it is used in a flow.
  • <Source> - Specifies the response variable that we created in the ServiceCallout policy. This is the variable from which this policy extracts data.
  • <VariablePrefix> - The variable prefix specifies a namespace for other variables created in this policy. The prefix can be any name, except for the reserved names defined by Edge's predefined variables.
  • <JSONPayload> - This element retrieves the response data that is of interest to us and puts it into named variables. In fact, the Geocoding API returns much more information than latitude and longitude. However, these are the only values we need for this sample. You can see a complete rendering of the JSON returned by the Geocoding API in the API's documentation. The values of geometry.location.lat and geometry.location.lng are simply two of the many fields in the returned JSON object.

It may not be obvious, but it's important to see that ExtractVariables produces two variables whose names consist of the variable prefix (geocoderesponse) and the actual variable names that are specified in the policy. These variables are stored in the API proxy and will be available to other policies within the proxy flow, as you will see. The variables are:

  • geocoderesponse.latitude
  • geocoderesponse.longitude

Most of the work is now done. We have created a composite of three policies that form a request, call a backend API, and parse the returned JSON data. In the final steps, we'll feed data from this part of the flow into another AssignMessage policy, call the second backend API (Google Elevation API), and return our mashed up data to the app developer.

Generate the second request with AssignMessage

The following AssignMessage policy uses variables returned from the first backend (Google Geocoding) that we stored, and plugs them into a request destined for the second API (Google Elevation). As noted previously, these variables are geocoderesponse.latitude and geocoderesponse.longitude.

In the sample download, you can find this XML in the file doc-samples/policy-mashup-cookbook/apiproxy/policies/AssignElevationParameters.xml.

<AssignMessage name="AssignElevationParameters">
<Remove>
    <QueryParams>
      <QueryParam name="country"/>
      <QueryParam name="postalcode"/>
    </QueryParams>
  </Remove>
  <Set>
    <QueryParams>
      <QueryParam name="locations">{geocoderesponse.latitude},{geocoderesponse.longitude}</QueryParam>
      <QueryParam name="sensor">false</QueryParam>
    </QueryParams>
  </Set>
</AssignMessage>

If you examine the Google Elevation API, you will see that it takes two query parameters. The first is called locations and its value is the latitude and longitude (comma-separated values). The other param is sensor, which is required and must be either true or false. The most important thing to note at this point is that the request message we create here does not require a ServiceCallout. We don't need to call the second API from a ServiceCallout at this point because we can call the backend API from the proxy's TargetEndpoint. If you think about it, we have all the data we need to call Google Elevations API The request message generated in this step does not require a ServiceCallout, as the request generated for the main request pipeline, and so will simply be forwarded by the ProxyEndpoint to the TargetEndpoint, following the RouteRule configured for this API proxy. The TargetEndpoint manages the connection with the remote API. (Recall that the URL for the elevation API is defined in the HTTPConnection for the TargetEndpoint. Elevation API documentation if you would like to know more. The QueryParams that we stored previously, country and postalcode, are no longer needed, so we remove them here.

Brief pause: Back to the flow

At this point, you may wonder why we aren't creating another ServiceCallout policy. After all, we created another message. How does that message get sent to the target, the Google Elevation API? The answer is in the <RouteRule> element of the flow. <RouteRule> specifies what to do with any remaining request messages after the <Request> part of the flow has executed. The TargetEndpoint specified by this <RouteRule> tells the API proxy to deliver the message to http://maps.googleapis.com/maps/api/elevation/xml.

If you downloaded the sample API proxy, you can find the TargetProxy XML in the file doc-samples/policy-mashup-cookbook/apiproxy/targets/default.xml.

<TargetEndpoint name="default">
  <HTTPTargetConnection>
    <!-- This is where we define the target. For this sample we just use a simple URL. -->
    <URL>http://maps.googleapis.com/maps/api/elevation/xml</URL>
  </HTTPTargetConnection>
</TargetEndpoint>

Now, we just need to process the response from the Google Elevation API and we're done.

Convert the response from XML to JSON

In this example, the response from the Google Elevation API is returned as XML. For "extra credit," let's add one more policy to our composite to transform the response from XML to JSON.

This example uses the JavaScript policy named GenerateResponse, with a resource file containing the JavaScript code, to perform the conversion. Shown below is the GenerateResponse policy definition:

<Javascript name="GenerateResponse" timeout="10000">
  <ResourceURL>jsc://GenerateResponse.js</ResourceURL>
</Javascript>

The GenerateResponse.js resource file includes the JavaScript used to perform the conversion. You can see that code in the file doc-samples/policy-mashup-cookbook/apiproxy/resources/JSC/GenerateResponse.js.

Apigee also provides an out-of-the-box policy, XMLToJSON, to convert XML to JSON. You can edit the ProxyEndpoint to use the xmltojson policy shown below instead.

<XMLToJSON name="xmltojson">
  <Options>
  </Options>
  <OutputVariable>response</OutputVariable>
  <Source>response</Source>
</XMLToJSON>

Testing the example

If you have not already done so, try to download, deploy, and run the policy-mashup-cookbook sample, which you can find in the doc-samples folder in the Apigee Edge samples repository GitHub. Just follow the instructions in the README file in the policy-mashup-cookbook folder. Or, follow the brief instructions here: Using the sample API proxies.

To summarize, you can call the composite API as follows. Substitute your {myorg} with your organization name:

$ curl "http://{myorg}-test.apigee.net/policy-mashup-cookbook?country=us&postalcode=08008"

The response includes the geocoded location for the center of the postal code provided by the app end user, combined with the elevation at that geocoded location. The data was retrieved from two backend APIs, mashed up with policies attached to the API proxy, and returned to the client in a single response.

{  
   "country":"us",
   "postalcode":"08008",
   "elevation":{  
      "meters":0.5045232,
      "feet":1.6552599030345978
   },
   "location":{  
      "latitude":39.75007129999999,
      "longitude":-74.1357407
   }
}

Summary

This cookbook topic explained how to use the policy composition pattern to create a mashup of data from multiple backend sources. Policy composition is a common pattern used in API proxy development for adding creative functionality to your API.