11436 SSO

Tutorial: Writing a Custom Transform Plugin for Apigee Edge Microgateway

wwitman
Jan 19, 2016

Apigee Edge Microgateway gives you the flexibility to deploy API management as a hybrid solution, where core, real-time API management functionality runs close to the API applications, while other API functions run on Apigee’s public (or private) cloud.

A plugin is simply a Node.js module that adds functionality to Edge Microgateway. Several standard Edge Microgateway features are implemented as plugins, including analytics, OAuth security, spike arrest, and quota.

Here, we'll show you how to write a custom Apigee Edge Microgateway plugin that transforms response data from XML to JSON. Along the way, we'll explore how plugins work and discuss some useful best practices for plugin development.

This example relies on two plugins:

  • One to accumulate response data into one object before transforming it
  • Another to transform the accumulated data from XML to JSON

We'll walk through their implementations one at at time.

Note: We assume you're familiar with Edge Microgateway and have configured and run an instance of Edge Microgateway and have built and tried the "hello world" plugin. If not, you can find getting started docs on the Edge documentation site.

Step 1: Accumulate response data

Because we’re transforming XML, a non-streaming data format, our goal is to ensure that we've accumulated the entire response in one data object before transforming it. This approach ensures that the XML is complete and valid before we apply the transformation.

Fortunately, Edge Microgateway provides a plugin called accumulate-response that handles the accumulation work for you.

Let's take a close look at the accumulate-response plugin code. We've removed some handlers that we don't need for this example:

plugins/accumulate-response/index.js (abridged)


module.exports.init = function(config, logger, stats) {

 function accumulate(res, data) {
   if (!res._chunks) res._chunks = [];
   res._chunks.push(data);
 }
 return {
   ondata_response: function(req, res, data, next) {
     if (data && data.length > 0) accumulate(res, data);
     next(null, null);
   },
   onend_response: function(req, res, data, next) {
     if (data && data.length > 0) accumulate(res, data);
     var content = Buffer.concat(res._chunks);
     delete res._chunks;
     next(null, content);
   }
 };
}

Let's look at the ondata_response event handler first. It is called whenever a chunk of response data is returned. The next() call can then pass the data (either transformed or not) to the next event handler in the plugin chain.

It typically works like this: the first argument to next() is reserved for an error object, and the second argument is the response data object. For example:


next(null, transformed_data);

Note that if you pass the data to next(), it will be passed along to the next event handler in the plugin chain and finally returned to the client. If another chunk comes in, the same pattern repeats. 

However, for our example, we don't want to pass the data through to the client each time the handler is called. Instead, we want to accumulate it so that we can transform it all at once. This is the only way to ensure that valid XML is returned. How do we do this? We pass null as the second argument to next() in the ondata_response handler.  For example:


next(null, null);

Following this pattern, the data is not passed to the next event handler in the chain, and therefore, it is not passed back to the client. This pattern allows us to accumulate the incoming data chunks until the onend_response handler is called.

Let's take a closer look at the accumulate() function itself:


function accumulate(res, data) {
   if (!res._chunks) res._chunks = [];
   res._chunks.push(data);
 }

Each time it is called, the function pushes the data into an array assigned to the response object.

Why not store the accumulated data in a global variable? Because in a given instance of Edge Microgateway, all requests and responses flow through a single plugin instance. Therefore, a global variable would be overwritten each time a new response or request is received. The best practice is to store the accumulated data in the request or response object, which is unique for each request or response.

For performance reasons, we accumulate the data in an array, and then do a single Buffer.concat() operation in the onend_response handler:


   onend_response: function(req, res, data, next) {
     if (data && data.length > 0) accumulate(res, data);
     var content = Buffer.concat(res._chunks);
     delete res._chunks;
     next(null, content);
   }

This is because Buffer.concat() is a more expensive operation than pushing data onto an array.

A couple of final points about the accumulator plugin:

  • In onend_response, the next() call passes the accumulated content to the next plugin handler in the chain. If there are no more plugin handlers, then the data is passed to the client. In the next section, we'll add another plugin to receive the data—the transform plugin.
  • The delete operator is optional, and is called to improve the efficiency of the garbage collector.

To use the accumulator plugin, all you have to do is add it to the plugin sequence in <microgateway-root-dir>/agent/config/default.yaml, like this:


plugins:
   dir: ../plugins
   sequence:
     - accumulate-response

Step 2: Write an XML to JSON transform plugin

Now, we need a plugin to transform the accumulated response data. In our example, we expect the target response to be XML data (as is the case for the weather API example used in the Edge Microgateway getting started docs), and we want to translate the response to JSON.

We'll start by adapting another plugin that is provided with each Edge Microgateway install called transform-uppercase. It illustrates a best practice for implementing a transform plugin. You're free to use it as a starting point for developing new transform plugins.  

Here's the plugin code—we're only interested in the onend_response handler, so the other handlers are omitted from the sample below:

plugins/transform-uppercase/index.js (abridged)


module.exports.init = function(config, logger, stats) {
function transform(data) {
   return new Buffer(data.toString().toUpperCase());
}
return {
   onend_response: function(req, res, data, next) {
       // transform accumulated data, if any
       next(null, data ? transform(data) : null);
   }
};
}

Now, let's create a new plugin in the <microgateway-root-dir>/plugins folder, like this:

  1. cd <microgateway-root-dir>/plugins
  2. mkdir transform-xml2json
  3. cd transform-xml2json

  4. touch index.js

  5. npm install xml2json

  6. In an editor, open the index.js file and copy the following code into it. It's the same code as the transform-uppercase starter plugin, but with a couple of changes:

plugins/transform-xml2json/index.js


var parser = require('xml2json');

function transform(data) {
   return new Buffer(parser.toJson(data));
 }

module.exports.init = function(config, logger, stats) {

 return {
   onend_response: function(req, res, data, next) {
     // transform accumulated data, if any
     next(null, data ? transform(data) : null);
   }
 };
}

First, we require the NPM module xml12json. Then, we changed:


return new Buffer(data.toString().toUpperCase());

to:


return new Buffer(parser.toJson(data));

When onend_response() is called, all of the data received and accumulated by the accumulator plugin will be transformed at once. The conversion work is done by the xml2json module, which is available through NPM.

Step 3: Add the transform plugin to the plugin sequence

This last part is important. We will add the XML to JSON plugin to the plugin sequence in <microgateway-root-dir>/agent/config/default.yaml.

But wait—response event handlers are called in reverse order of the sequence! Therefore, we need to place the transform plugin before the accumulator plugin:


plugins:
   dir: ../plugins
   sequence:
     - transform-xml2json
     - accumulate-response

To learn more about handler execution order, check out the doc topic "About plugin handler execution order" in the Edge Microgateway docs.

With this configuration, the onend_response function in the accumulate-response plugin is called first. The next() call passes the accumulated data to the next handler in the plugin chain, which is the onend_response in the transform-xml2json plugin, where the transformation is performed.

Try it

If you restart the agent, and restart Edge Microgateway, and call an API that returns XML data (like the weather API used in the getting started docs), the response will be converted to JSON. For example, the weather API returns XML, but our plugin transforms it to JSON:


{
   "rss": {
       "channel": {
           "description": "Yahoo! Weather for Palo Alto, CA",
           "image": {
               "height": "18",
               "link": "http://weather.yahoo.com",
               "title": "Yahoo! Weather",
               "url": "http://l.yimg.com/a/i/brand/purplelogo//uh/us/news-wea.gif",
               "width": "142"
           },
           "item": {
               ...
}

A final note: this entire example operates on response data. You can follow the same coding pattern to transform request data (such as POST or PUT data). But remember that if you operate on request data, plugin request handlers are executed in the order in which they appear in the configuration sequence. So you might have a sequence like this if you were transforming request data:
 


plugins:
   dir: ../plugins
   sequence:
     - accumulate-request
     - transform-xml2json

For more information about Edge Microgateway, go to the Apigee docs site. You can also ask questions and join conversations on the Apigee Community. For a deep dive into Edge Microgateway, check out this recent webcast.

Thanks to Akhil Arora for reviewing this blog and providing feedback.

Image: Nico Tzogalis/The Noun Project

Microservices Done Right

Next Steps

 
 

Resources Gallery

News