API design

Building StreetCarts: Sample API Proxies to Access Food Truck Data

StreetCarts needs you!

We've got a few features we want to add,  including a client app (maybe an Ionic or iOS app?), support for payment processing, and support for reviewing food carts (i.e., finishing the /reviews proxy).

If you're an Apigee developer and would like to participate,  post a suggestion or question in the Apigee Community. Be sure to use the "streetcarts" tag when posting.

 

Back in October, Apigee's doc team started building an app using the technologies we write about every day. We recently finished the first version of StreetCarts, a set of sample proxies for accessing data about food carts (or food trucks, if you prefer).

You'll find the code in the GitHub repository we've put up, including instructions on how to get started, how to deploy the app to your Edge and API BaaS instances, and how to seed your data store and start experimenting.

Note: StreetCarts is a sample, not production-level code. Please don't try to run it as a production app!

In an October post on the Apigee Community, I said that doc folks often don't generally have time to cover more than the pat use cases. We usually doc a feature in isolation, rather than in combination with other features. But in the real world, combinations are what can make an app both interesting and tricky to code.

This post describes a few of the turns and reversals we experienced along the way while combining things to get the results we wanted.

StreetCarts includes the following technology features:

  • Multiple client-facing proxies calling a single internal proxy (using proxy chaining) to share a connection with a backend resource
  • Authentication combining Edge OAuth policies and an API BaaS data store
  • Authorization combining OAuth scopes in Edge and permissions in API BaaS
  • Node.js modules to integrate an API BaaS data store with Edge proxies
  • Sensitive credential data stored in the Edge secure store (vault), then retrieved with Node.js JavaScript
  • spike arrest policy to suppress potential traffic spikes
  • Support for two distinct client access levels using the quota policy to enforce product-scoped quotas.

So what does it do?

We're putting together a list of the features for the next revision, but so far we’ve implemented these basic features:

  • All users (even anonymous ones) can browse food carts, menus, and food items.
  • Registered users who are owners can create/edit/delete food carts, menus, and food items.
  • Registered users who are cart managers can create/edit/delete menus and food items.

We'd like to add the ability to purchase food items (in the example sense) down the road. Then again, we think it would be cool for others to code additional features and submit pull requests.

Could the code be more efficient? Without doubt. Are some of our design decisions debatable? Probably. But we're looking forward to turning the experience into more content. And we've had a good time learning how some of this stuff really works.

API design

StreetCarts ended up with seven API proxies. Dividing the endpoints among these seven made sense from a functional standpoint. It also made it a little easier for us to work on them by helping us avoid collisions.

  • Three proxies are for core data requests (such as creating or updating food carts, menus, or food items)
  • One proxy is for user data operations
  • One proxy is for OAuth token requests
  • An "internal" proxy for accessing the data store is not accessible to clients. Its clients are the other StreetCarts proxies
  • A proxy for reviews is not yet fully implemented

The /foodcarts proxy is the fattest, partly because it contains most of the POST endpoints. We wanted to be sure that we associated menus and items with their respective food carts. That was easier with a path such as the following, which is in the /foodcarts proxy:

POST /foodcarts/:cartID/menus

If we'd had the menu creation endpoint in the /menus proxy, we'd have had to find another, maybe messier way to pass in the ID of the cart to which the new menu belonged.

GET, PUT, and DELETE requests for a single item or menu, on the other hand, are made to /items or /menus by specifying the entity ID.

Security

Authentication and authorization comprise one of the areas where we had the most design churn. For example, early on we were using Edge product definitions to capture the difference between requests that guest (non-authenticated) users were authorized to make versus those allowed for registered users who were logged in.

In the end, as described below, we used OAuth scopes to define broad authorization categories and used products instead to define client access levels based on the number of requests allowed (quotas).

Authentication

StreetCarts stores user data in an API BaaS data store and uses OAuth v2 for authentication. When a user authenticates, the accesstoken proxy calls the data-manager proxy to verify that the username and password correspond to a user account in the data store.

Because we're using the API BaaS token endpoint on the backend, we get an OAuth token back from BaaS. We store this token in the Edge-generated token we send back to the client. That way, every subsequent request can use the API BaaS-generated token to authenticate with the backend. That allowed us to use API BaaS permissions for small-grained authorization.

Authorization

One of the challenges we had along the way was figuring out how much of the authorization logic should be in Edge versus in API BaaS. Was there a way to define user access only in a proxy and save a trip to the data store for invalid requests?

In the end, it made more sense to have Edge and API BaaS share the authorization role. We leaned on the built-in role-based access control tools in API BaaS for small-grained authorization. There, it's easy to define user groups, roles, and permissions-per-resource, then associate user accounts with those things. What's more, we were able to define permissions at run time using JavaScript so that we could protect data independently for each new food cart added to the system.

Closer to the front end of the request, in Edge, we defined OAuth scopes that established broad permissions. For example, a request from a user asking to update a food cart won't even reach the backend if they aren't in the owner.update scope. So it potentially cuts down a bit on the traffic to the backend by stopping requests in an area where the user doesn't belong.

Data access

At the start, we wanted to use API BaaS as a data store only. All of our logic around authentication and authorization, for example, would be in Edge. And the Node.js code we wrote to connect the front-end proxies to the backend data store would be swappable with other TBD connective code so that we could show how to use a different data store in another sample.

But it was a lot simpler to use API BaaS to get some non-DB things done, including parts of authentication and authorization. And trying to do all of that authorization in Edge felt like we were bending it to a shape it didn't like. In the end, we figured that if we were using a data store other than BaaS, we'd probably be using a separate user management system (like LDAP) for authorization. It just happened that BaaS includes both.

Another data store design decision was to use the API BaaS connections mechanism rather than query strings and primary/secondary keys. If you're not familiar with it, connections can look a bit like syntactic voodoo. But it helped us avoid writing queries against a Cassandra-based

API BaaS for capturing one-to-many associations (like food cart-to-menus and menu-to-food items).

Development process

One of the most challenging parts of this project was doing iterative development. Edge is a product that only runs on the server, with browser-based development tools also on the server. Yet we were keeping source code in a GitHub repository with local branches.

The development tools were handy, especially at the start. We could change code, then trace and make requests from a client to see what worked. But once something worked on a personal Edge server, the developer still needed to get the changed code into the repo. If they'd changed multiple files to get to that "working" place, they needed to figure out which files they'd changed, then somehow copy the changed code out of the management console into their local repo.

We each had a slightly different process, from comparing downloaded code against a local repo to deploying from source to test anew. Here's one process that seemed to work:

  1. Code and deploy your app to your personal Edge testing organization.
  2. Use a client (such as Postman) to make calls, tracing in Edge to discover where things fall down.
  3. Edit your local code to fix issues, then paste your local changes into the the Edge console's editing window. Save the changes in your personal Edge testing organization.
  4. Run and trace again.
  5. Repeat steps 2 through 4 as needed to get committable code.
  6. When you're happy with your changes, push your code to the repository (remember that you've been making changes to your local repo all along).
  7. Pull changes from others, then build and deploy to a shared staging environment for validation by the team.

StreetCarts needs your help!

We've got a short list of features we want to add. Some of the low-hanging fruit includes a client app (maybe an Ionic or iOS app?), support for payment processing, and support for reviewing food carts (i.e., finishing the /reviews proxy).

If you're a developer working with Apigee products, why not get involved? Let us know if you'd like to start working on a feature by posting a suggestion or question in the Apigee Community. Be sure to use the "streetcarts" tag when posting.

In the meantime, we'll post more in-depth content about the implementation in the Apigee Community.

Rapier: Cut Through the Tedium of API Specification

At Apigee, we're big believers in the importance of API metadata for all aspects of the API lifecycle—we were one of the early promoters of Swagger and a founding member of the OpenAPI Initiative. We never stop thinking of ways to make API design and development better, and to that end, we've set up an Apigee Labs organization on GitHub as a place to incubate new ideas.

Today, we want to share with you a new project that we're working on—Rapier—that is a proving ground for some of these ideas. Some are ideas that we may propose for inclusion in future versions of the OpenAPI Specification, and others may become part of Apigee's roadmap, but for now, we just want to get these into a wider discussion and solicit feedback in the spirit of open development.

Rapier is a new API specification language created by Apigee. The goals of Rapier are to allow REST APIs to be specified and learned with one-tenth the effort required with other API specification languages, and to produce specifications that describe higher-quality APIs1. Rapier is published in this public repository under an Apache 2.0 license—it will be managed as an open source project.

The name Rapier is an acronym for “REST APIs from entities and relationships.” With Rapier, you specify an API in YAML by specifying the entities and relationships of the data model that underlies the API, along with query paths traversing the relationships.

The details of the API's HTTP messages are deduced from this specification using the standard patterns described in the HTTP specifications, plus a few conventions that we have added. Rapier thereby eliminates the need to repetitively document individual URLs and their methods, which vary only in the entities they accept and return or the queries they express.

Rapier is for specifying new APIs. You won’t be able to describe an existing API with Rapier unless that API uses the same conventions that Rapier does and is perfectly consistent in applying them.

A data-oriented approach to specifying new APIs

Rapier takes a data-oriented approach to API design, which aligns with the model of the world-wide web. If your mental model of an API is a network of HTTP resources identified and located using URLs, you should be comfortable with Rapier. If you think of a web API as a set of “end-points” with “parameters” (a traditional service-oriented or RPC model), the Rapier approach may not resonate with you. While Rapier APIs conform to the principles of REST, including the provision of hypermedia links, Rapier APIs do not require special clients that adapt to changing server data formats—most clients of Rapier APIs are quite conventional.

OpenAPI specifications

Because the Rapier specification language is an experiment and neither widely known nor adopted, we provide a tool that will generate an OpenAPI Specification (or OAS, formerly known as Swagger) document from a Rapier specification. The generated OAS document allows you to learn the precise details of the HTTP messages implied by the Rapier specification, the HTTP specifications, and our additional conventions. Generating OAS documents is also useful for integrating with tools that are based on OAS, or for communicating with people who know OAS but not Rapier. OAS remains important for documenting APIs that follow a service-oriented rather than a data-oriented design pattern, or follow different conventions than the ones Rapier currently understands, or are less consistent than Rapier APIs. Rapier is designed to complement, not replace, OAS.

Rapier also includes SDK generators for JavaScript and Python. In the future, we might work on test tools and server implementation frameworks.

Try it out

Rapier is very easy to understand and learn. The easiest way is by example. Rapier builds on top of JSON Schema, so if you aren’t familiar with that standard, you should spend a few minutes getting some level of understanding of what it looks like and what it does. Then you should be ready for this tutorial.


1Following Fred Brooks, we view consistency as being the primary measure of quality of an API. “Blaauw and I believe that consistency underlies all principles. A good architecture is consistent in the sense that, given a partial knowledge of the system, one can predict the remainder” - Fred Brooks, The Design of Design, 2010.

API-First Development with Apigee-127 (and Swagger-Node)

Chapter 3: More models for building APIs

This is the third in a series of posts examining the considerations that went into building an API to power our intelligent API management platform solution.

In the previous installment, I explored how we started to close the loop between API design and implementation and the role Swagger played. I also discussed a framework for building APIs in which the code defines the API.

In this post, we’ll look at two more models—one in which the code is generated from the API definition and one which has no API definition at all—and we’ll tee up a third approach that gives us the advantages of both.

Generated source code: the API generates the code

This category is represented by “IDL” (Interface Definition Language) systems such as SOAP, CORBA, and the many RPC systems that have been developed over the years.

In these types of systems, the interface is formally defined in a separate file, and then used to generate client- and server-side “stubs” that connect the bits and bytes sent over the network to actual code written by a developer. Developers on both sides then incorporate their stubs into the source code that they build and ship.

The advantages of the generated-code approach are performance, and simplicity for the developer. Since the stub code is generated at compile time, the generator can take care to make it efficient and pre-compile it, and starting it up at runtime is fast since there is no need to parse the IDL. Furthermore, once the developer specifies the IDL, the interface that's needed to code to in order to make the stubs invoke the code is typically simple.

However, with generated stub code, the possibility of the code falling out of sync with the specification is still present, because the synchronization only happens when the code is re-generated. Good RPC systems make this simple, whereas more primitive systems that generate code and expect the user to modify it are much harder to maintain.

In addition, we have the same documentation problems that we had before. If we annotate the IDL to contain the docs, and then generate the “real” docs from the IDL, then how do we keep everything in sync? Do we re-generate the stubs and re-build the product just because we changed the docs? If we don’t, then will we miss the re-generation the next time we make a “real” change, and end up with clients and servers that don’t interoperate?

And dealing with generated code in a modern software development process is painful. Do you manually run the stub generator and check in the results? If you do, then once again you'd better not forget to run it, and better remember never to modify the generated code manually. Or do you make the build system run it on every build? That may mean creating and testing custom build steps for whatever build system you use.

One advantage of this mechanism is the performance gain received by building the stubs at build time, rather than when the system starts up. That made a lot of sense in the 1990s. But today, with CPUs immensely faster, the time needed to parse and validate an IDL file and generate a run-time representation is just not there, and languages like JavaScript (and even Java) are dynamic enough that they can work without loads of generate code.

Node.js itself is a great example of this—even a simple Node.js-based server loads and compiles tens of thousands of lines of JavaScript code when it is first started, and yet it's rare for the startup of a Node.js-based application to take more than one or two seconds. (And yet, even a simple Java app, fully compiled, takes seconds if not more to start—but I digress.)

No connection at all: the code is the API

Many other server-side frameworks, especially popular Node.js-based frameworks like Express, do not have the concept of an API description at all. In the Express framework, the developer simply wires JavaScript functions to URI paths and verbs using a set of function calls, and then Express handles the rest.

Here’s a simple example of code written in the Express framework:


// GET method route
app.get(‘/’, function (req, res) {
 res.send(‘GET request to the homepage’);
});

// POST method route
app.post(‘/’, function (req,res) {
 res.send(‘POST request to the homepage’);
});

These frameworks are very nice for quickly building APIs and apps without a lot of pre-planning and configuration. However, they don't offer any model for automatically generating documentation or for sharing the API design with a larger community.

So, you might ask, why not write code like above, which admittedly every Node.js developer in the world knows how to write, and then annotate it with Swagger later? Because by doing that, we end up with the API defined in two places—once in Swagger and the other time in the code.

Which one is correct? The code, obviously! But what if the code doesn’t implement the API design that everyone agreed on after carefully following the design principles? Now we’re back to the problem that we were describing at the very beginning of this series.

Based on our experience and on experience with popular frameworks, we proposed a third approach, where the API design drives the code.

We’ll explore this approach in the next and final installment of this series.

Photo: Moyan Brenn/Flickr

API-First Development with Apigee-127 (and Swagger-Node)

Chapter 1: How and why we built an API to power our platform

This is the first in a series of posts examining the considerations that went into building an API to power our intelligent API management platform solution.

Starting in in early 2011, we set out to re-design our product for managing APIs. The project had a lot of requirements based on our experience running our product on our own and in our customers’ data centers. One of the requirements would be to make sure that everything it could do was powered by a well-designed, intuitive API.

Furthermore, the bar for this API was pretty high. We’ve made a big deal at this company about APIs (hence the name) and have been clear that the most successful APIs are ones that are designed in a clear and consistent way. We knew that this management API would be used by everyone who uses our platform, which we hoped would be tens of thousands of developers, if not more. We also knew that we had a reputation to maintain, so the design had to be pretty good.

That meant following certain patterns that we had seen used in other successful APIs, and following them in a consistent way. For instance:

  • To create a new “application,” POST to a URI named /applications
  • To get a particular application, GET from a URI named /applications/{name}
  • To delete an application, DELETE that same URI
  • To update that application, PUT to the same URI

There are other aspects to the pattern, but it was one that's well understood around the industry and which, if done consistently, would result in an intuitive API that didn’t require a huge cognitive load. Once you understood “applications,” the same patterns would work for “developers” or “messages,” and so on.

Building the API

Now the next step was to build it. We had a small but effective team located in Bangalore that was tasked with building this new platform. While they were devoted to building a great product, they didn’t have as much experience with our particular way of designing APIs as I did, nor did they feel as much passion about it. More importantly, they had many more urgent things to do than debate the finer points of the design of each API. Plus, they were 12 (and a half!) time zones away.

I felt that I had to design the API myself. I ended up doing what generations of API designers did, which is to open up a text editor and start typing. I ended up writing up little snippets like many others have done:

GET /applications Return a list of the names of the applications

GET /applications/{id} Return the application with the specified ID

POST /applications Create a new application

And so on.

The developers then did what they did, which was to turn those high-level descriptions into APIs. We were building this API in Java, so the code looked a bit like this:

@GET

@Path("/applications")

public List<String> getApplications() {

// Blah blah blah

}

@GET

@Produces({"application/json","application/xml"})

@Path("/applications/{id}")

public Application getApplication(@PathParam("id") String id) {

// Blah blah blah

}

@POST

@Produces({"application/json","application/xml"})

@Consumes({"application/json","application/xml"})

@Path("/applications")

public void createApplication(Application app) {

// Bleh bleh bleh

}

So far so good—we have an API!

The disconnect

Of course, things didn’t always work out so cleanly. For instance, a developer might end up wanting to add a method to get all the applications, but to get all the details, not just their names.

According to our API design, this would happen by adding a query parameter called “expand” to the “/applications” URI. A GET with “expand=true” would return more information than one that did not.

The developer, however, might have a different idea, and instead add a completely new URI like this:

@GET

@Path("/getApplicationsWithDetails")

public List<Applications> getApplicationsWithDetails() {

// Blah blah blah

}

(Note: Our actual developers understood our standards better than that and wouldn’t have done something that far from our design principles, but I’ve seen things like this happen elsewhere.)

As the product grew, and as the team grew, these kinds of problems kept cropping up.

We eventually grew to realize that API design is like user interface design. Just as we would not roll out a new user interface without letting our user interaction people work on it, and wouldn’t put something on the corporate web site without approval from our chief graphic designer, we would not want to put out an API without input, or even approval, from the small, passionate group of API experts on our product team.

But that didn’t happen. New APIs kept cropping up, and the PMs and others would ask questions like, “did you know about that API? Why did they decide to call it ‘audit’ rather than ‘audittrails’? Why is that a POST and not a DELETE?” And so on.

Plus, we didn’t have an effective way to decide on the API design before the code was done and pushed to production. In lots of cases, if we changed it “now” we’d have to stop-ship the whole release, or go back to customers and tell them that we made an incompatible change to an API which would break all the scripts that they wrote, not to mention our own UI and other tools.

The documentation

What made things extra difficult was that we’d now have to go back and document these APIs. Since we had built them in Java and the truth was in the code—and nowhere else—that meant the process was expensive:

  1. Either the (expensive) engineers would have to sit down and write out, in a text doc of some sort, exactly what the API did, what all the parameters were, example inputs and outputs, error conditions, and so on. Some engineers are also great writers who love to describe what they did, but that’s not everyone. Furthermore, documenting every aspect of every parameter is no fun for anyone to have to do more than once.

  2. Or the (expensive) technical writers would have to read the code and figure those things out, which slowed them down since the answers were not always obvious and required asking more questions of the engineers.

  3. Or, someone like me would have to read the code, read the docs, and do a two-way mental diff until we felt that the docs were correct enough.

But Roy Fielding said ...

The API design principles I just described are popular in the world of APIs, and in my experience represent what nearly every developer today recognizes as an “API.” The term “REST” is often used to describe such APIs, but that’s not what “REST” is really supposed to mean.

It is popular in API design circles today to advocate for “hypermedia-driven” APIs, which are constructed just like web pages, as a series of documents containing hyperlinks. The more extreme proponents of hypermedia claim that documenting such APIs is unnecessary, because a smart client will discover what they do by following links, just as a human discovers what a web app does by looking at rendered HTML pages.

While the APIs described here are not hypermedia-based and thus aren’t actually “REST,” I believe that the principles are the same. The idea that a set of hypermedia links in an API response obviates the need to create API documentation for humans just doesn’t make sense.

At this point, we knew that we could come up with something better—something that adhered to our central tenets of API design and helped us solve some of these problems.

In the next installment of this series, I’ll describe how we started to close the loop between API design and implementation, the role Swagger played, and alternative frameworks for building APIs.

Photo: webtreats/Flickr