REST APIs

API Design Best Practices & Common Pitfalls

Webcast replay and Q&A

The job of an API is to make the application developer as successful as possible. When crafting APIs, the primary design principle should be to maximize application developer productivity and promote adoption.

So what are the design principles that help optimize developer productivity? I recently presented some ideas about this in a webcast (you can watch the replay here).

I was pleasantly surprised by the discussion that my talk sparked; many interesting questions were asked, so I thought I’d share some of them (and my attempts to answer) here. 

(Editor's note: Some questions were edited for clarity)

How does HATEAOS fit into pure “HTTP” APIs?

I have seen different interpretations of what HATEOAS means in the context of APIs. One interpretation leads to the practice of trying to describe all the actions that can be performed on a resource in the representation of the resource. For example, if I did a GET on http://etailer.com/orders/1234567, then, in addition to describing the order itself, the JSON that I got back would try to describe all the actions I could perform on that order (cancel the order, re-order the goods, track the shipment, ...).

This is not commonly done, and I myself do not design APIs that work this way. The JSON I design only describes the order itself, including any relationships it has to other entities expressed as URLs. I make the assumption that it is the job of the client code to know what actions it wants to perform and how to perform them [using standard HTTP methods, of course]. This is how most clients are written in practice. Even the modern browser works this way, since operations are now usually coded in Javascript executed in the client rather than using old-school HTML forms prepared on the server.

Does this mean that I am violating the HATEOAS constraint of REST? I'm not sure, but I don't see a reason to worry about it. I don't make any claims for my APIs regarding REST compliance; I simply try to use HTTP as simply and directly as I can and avoid all invention where HTTP already specifies a solution (which in my experience it usually does).

Can you speak about API layering (experience APIs vs others)? How do you avoid "spaghetti APIs" over time?

I'm not fond of layering in general, although I recognize that you sometimes need to do it. It is common for companies to have some sort of "generic" API for their problem domain, and end up layering other APIs on top of it. For example, assume I'm in retail and I have APIs for catalog and orders. The mobile app team looks at my API and decides they don't like it for mobile development, so they put another server in front of the generic one that implements its own API and delegates onto the generic one.

So now which of the two APIs should others use? Once a few teams have done the same thing, it's not clear anymore which is the real API, if any. Some people make a virtue out of this (hence the concept of experience APIs), but I like to minimize layering. If the mobile team needs function that the generic API does not have, they can extend the generic API (possibly in their own server), but ideally they should not create a new layer on top.

How can one indicate specific error conditions when no HTTP response code is a good fit, or is not fine-grained enough?

I pick the HTTP response that is the closest fit, and also return a body with more information. I don't know of an accepted standard for the body format, but standards have been proposed, e.g. https://tools.ietf.org/html/rfc7807

How do we manage resource mapping and up to what depth it's good to go—like /users/{Id}/orders/{oid}/articles?

What you are doing here is inventing a query language. This query says something like "SELECT * FROM articles, orders, users WHERE user.id = $1 AND order.userID = user.id AND article.orderID =$2". This query is not optimal if {oids} are unique across all orders, because then the query can be reduced to just /orders/{oid}/articles. Designing your own query language is hard, which is one of the reasons that GraphQL has got attention.

Personally, I don't like the idea of encoding queries in the path portion of a URL rather than in the query string, because it encourages people to confuse query with identity lookups. But many people do what you are doing, so I won't claim it is wrong. I also used to do it before I had a change of heart.

For POST, can you speak to the pluses / minuses of query string and JSON body?

Putting queries in a query string and using GET to evaluate the query is attractive and a good fit for HTTP. An example is GET /pets?{your query here}. Unfortunately there is a practical limit on the size of a URL: if you go above about 4k characters you run the risk that some proxy in the chain between a client and the server will mess up the request or reject it. Because of this, I always offer both GET and POST options for the same query.

Would it be a generally decent design if we just expose a single API point (e.g. me.com/api) for all communication and use JSON as the payload?

Endpoint is not an HTTP concept, but it is fine if all your well-known URLs (and even all the dynamically-allocated ones) begin with the prefix me.com/api. The important thing is that every URL should identify some resource. If you have any URL for which you could not easily answer the question "what resource does this URL identify, and therefore what would it look like if I performed a GET on it", then you are probably not working within the HTTP model.

I understand the limit on linkability rationale for eliminating a version number in a URL, but what would your strategy be then for handling version differences?

See this blog post on versioning.

How important is it to strive for the consistency of the API design in terms of resource/entity planning, error message standards, header extensions, etc?

There is a nice quote from Fred Brooks on this:

“Blaauw and I believe that consistency underlies all principles. A good architecture is consistent in the sense that, given a partial knowledge of the system, one can predict the remainder.” - Frederick P. Brooks, Jr., The Design of Design: Essays from a Computer Scientist, 2010.

In other words, consistency is paramount. The easiest way to get consistency is to just use HTTP without adornment or invention. Where HTTP does not provide answers (this is less common than many people think), try to pick one solution and stick with it.

Is there any concept of statuses in RESTful APIs (viz. draft, dev, test, released, obsolete)? How do you implement this lifecycle of statuses? Is there any documentation on this?

HTTP does not address this—HTTP would view this as part of the modelling of your problem domain, and therefore out of scope of HTTP itself. One piece of guidance would be "don't put the state (or status) of a document into its URL, because that will change".

See this article for more.

What are your thoughts on using an API gateway as an internal enterprise integration hub/gateway?

We have many customers using Apigee Edge as both internal and external hubs/gateways. This is also an investment area for us. If this is an important topic for you, you should ask for a presentation/briefing focused on the topic.

You said that you implemented something similar to GraphQL? Can you share what made you implement that?

The API had a set of well-known URLs of the form /widgets, /thingummies, /doohickeys. We wanted to offer URLs of the form /widgets?{query}, /thingummies?{query} and /doohickeys?{query} and we also needed to offer /query?{query} for queries that "join" across resource types. We had two needs: define a syntax for queries and provide an implementation. We looked at GraphQL, but we were nervous of a design that runs a complex query engine in application space and relies on primitive APIs for raw data access. We have no objective evidence that GraphQL would have been problematic, but the idea made us nervous.

We designed our system so that it stores a copy of all the data in a set of read-only tables in a standard database system with replication and scale-out. This allows us to push the queries down to our database rather than implementing them in a system like GraphQL. This option will not be open to everyone, because you can't always get all the data into a database. If you can, it enables the queries to execute on a standard database query engine and, more importantly perhaps, execute very close to where the data is stored.

We designed a fairly simple query language that happens to be conceptually similar to GraphQL's (although its design predates our exposure to GraphQL) and a simple processor that translates these queries into the query language of our database. Our language is not as rich as GraphQL but has proven very effective. The whole thing is very simple, and has worked well with good performance. Creating the right indexes on the database table(s) can take a little thought, but our experience is that a few well-chosen indexes enable decent performance on a wide range of queries.

Perhaps the biggest lesson we learned is that having a good query capability (in addition to standard HTTP CRUD, of course) is very powerful—people have done all sorts of interesting things on top of our API without ever having to talk to us or request new features. This has also helped avoid the need for "experience APIs" (see response above) layered on top of our API.

If versions are not part of links, how do we make links to new resources existing in v2 but not v1? Or how do we link between multiple APIs—is this making an assumption that the versioning is done at the header level?

Links are always written using URLs of resources that do not contain version numbers. If clients want to request a specific version of a resource, there are two choices. The first is to allow clients to provide an Accept-Version header in their requests. The second is to provide clients with a rule for transforming the URL of a resource into the URL of one of its versions.

Since we can't get people to agree which one they want, we just implement both in the APIs I work on. In my experience, the energy required to implement both is much less than the energy required to argue about it. They both make sense in the HTTP model. I personally prefer the first approach (header), because it doesn't require clients to learn a "rule" that is specific to our application for transforming URLs. The argument for a header would be stronger if someone would standardize Accept-Version so it could be referenced by all applications.

You mentioned the risks of coupling an API entity model to its domain or data model. Can you talk more about those risks and suggest some ways to mitigate them?

The model you expose through an API is the conceptual model of the problem domain as seen by a client. The actual storage model is often more complex than the conceptual model, but it doesn't always have to be. Decoupling them is useful because it allows you to keep the conceptual model simple even if performance or other concerns force compromises in the storage model.

Do you have advice on the use of patterns like idempotency and upsert vs. discrete CRUD?

If you are writing a microservices application rather than a monolith, you will probably face the problem where a single conceptual create, update, or delete requires changes to state stored in more than one microservice. I have not found a very simple way of doing this reliably. My current approach relies heavily on idempotency, but also requires a sort of application-level two-phase commit strategy. It works, but it's not simple.

HTTP has standard support for upsert—it is called PUT. I used to rely on POST for create, and either ignore PUT completely in favor of PATCH, or (shame on me) implemented only half the function of PUT (update but not create). Recently I have been working on some APIs where there is a requirement that clients be able to synchronize the state of the API from a set of external files. This has given me a new understanding of the value of PUT.

What factors should be used on breaking the resources into APIs? Does 100 resources mean 1, 2, 5, or 100 APIs?

As I said above, the word API is a slippery word. In HTTP there are only resources. A simple strategy is to have one URL for each type (e.g. /dogs, /people) plus one URL for each instance (the format of these URLs does not have to be specified unless you allow create via PUT). How many APIs is that? 1? 2? 102? Somewhere in between? I'll let you decide.

For more on API Design, read the eBook "Web API Design: The Missing Link."

 

Why Your Web APIs Should Be Entity-Oriented

The dominant model for APIs in distributed computing for decades has been Remote Procedure Call (RPC). This isn't surprising—ever since Fortran II introduced functions in 1958, the function  (or procedure) has been the primary organizing construct that programmers use to write code.

Most distributed APIs are defined by programmers, and the simplest way for them to think about a distributed API is that it allows some of the procedures of their program to be invoked from outside the program. Historically, systems like DCE and CORBA provided system software and tools to help implement RPC.

When people started implementing APIs on the world-wide web, they naturally carried over the familiar RPC concepts from previous environments. This led initially to standards like WSDL and the so-called WS-* standards, which were heavyweight and complex. Most web APIs now use HTTP in a much more lightweight way—often called RESTful—that retains concepts from the RPC model while blending in some of the native concepts of HTTP/REST.

HTTP itself is purely entity-oriented, not procedural—it defines a small number of standard “methods” for manipulating entities, but otherwise does not model procedures. A minority of web API designers have abandoned the traditional RPC model completely and design web APIs that are based entirely on HTTP's entity-oriented model; they are using HTTP simply and directly, without layering RPC concepts on top of it. In a moment, I'll explain why they are doing this.

At this point, you might be confused, because much of the available information on the web would lead you to think that the current crop of popular web APIs follows the entity-oriented model called REST. For example, the introduction to the OpenAPI Specification (formerly known as Swagger) says "The goal of the OpenAPI Specification is to define a standard, language-agnostic interface to REST APIs."

In fact, OpenAPI is a fairly traditional RPC Interface Definition Language (IDL) and describing an entity-oriented API with OpenAPI is awkward and imprecise. The fact that OpenAPI can fairly easily and accurately describe the majority of the APIs currently on the web is a reliable indication of their nature. There have been some attempts to define IDLs for entity-oriented APIs—Rapier is an example. One way to understand the challenges of representing an entity-oriented API in OpenAPI is to look at the output of Rapier's OpenAPI generator.

Why entity-oriented APIs matter

So why would you care about this? Why are some people interested in entity-oriented rather than procedure-oriented APIs? Imagine an API for the classic students and classes problem. In a procedural API, I might need the following procedures:

  • add a student record
  • retrieve a student record
  • add a class record
  • add a student to a class
  • list classes a student is enrolled in
  • list students enrolled in a class
  • assign an instructor to a class
  • transfer a student between classes

In practice there would be dozens of these procedures, even for a simple problem domain like this one. If I looked at the API for another problem domain, I would start over again—nothing I learned about students and classes will help me learn the next API, whose procedures will all be different and specialized to its own problem domain.

What’s wrong with that?

Most programmers are not surprised or dismayed by the proliferation of APIs with little commonality between them—learning all this detail and diversity is simply part of the life of a programmer. However, even programmers have a different expectation when they program to a database. Database management systems (DBMSs) offer a standard API for interacting with data regardless of the details of the data itself.

If you are programming to MySQL, PostgreSQL, or any other DBMS, you have to learn the schema of the data you are accessing. But once you have done that, all the mechanisms for accessing that data—the API—are standardized by the DBMS itself. This means that when you have to program to a different database, the learning burden is much lower, because you already know the API of the DBMS; you only have to learn the schema of the new data. If each database, rather than each DBMS, had its own API, even the most tolerant programmers would balk.

Implementing a purely entity-oriented API on the web enables HTTP to function as the standardized API for all web APIs, in the same way that the API provided by a DBMS functions as the standardized API for all the databases it hosts. HTTP becomes the universal API for all web APIs, and only the schema of the data of a specific API needs to be specified and learned.

Separation of interface from implementation

Separating an entity-oriented API model from the procedural implementation model has another major advantage—it makes it easier for each of them to evolve independently. This can be done in the procedural model too, by having one set of procedures for the external model and a separate set for the implementation. However, maintaining this separation when they are both expressed as programming-language procedures requires a lot of discipline and design oversight, and is rarely done well.

Why the extra effort is worthwhile

If entity-oriented web APIs are better, why is only a minority of web APIs designed this way? There are multiple reasons. One is that many programmers don't yet know how to do this, or they don't know why it's better. A second reason is that entity-oriented APIs require a bit more work to produce, because implementing an entity-oriented API requires programmers, whose code is in the form of procedures, to implement a mapping from the exposed entity model to the procedural implementation.

The mapping isn't inherently difficult—it consists mostly of implementing create, retrieve, update, and delete (CRUD) procedures for each entity, plus a set of procedures that implement queries on the entities. Many API developers start from this point but stray by exposing procedures that do not correspond clearly to a standard operation on a well-defined entity.

A third reason is that most of the popular API programming education, tools, frameworks, and examples illustrate the procedural style or a hybrid style—not a purely entity-oriented style. Staying true to the entity-oriented model requires a little more effort and mental acuity, but most programmers are neither lazy nor stupid; what is usually lacking is an understanding of why a little extra effort is worthwhile. In short, although it is not terribly hard, you have to have some vision as motivation to implement entity-oriented APIs.

The popularity of entity-oriented web APIs is increasing slowly. Some widely used APIs, like the Google Drive API and the GitHub API, are almost completely entity-oriented. Others have understood that entity-oriented interfaces can be constructed for almost any problem domain. I believe the industry will continue to move in this direction.

For more on API design best practices, read the eBook, “Web API Design: The Missing Link.

Image: Flickr Creative Commons/webtreats

Rapier: Cut Through the Tedium of API Specification

At Apigee, we're big believers in the importance of API metadata for all aspects of the API lifecycle—we were one of the early promoters of Swagger and a founding member of the OpenAPI Initiative. We never stop thinking of ways to make API design and development better, and to that end, we've set up an Apigee Labs organization on GitHub as a place to incubate new ideas.

Today, we want to share with you a new project that we're working on—Rapier—that is a proving ground for some of these ideas. Some are ideas that we may propose for inclusion in future versions of the OpenAPI Specification, and others may become part of Apigee's roadmap, but for now, we just want to get these into a wider discussion and solicit feedback in the spirit of open development.

Rapier is a new API specification language created by Apigee. The goals of Rapier are to allow REST APIs to be specified and learned with one-tenth the effort required with other API specification languages, and to produce specifications that describe higher-quality APIs1. Rapier is published in this public repository under an Apache 2.0 license—it will be managed as an open source project.

The name Rapier is an acronym for “REST APIs from entities and relationships.” With Rapier, you specify an API in YAML by specifying the entities and relationships of the data model that underlies the API, along with query paths traversing the relationships.

The details of the API's HTTP messages are deduced from this specification using the standard patterns described in the HTTP specifications, plus a few conventions that we have added. Rapier thereby eliminates the need to repetitively document individual URLs and their methods, which vary only in the entities they accept and return or the queries they express.

Rapier is for specifying new APIs. You won’t be able to describe an existing API with Rapier unless that API uses the same conventions that Rapier does and is perfectly consistent in applying them.

A data-oriented approach to specifying new APIs

Rapier takes a data-oriented approach to API design, which aligns with the model of the world-wide web. If your mental model of an API is a network of HTTP resources identified and located using URLs, you should be comfortable with Rapier. If you think of a web API as a set of “end-points” with “parameters” (a traditional service-oriented or RPC model), the Rapier approach may not resonate with you. While Rapier APIs conform to the principles of REST, including the provision of hypermedia links, Rapier APIs do not require special clients that adapt to changing server data formats—most clients of Rapier APIs are quite conventional.

OpenAPI specifications

Because the Rapier specification language is an experiment and neither widely known nor adopted, we provide a tool that will generate an OpenAPI Specification (or OAS, formerly known as Swagger) document from a Rapier specification. The generated OAS document allows you to learn the precise details of the HTTP messages implied by the Rapier specification, the HTTP specifications, and our additional conventions. Generating OAS documents is also useful for integrating with tools that are based on OAS, or for communicating with people who know OAS but not Rapier. OAS remains important for documenting APIs that follow a service-oriented rather than a data-oriented design pattern, or follow different conventions than the ones Rapier currently understands, or are less consistent than Rapier APIs. Rapier is designed to complement, not replace, OAS.

Rapier also includes SDK generators for JavaScript and Python. In the future, we might work on test tools and server implementation frameworks.

Try it out

Rapier is very easy to understand and learn. The easiest way is by example. Rapier builds on top of JSON Schema, so if you aren’t familiar with that standard, you should spend a few minutes getting some level of understanding of what it looks like and what it does. Then you should be ready for this tutorial.


1Following Fred Brooks, we view consistency as being the primary measure of quality of an API. “Blaauw and I believe that consistency underlies all principles. A good architecture is consistent in the sense that, given a partial knowledge of the system, one can predict the remainder” - Fred Brooks, The Design of Design, 2010.