Personally, I deal with fairly large and complex business software like ERP, CRM, warehouse management, manufacturing execution, data warehouses, etc. Most of these applications have very complex (mainly relational) data models, that are built around business objects: customers, orders, etc.
In the upcoming couple of articles, I would like to share my experience and give an overview of modern open web API standards from the point of view of business applications. Using a standard API is always a good idea, but which one to choose? Before we dive into JSON, XML, etc., let's take a closer look at the requirements and their origins.
In a nutshell, a web service interface for an app dealing with business objects must primarily enable a remote application to read, write and and perform other actions upon these business objects. Reading must be flexible (i.e. providing filtering, aggregations, etc. but always respecting access rights), writing must be safe (i.e. handling nested data, bulk operations, access rights again, and so on) and all other operations must work in a concisten manner.
It is important, however, to take into account, what type of clients the API will mainly serve:
- GUI clients like web apps typically need user-configurable read-rquests with all sorts of filtering and sorting. Pagination is important to keep responses small and fast. They will also often need nested data in the response (e.g. a product with all its variants or an order with it's positions) to decrease the number of network requests required.
- Batch-clients like background processes or import/export-services need tabular data with some predefined filters, but with lots of items per request.
Last, but certainly not least, it is also very important to make the web service easy to understand for humans - namely those, who are going to develop the clients at the other end. This is especially important if the data model is complex - and that is often the case! Thus, documentation, schemas, etc. must be part of a good web service too!
Reading business objects
Modern web APIs should provide a more-or-less generic read-service for every business object. In REST services it is a URL, in GraphQL - a query, etc. This service yields one or more instances of the business object depending on the parameters passed.
In reality read operations mostly happen for core business objects like orders or customers. However, the client will probably be interested in data from all sorts of satellite objects: order positions, customer groups, etc. Together these objects produce far more data, than a typical web-consumer can process at once, so we also should give the client tools to request the really needed data efficiently.
Filtering and searching
Pratcically all reading APIs need filtering. A web API is no different: it will need some filter syntax with predicates consisting mostly of a business object's attribute, a value to compare to and the comparison operator (=, <, >, etc.). If you need complex filters with multiple predicates, pay attention to the support of logical operators like AND, OR, NOT, etc. It might seem obvious, that an API should support things like that, but don't take this for granted: even popular standards like GraphQL or JSON:API do not include filtering syntax!
Searching for keywords is a similar requirement at first sight, but technically quite different. The search term mostly needs to be compared to different properties of multiple objects and the results should get ranked somehow to show the most relevant result first. It is uncommon for API standards to provide a specification for search queries.
Pagination
Requesting the first N results only and gradually moving forward "by-page" helps improve performance a lot (if this pagination is done by the server, of course). This is only a requirement for UI-type clients, however.
There are multiple pagination strategies:
- by offset: e.g. request 10 rows, skipping the first 30,
- by id: e.g. request 10 items starting with item id X, or
- by cursor: e.g. request 10 rows starting with row id Y.
Most web API support the offset strategy. In my opinion, this is enough for the absolute majority of use cases.
Also note, that UI controls, that support pagination will often display the total number of records found. If this is an important feature, make sure to include this number in the read request instead of using a separate count-request as the number of records may change between two requests. Remember: it is practically impossible to span a transaction across multiple HTTP requests!
Sparse field sets
Business objects often have lots of attributes, while UIs can mostly show only few. Knowing that UI-clients often need data from multiple objects at a time, it becomes obvious, that sending all the data for each of thoes objects over the network is an absolute overkill. Instead, give the clients a possibility to specify, which attributes to read. This adds some complexity to the server implementations, but this feature is important enought to be one of the key reasons for the popularity of GraphQL!
If you expect lots of tabular or graph UIs as clients, make sure your API supports sparse field sets in a comfortable way: the developers of the clients will probably need to use it often!
Another case, where a good implementation of sparse field sets is important, are "heavy" attributes that need a lot of calculations. For example, you may want to show current list prices for order positions. Those prices can depend on various customer, supplier and material data, price lists, time, etc. They take time to calculate, so always loading them with order positions by default would probably kill the performance. It is much better, if they can be excluded from the field set (or need to be explicitly included).
Nested data, joins
Both types of clients will often require data from multiple business objects (e.g. orders and their positions). Putting it all into a single request is definitely good for the performance.
However, combining data can be done quite differently!
- In a master-detail scenario, like orders and positions, you request master data and need to embed the details into each master item. This produces hierarchically nested data, which is a typical requirement for UIs like web shops, catalogues, etc. On the other hand, hierarchies are not so good for tabular clients though, because they will need to "flatten" the data to put it into a table.
- Another use case would be adding order data to each item in a list of purchases (= order positions): this corresponds to an inner join in SQL.
- Yet a different challenge are cross joins: e.g. showing product data with prices from all price lists - a Cartesian product, that is.
While most web APIs support nested data while joins are rare. This is understandable: joining data of multiple business objects requires either deep knowledge of the underlying data on client-side or a fairly complex server-side implementation.
Note, however, that most requirements can be solved in different ways. Instead of inner joins we can easily add more attributes to an object or even a nested attribute set (e.g. embed all required data of an order into each position). This helps to keep all the relation logic hidden in the server. For cross joins we can add a separate business object (product-prices in our example), which will even be easier to understand for a client-side developer. Indeed, a separate attribute or object can be easily described in the docs, while describing results and conditions for different joins is way more complicated.
There are, of course, special cases, where flexible (cross) joins are important: mainly user-configurable reports like sales per store, uptime per server, and other pivot-type analysis. These are quite special however, so I would not try to press them into a standard web API at all costs. If the reporting part is really important, take a look at specialized APIs like XMLA.
In any case - with nested data, joins, etc. - all the relation logic (keys, constraints, etc.) should be kept private within the server. It's not the client's business!
Aggregating data
Having a sum in the footer of a value column in a table is quite a common requirement, but for some reason, standard web APIs seem to ignore it: there is no commonly used option to tell the server to add a total to some part of the read data.
Aggregations and total rows calculated on server-side are mainly important for clients using pagination (without pagination, the client can easily calculate the total itself). The total is expected to include all the data while the actual result only contains one page of it. The challenge here is similar to displaying the total row count in a paged list: it would be great, to put the total into the same response, as the data for performance and consistency reasons.
Flexible aggregation is also useful for reporting UIs with configurable charts, because charts often display aggregated data (e.g. the sum of produced quantity per day or so). In this case, however, the aggregation is applied to the data directly instead of being appended to it. Nevertheless, the API will need to provide a way to tell the server, what and how to aggregate.
Batch requests
Bundling multiple reads in a single batch request is a good idea if
- you need to read multiple datasets isolated in a transaction
- you have many (>5) lazy-loading widgets on a page, that use different (non-related) data and would cause too many requests when loading separately.
Batch reads are quite rare among standard web APIs and also not easy to handle on client-side because they add additional dependencies between widgets (e.g. you could only refresh all widgets at once).
Non-standard reads
There are situations, where you need to fetch a set of business objects based on some information, which is not directly part of the data model - e.g. a list of stores based on the user's location. This can't be done by setting a couple of filters for the general store list - it requires some use-case specific logic. In these occasions, a separate remote function (RFC/RPC) helps. We will talk about RFC-type services later on.
Writing business objects
When writing business objects via web APIs (i.e. create, update, delete operations), the data should be sent in the same format, as it was read from the server. This enables editors on client-side to be simple forms without much logic. Even external tools like Excel can be used, which do not know anything about the business logic at all.
Validation
In my opinion, web APIs must always make sure, the received data is valid. The client may do it's own validation, but ultimately, the server is responsible for it. After all, if the data becomes corrupted, it's kind of strange to blame a remote client that basically just provides a wrapper for the interaction.
API formats do not affect validation explicitly, but their schemas and metadata do. If you publish information about data types and constraints, your clients can help validating. Especially GUI-clients benefit from the possibility to display specialized controls or at least hints for the user to know, what kind of input is expected.
Partial updates
From my point of view, a web service must be able to accept partial field sets for updates on objects: for example, update just a single attribute - a status or so.
This is not so much a requirement for the API format, but rather one for the implementation. Still, it's very important and sometimes not easy to implement - especially if you think of web forms, where distinguishing between "empty" and "not filled out" is not trivial.
Mass update/delete
While a partial update changes few attributes of a given object (mostly identified by it's key), a mass update would set values for all objects matching a filter: e.g. cancel all orders, that were not confirmed within 3 days. This is particularly useful in UIs that work with tables and filters: the user can then easily perform an action on a set of objects defined by the current filters.
To my knowledge, such requests (with data and filters at the same time) are uncommon for standard web APIs: I can only think of OData 4, that has this feature. If you need this functionality in another web API, you will need to create custom remote functions (see below) and think of a suitable format yourself. One thing to take special care of - even with OData - would be error handling: a mass update can produce many different errors. An important question would also be, whether to cancel the entire operation after the fires error (i.e. wrap the updates in a transaction) or not.
Mass deletes as well as other operations to be performed on all objects matching a filter, may be very useful to. Technically, they are not much different than updates discussed above.
Batch requests
Similarly to batch reads discussed above, it is sometimes very useful to put multiple writing operations into a single requst - especially, if they need to be wrapped in a transaction.
In contrast to mass updates, batch requests need to be explicitly supported by the service's format. This is not a standard feature for web API, so pay attention to it, if you deal with multi-object forms, in-table editing, etc.
Locking and concurrency detection
Business applications practically always need multiple concurrent users. This means, proper locking is important to prevent concurrent writes on a single object. Dealing with complex hierarchical objects like orders makes the whole thing even more difficult as these objects actually consist of multiple separate objects.
Due to the asynchronous nature of the web, the most wide spread locking strategy is optimistic locking (checking for conflicts right before writing in contrast to locking when reading referred to as pessimistic locking). As with other types of validations and checks, the server is responsible for locking.
With optimistic locking the client receives some sort of validation key that it must send back to the server so it can check if conflicting changes occurred. In the simplest case, the key is just the timestamp of the last modification of the data. The HTTP standard even includes a special header for this: the ETag. It sure is a good idea to use it. However, neither the header specs nor any standard APIs define, what exactly the content should be: a timestamp, a hash, etc. - this is up to your implementation. Using the ETag also assumes, each request contains only a single lockable object. For mass updates, you will need to invent your own solution.
Pessimistic locking can be done too - also on server-side. You will need categorize read request in those for writing and read-only ones. A read-for-writing should trigger a lock on its object. The tricky part is releasing the lock: you never know if or when the client will actually deal with the data and send it back. Therefore, try to avoid pessimistic locking in web APIs.
Remote function calls
Until now, we have dealt with CRUD-operations (create, read, update, delete) for business objects. There are, of course, all kinds of business logic that need to be part of an API. Typical examples are creating an order from shopping cart items, confirming an order or any other status changes of complex objects, reserving stock, getting stores close to given coordinates, etc.
In contrast to CRUD, the API for such business logic is more general: you simply call some sort of function/procedure remotely providing the required input parameters and get some output. Each remote function call (RFC) has its own input and output, w hoch can be simple values (like coordinates), one or more business objects, or anything else. In other words, the client muss know more about the internal logic of an RFC, than about a CRUD-operation.
Try to avoid complex input and out parameters, that are not business objects: e.g. instead of using an array of tuples of product id, quantity and price as input for creating an order from a shopping cart, introduce a business object for a cart item even if it does not exist in the persistence layer. This makes RFC a lot more understandable! Therefore, it is important, that the web API can handle business objects as RFC-parameters easily.
File uploads and downloads
While downloading a file is pretty straightforward in most web APIs (just a URL), uploading is less obvious. Especially if we are talking about UI-clients with asynchronous AJAX-uploads (Gmail-style). User-friendly features like multi-file uploads, image preview, etc. mostly needs to be supported on API-level.
While few API standards define handling media explicitly, your specific API spec should do! Take a look at the corresponding OData section for an idea about how it can be done.
Transaction handling
I don't think, there are common web APIs that allow explicit transaction handling. Transactions are business of the server. A web API is merely a remote control. In most cases, all operations in a request are wrapped in a transaction.
If you need multiple requests to be wrapped in a single transaction, better create a separate RFC. Another option would be using batch requests, but that would surely be more confusing for clients: how to know, which parts of the batch are going to be rolled back on failure?
Caching
Caching API-requests does not only improve performance, it is also vital for progressive web apps (PWA) that use service workers and browser storage to make data available offline.
Browsers cache Get-requests automatically based on their URL and special headers (e.g. the ETag, that was already discussed above), that can be used to control cache validation. The same technique is recommended for PWAs, where the programmer is responsible for the implementation, however.
So, if the API uses GET-requests for reading, caching data and even making it available offline is pretty much straightforward.
Discoverability and documentation
Apart from the technical requirements, an API needs good documentation to be truly useful. The developers of the clients need to understand the business objects, their attributes, relations, actions, etc. Unfortunately, documentation is often underestimated in software development projects. But for a web API it is especially crucial, because in contrast to graphical user interfaces, there are now power users that work with it daily and build up extensive knowledge over time.
In my opinion, plain-text docs are not enough. Every API needs structured metadata - ideally a schema that is involved in testing or even productive use to ensure, that it is always kept up to date. Apart from documentation, such a schema can be used for validation, live exploration, code-generation, generic clients and much more. In fact, I would pick a schema over written docs if I had to choose.
Some web API standards like OData or GraphQL provide their own schemas. Others can be described using the common API specification standard called OpenAPI (formally Swagger).
Hyperlinks (HATEOAS)
Hypermedia As The Engine Of Application State (HATEOAS) is an important part of RESTful architecture, which basically means, that the server provides URLs for actions upon a business object right built into the data of that object: for example, a link to a service showing all the details or forward/backward-links in paged lists.
Hypermedia is not part of the object's data, but it helps discover it. A client can interact with the API using hypermedia without the need to know the structure of the API in advance.
While being a good tool to further loosen the coupling between client and server, hypermedia does not replace metadata and schemas!
Versioning
Even if designing an API from scratch, it is worth to think about introducing changes in more distant future. Most larger web APIs simply use different URLs for every major version of the API: for example http://domain.com/api/v1, http://domain.com/api/v2, etc.
Changes within a major version are typically considered backwards-compatible, which means that services or fields are never removed. Additions often simply get a comment in the documentation containing the exact number of the version, where they were introduced.
A less common approach involves adding deprecation information to the metadata: like in GraphQL. Although being more flexible, this assumes, that clients are kept more or less up to date. From my point of view, this is only possible for domain models, that practically never change, or internal APIs, where all clients are known and can be forcibly updated.
Error handling
Errors will happen. Clients will need to either deal with them or pass them on to the user. In both cases, well structured and informative error data is very important! An error should contain technical data (i.e. an error code) and a human-readable description. Unfortunately, API standards do not provide many recommendations upon error handling, so you will probably need to think of a suitable format yourself. Nevertheless, error handling should be always part of your specs, docs and requirements!
Things get complicated, if you need to deal with non-critical errors. A typical scenario are batch requests, where only parts of the operations fail. The question here is wether to make the entire request fail (and concequently roll back eventually performaed changes) or send error messages to the client along with other data - possibly for multiple different errors! How to handle these issues really depends of your specific business logic and even on the capabilities of the server. Think about error handling early in the project to avoid the need to change the response format later on.
Using API standards
As stated at the very beginning, personally I like the idea of using a standard specification: OData and JSON:API include lot's of best practices for RESTful APIs, while GraphQL offers a totally different approach, which is still battle-tested an widely used. Standards also make it easier for other people (i.e. client developers) to understand your API.
On the other hand, using a standard forces you to adhere to it even if it does not fit your needs at some points. You must either support the standard completely or perform additional checks to throw errors if clients attempt to use unsupported features.
One thing, that is definitely a pro on standards is the availability of ready-to-use libraries for clients and servers. Take time to look for a library to suite your needs: if you find one, it will save you lot's of time and effort and will probably lead to the most elegant solution.
In my next article, I will compare the currently most common standard web APIs from the point of view of support for the above requirements: OData, JSON:API, pure REST, GraphQL and PartiQL.
