{"body":"

Our Ocado Smart Platform (OSP) is a unique end-to-end eCommerce, fulfilment, and logistics platform. It was built to enable some of the world’s largest retailers to benefit from the best customer experiences and unmatched economic returns allowed by technology.

Its architecture utilises hundreds of microservices to support our partners’ online grocery businesses.

With the demand for online grocery services increasing rapidly, and the data we handle increasing too, it’s important we build and use performant and scalable solutions.

When it comes to data storage - Amazon’s DynamoDB is a key tool for us and is usually suggested as an alternative to Amazon’s RDS. As a fully managed NoSQL database, it provides single-digit millisecond performance at scale, scales to any workload, and can handle any volume of data.

At Ocado Technology we are constantly challenged to think of unique and innovative solutions for challenging problems. We’ve successfully applied DynamoDB for several business-critical areas such as the microservices responsible for shopping baskets, applied promotions, placed orders and refunds.

If you come from a relational database background, like many of our engineers, it’s difficult to change your thinking about data modeling. So I’ve gathered some good practices and design patterns that helped us grasp the principles of DynamoDB and make the switch.

Throughout this blog, I use some well-established DynamoDB terms. Make sure you’re familiar with them before you read on:

Please note that the examples provided below are just for presentation purposes and don’t represent real OSP solutions.

Data access patterns first!

The relational model’s primary focus is on data, or rather relations. You model your entities into different tables and follow the normalisation rules to organise them in a memory-efficient manner.

In contradiction to that, the NoSQL approach optimises CPU cycles, not the memory footprint. You want to store the data in a denormalised way, in structures as close to business needs as possible. That’s why you need a thorough data access patterns analysis before you start data modeling.

What are data access patterns exactly?

First and foremost, it’s all the read patterns that your application must fulfil. Think of the different ways that users query data, which attributes they use to filter it, are there any attributes that always come together, and so on.

Sample read access patterns

The access patterns are not only about the reads though. The way you write data in your application may define your access patterns too. Your team needs to assess which access patterns should have a bigger impact on your data model—is your app write-heavy, read-heavy, or perhaps balanced?

Consider an example of a shopping basket. Users typically add/remove a single item to it, increase item quantity, or check the whole basket contents. You may think of modeling it for fast reads—that’d require storing the full contents of the basket every time (1). This, however, seems wasteful and may impact the performance at scale, as every change to a basket will write all the contents back to the database. If you follow the write access pattern instead, you’d store each item separately (2). Only the first one added to the basket would require two writes—basket metadata and item details. Items added later will require only a single write with a limited payload.

(1) Shopping basket items in a single attribute

(2) Shopping basket items stored separately

Why are the access patterns so important?

Whenever you approach data modeling for DynamoDB it’s important to think carefully about your data access patterns, write them down, review, and repeat. It’s just too easy to miss some use-cases or attributes, and when modeled incorrectly, it’s going to be difficult or even impossible to get the data you need from your DynamoDB table.

Remember, you optimise the data model for performance, but also sacrifice some flexibility at the same time.

A single table is (usually) enough

It’s tempting to store your business entities in multiple, dedicated DynamoDB tables. At first sight, it makes it easier to understand the data model and it’s not that far from the good old relational database management system (RDBMS) approach, which makes people more comfortable.

But it's not optimal. Any use case requiring joining data from multiple DynamoDB tables will come with many round trips to your database, and likely some additional logic in the application itself. This may impact your service performance, especially at scale.

Using a single table design and partition and index overloading techniques, on the other hand, will allow you to store the related items close together. You shouldn’t look for ad-hoc joins of data across multiple tables anymore but focus on the selected, modeled joins that matter for your application logic.

Partition overloading

It may take a while to wrap your head around the concept of storing multiple entity types in a single DynamoDB table. A nice trick that frees you from thinking in an entity-centered way is to name your partition key and sort key attribute names as generic as possible—from our experience PK and SK work just fine.

Generic PK and SK attribute names allow you to divide your table into different namespaces, or rather, data containers without embedding their type into the table schema. The PK attribute name doesn’t say if it’s an orderId, itemId, customerId, or some other entity. Overloading it, or simply speaking, adding a new data container to your table is a matter of defining a new format for its PK. In addition to that, if you use composite keys in a different form for each entity type, you can model multiple, complex data relationships in a single table.

Let’s consider an example of customer orders table (3) that contains two different data containers:

Customer metadata, where PK and SK = customer id
Customer’s orders, where PK= customer id and SK = order id

(3) single table containing customer metadata and orders

A recommended practice when using partition overloading is to create an entityType attribute with human-readable values, like order, item or customer. This would make it easier to identify your entities, both as a human reader and in implementation—it will save you some time decoding the format of your PK or SK. One more thing—don’t forget to document your overloaded partitions and key formats.

Index overloading

Secondary indexes in DynamoDB offer you a way to add different perspectives to your data. They are usually a crucial ingredient required to provide all the data access patterns in your application.

A Local Secondary Index has the same partition key, but a different sort key, allowing you to define yet another way of sorting your main data. Simply speaking, it’s an alternate sort key.

A Global Secondary Index may have a different partition key and sort key than the ones defined in the main table. Practically speaking, they provide a new, virtual table with the same data rows, but a different main key.

The simplest example is an entity for which your application uses two sets of identifiers:

one for internal use by other services in your architecture (4),
and the other for external use via public API (5).

(4) Internal id mapping

(5) External id mapping

There’s a catch though—a single DynamoDB table may define only 20 global secondary indexes (GSIs). This sounds quite constraining at first, but you can use the overloading approach when defining secondary indexes. Similar to the partition key and sort key of your main table, use generic names for attributes defined solely for GSIs. Now a pair of attributes named GSI1_PK and GSI1_SK may cover multiple access patterns.

Summary

Storing data at scale has its own unique challenges. For a growing organisation, like ours – performant and scalable solutions are vital. One of the many benefits of working at Ocado Technology is having the flexibility to work with the right tools for the job, helping us solve new challenges across the business. DynamoDB has been useful and to make the best out of it, you need to understand your application access patterns and utilise partition and index overloading to model your “joins”.

In the end, your optimised table will likely look closer to the computer language than human-readable RDBMS tables, and you might lose some of the flexibility of ad-hoc querying data. That’s, however, a sacrifice to make to get the great characteristics of DynamoDB—speed, reliability, consistency, and ability to scale to any workload.

","presentation":{"classes":"","theme":"theme-default","columns":""}}

Storing data at scale using Amazon DynamoDB