Data Modeling and Partitioning Best Practice in Partitioned Repository Pattern With Cosmos DB¶

Even though Cosmos DB is schema-free and supports storing and querying unstructured data, it is still critical to spend good efforts in the planning phase on data modeling and partitioning. Particularly, poorly designed data models and partition keys would hurt the performance and also restrict the scalability.

In this article, we will discuss:

• how partitioning and logical partitions work in Cosmos DB
• how data is stored in different logical partitions in Cosmos DB
• how data is read from Cosmos DB, including Point Reads, Single-partition Query and Cross-partition Query
• how to design partition keys for better performance
• how partitioning can be implemented in Partitioned Repository Pattern using .NET SDK

How is data stored in different partitions in a Cosmos DB container?¶

If you come from a SQL background, here is a quick table to compare the terminology between a SQL DB and Cosmos DB.

For every container, a partition key must be pre-defined. The partition key is used to divide a container into many subsets, AKA logical partitions, and each logical partition is used to store items with the same partition key values. An example will make this much more straightforward.

For example, if a Todo list container has a partition key of “/Category”, the following todo items for grocery category will be stored in one logical partition:

{
    "Category": "Grocery",
    "Title": "Get more milk",
    "IsCompleted": false,
    "id": "Grocery:41500e6e-4e08-495b-9d09-824f03961f4a"
}
{
    "Category": "Grocery",
    "Title": "Get more eggs",
    "IsCompleted": false,
    "id": "Grocery:9797922f-0a28-439f-bdb8-f2843fc60a8f"
}

While todo items for household category will be stored in a different logical partition.

{
    "Category": "Household",
    "Title": "Paint garage",
    "IsCompleted": false,
    "id": "Household:b5d736bb-b856-4a05-878a-a2d6c8054cf3"
}
{
    "Category": "Household",
    "Title": "Mow the lawn",
    "IsCompleted": false,
    "id": "Household:82c0d6e5-5615-4443-afd5-4a81b0b9c8f0"
}

Even though all the todo items are stored in the same container, but they may be stored in different logical partitions. Because different logical partitions may be physically stored on different machines, a container technically can support unlimited amount of logical partitions, allowing Cosmos DB to scale infinitely! According to Microsoft documentation on logical partition,

• A container can have unlimited number of logical partitions
• Each logical partition has a storage limit of 20G

In our example case here, 20G is way more than enough to store just grocery items that I need! In real-world application, userId can probably be used in addition to just category, so that each user may get a logical partition for each of their todo categories.

How is data retrieved from different partitions in a Cosmos DB container?¶

Point Reads¶

If you come from a SQL background, a table most likely will have a primary key, such as UserId in User table, and the primary key is used to determine how the rows are stored and sorted physically. When you read a record in SQL by its primary key value, say UserId, the read request is considered a primary key seek and is probably the quickest read request. For example, your request may look like this if you use raw SQL query:

SELECT * FROM User WHERE UserId = 1

OR, most likely you will use some sort of ORMs like Entity Framework, then your will see code like this:

User user = await _dbContext.User.Where(x => x.UserId == 1).FirstOrDefaultAsync();

Similarly, when you read one item in Cosmos DB using the item’s partition key value and ID value, it is considered a point-read and is also the fastest and cheapest read request in Cosmos DB, since all the request need is to look up one specific item in one specific logical partition in the container. For example, Microsoft documentation example code for reading one item using .NET SDK:

ToDoActivity toDoActivity = await this.container.ReadItemAsync<ToDoActivity>(“id”, new PartitionKey(“partitionKey”));

Queries¶

Compared to Point Reads, queries are used to search for 1+ items in a container instead of seek one item by its partition key value and id value. There are multiple levels of queries, each has different performance and cost.

• In-partition query, or single-partition query
• Cross-partition query

Single-partition queries search data from a container with a specific partition key value, which means the database engine only needs to look into one logical partition. Cross-partition queries don’t have a specific partition key value, so the database engine would have to fan-out the request to all of the logical partitions in order to collect all data that meet the search criteria. A cross-partition query definitely is going to cost more resources and be slower than a single-partition query.

Point Reads > Single-partition queries > Cross-partition queries

How to incorporate above best practices into Partitioned Repository Pattern?¶

I will use a simple database for Todo items to demonstrate how the above practices can be implemented using Partitioned Repository Pattern.

Todo Container

Here is a screenshot of our Todo container, whose partition key path is “/Category”. This means all todo items having the same category value will be stored in the same logical partition. E.g. Grocery todo items. This allows me to ask for “all the grocery todo items” using a Single-partition query.

Why “/Category” instead of just “/id”?¶

Using Id as the partition key gives me a much higher cardinality, which means the Todo container will be broken into a lot more logical partitions. In fact, using Id as the partition key means every todo item has its own partition. The tradeoff is that when I need to “get all grocery todo items”, the read request will become a Cross-partition query.

Audit Container¶

I also have a container called Audit, which is used to automatically audit all the changes to my todo items. For example, When I change my item from “Get more 3% milk” to “Get more milk”, one record automatically gets stored in the Audit container. Here is a screenshot of the audit record.

Notice each audit item has

• id: unique id for the item itself
• DateCreatedUTC: a timestamp to track the time of change
• EntityId: a reference to the todo item that was changed
• Entity: a copy of the todo item before the change, in order to support rollback to any version.
• EntityType: the type of entity. Even though we only have one entity type, ToDoItem, in this case, I have designed the Audit container to be able to store any types of entity down the road and still use the same Audit container for auditing purpose. Why bother having “TodoItemAudit” and “CustomerAudit” right?

I’d like you to pause here for 5 seconds and think what property you would choose to use as the Partition Key for the Audit container?¶

I chose EntityId, so that every single Todo item will have it’s own logical partition to store it’s change history. Why? Almost all of my read queries for the auditing records will be like: get me the history of changes for todo item “Get more milk”, and I want it to be a Single-partition query.

Code — Partitioned Repository implementation¶

Interfaces: IRepository.cs, IToDoItemRepository.cs, IAuditRepository.cs. These define the data access contracts at the abstraction level. The actual implementation is more relevant to this article, but I will list it here for quick reference.

• Generics T is used in IRepository, so that any type specific repositories can just inherit this interface without duplicating the method definitions.
• Specification Pattern is used for GetItemsAsync(). Specification allows me to build my Expression> using LINQ in a separate file instead of here in the IRepository, in order to achieve Separate of Concern principal. If you wonder what Expression> is, it defines what the function does, but does not execute it, otherwise the entire database container/table will be scanned. For more information on Expression> vs a delegate Func, here is a good stack overflow answer.
• Specifications are resolved using a NuGet package by Ardalis.

namespace CleanArchitectureCosmosDB.Core.Interfaces
{
    public interface IRepository<T> where T : BaseEntity
    {
        /// <summary>
        ///     Get one item by Id
        /// </summary>
        /// <param name="id"></param>
        /// <returns></returns>
        Task<T> GetItemAsync(string id);

        /// <summary>
        ///     Get items given a specification.
        /// </summary>
        /// <param name="specification"></param>
        /// <returns></returns>
        Task<IEnumerable<T>> GetItemsAsync(ISpecification<T> specification);

        // other code skipped
    }
}

Interfaces like IToDoItemRepository and IAuditRepository only need to inherit IRepository.cs without defining the methods again. Less duplicated code and really DRY!

using CleanArchitectureCosmosDB.Core.Entities;

namespace CleanArchitectureCosmosDB.Core.Interfaces
{
    public interface IToDoItemRepository : IRepository<ToDoItem>
    {
    }
}

using CleanArchitectureCosmosDB.Core.Entities;

namespace CleanArchitectureCosmosDB.Core.Interfaces.Persistence
{
    public interface IAuditRepository : IRepository<Audit>
    {
    }
}

ToDoItemRepository.cs

This is the concrete implementation of the repository for the Todo container. Take-away notes are:

• The specific container name, “Todo”, is a property.
• To create a new todo item, GenerateId() method is used to generate its new id. It returns a concatenated string of the item’s category plus a new GUID. For example, “Grocery:41500e6e-4e08–495b-9d09–824f03961f4a”. Having both the partition key value and the unique identifier in the id allows us to define GetItemAsync(string id) to only require one parameter in the signature instead of two parameters, like GetItemAsync(string id, PartitionKey partitionKey). This allows the developers who only need to use the repository, such as the API developers , to work with the repository just like they are working with a SQL database. Less code, less error! In fact, this almost makes the API database-agnostic!
• To point-read an item by id, ResolvePartitionKey() method is used to retrieve the partition key value from the id. For example, “Grocery” as the partition key value from “Grocery:41500e6e-4e08–495b-9d09–824f03961f4a”. With the partition key value and id, we will be able to perform a point-read.

namespace CleanArchitectureCosmosDB.Infrastructure.CosmosDbData.Repository
{
    public class ToDoItemRepository : CosmosDbRepository<ToDoItem>, IToDoItemRepository
    {
        /// <summary>
        ///     CosmosDB container name
        /// </summary>
        public override string ContainerName { get; } = "Todo";

        /// <summary>
        ///     Generate Id.
        ///     e.g. "shoppinglist:783dfe25-7ece-4f0b-885e-c0ea72135942"
        /// </summary>
        /// <param name="entity"></param>
        /// <returns></returns>
        public override string GenerateId(ToDoItem entity) => $"{entity.Category}:{Guid.NewGuid()}";

        /// <summary>
        ///     Returns the value of the partition key
        /// </summary>
        /// <param name="entityId"></param>
        /// <returns></returns>
        public override PartitionKey ResolvePartitionKey(string entityId) => new PartitionKey(entityId.Split(':')[0]);

        public ToDoItemRepository(ICosmosDbContainerFactory factory) : base(factory)
        { }

        // other code skipped
    }
}

CosmosDbRepository.cs this is an abstract class that defines virtual/default implementations for entity-specific repositories like ToDoItemRepository. It also defines the container level information. What we are interested for this article is:

• GetItemAsync(string id), which uses the .NET SDK example code provided by Microsoft documentation mentioned earlier, but hides the complex requirement of both a partition key and id from the developers that only need to use the repository, such as the API developers.
• Audit(T item) and UpdateItemAsync(string id, T item), which demonstrate how automatic auditing can be done behind the scene, without the repository users like API developers to even know about it existing.

namespace CleanArchitectureCosmosDB.Infrastructure.CosmosDbData.Repository
{
    public abstract class CosmosDbRepository<T> : IRepository<T>, IContainerContext<T> where T : BaseEntity
    {
        // some code skipped

        /// <summary>
        ///     Name of the CosmosDB container
        /// </summary>
        public abstract string ContainerName { get; }

        /// <summary>
        ///     Generate id
        /// </summary>
        /// <param name="entity"></param>
        /// <returns></returns>
        public abstract string GenerateId(T entity);

        /// <summary>
        ///     Resolve the partition key
        /// </summary>
        /// <param name="entityId"></param>
        /// <returns></returns>
        public abstract PartitionKey ResolvePartitionKey(string entityId);

        /// <summary>
        ///     Generate id for the audit record.
        ///     All entities will share the same audit container,
        ///     so we can define this method here with virtual default implementation.
        ///     Audit records for different entities will use different partition key values,
        ///     so we are not limited to the 20G per logical partition storage limit.
        /// </summary>
        /// <param name="entity"></param>
        /// <returns></returns>
        public virtual string GenerateAuditId(Audit entity) => $"{entity.EntityId}:{Guid.NewGuid()}";

        /// <summary>
        ///     Resolve the partition key for the audit record.
        ///     All entities will share the same audit container,
        ///     so we can define this method here with virtual default implementation.
        ///     Audit records for different entities will use different partition key values,
        ///     so we are not limited to the 20G per logical partition storage limit.
        /// </summary>
        /// <param name="entityId"></param>
        /// <returns></returns>
        public virtual PartitionKey ResolveAuditPartitionKey(string entityId) => new PartitionKey($"{entityId.Split(':')[0]}:{entityId.Split(':')[1]}");


        private readonly ICosmosDbContainerFactory _cosmosDbContainerFactory;
        private readonly Microsoft.Azure.Cosmos.Container _container;
        /// <summary>
        ///     Audit container that will store audit log for all entities.
        /// </summary>
        private readonly Microsoft.Azure.Cosmos.Container _auditContainer;

        public CosmosDbRepository(ICosmosDbContainerFactory cosmosDbContainerFactory)
        {
            this._cosmosDbContainerFactory = cosmosDbContainerFactory ?? throw new ArgumentNullException(nameof(ICosmosDbContainerFactory));
            this._container = this._cosmosDbContainerFactory.GetContainer(ContainerName)._container;
            this._auditContainer = this._cosmosDbContainerFactory.GetContainer("Audit")._container;
        }

        public async Task<T> GetItemAsync(string id)
        {
            try
            {
                ItemResponse<T> response = await _container.ReadItemAsync<T>(id, ResolvePartitionKey(id));
                return response.Resource;
            }
            catch (CosmosException ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
            {
                return null;
            }
        }

        /// <inheritdoc cref="IRepository{T}.GetItemsAsync(Ardalis.Specification.ISpecification{T})"/>
        public async Task<IEnumerable<T>> GetItemsAsync(ISpecification<T> specification)
        {
            var queryable = ApplySpecification(specification);
            var iterator = queryable.ToFeedIterator<T>();

            List<T> results = new List<T>();
            while (iterator.HasMoreResults)
            {
                var response = await iterator.ReadNextAsync();

                results.AddRange(response.ToList());
            }

            return results;
        }

        public async Task UpdateItemAsync(string id, T item)
        {
            // Audit
            await Audit(item);
            // Update
            await this._container.UpsertItemAsync<T>(item, ResolvePartitionKey(id));
        }

        /// <summary>
        ///     Evaluate specification and return IQueryable
        /// </summary>
        /// <param name="specification"></param>
        /// <returns></returns>
        private IQueryable<T> ApplySpecification(ISpecification<T> specification)
        {
            var evaluator = new CosmosDbSpecificationEvaluator<T>();
            return evaluator.GetQuery(_container.GetItemLinqQueryable<T>(), specification);
        }

        /// <summary>
        ///     Audit a item by adding it to the audit container
        /// </summary>
        /// <param name="item"></param>
        /// <returns></returns>
        private async Task Audit(T item)
        {
            var auditItem = new Core.Entities.Audit(item.GetType().Name,
                                                    item.Id,
                                                    Newtonsoft.Json.JsonConvert.SerializeObject(item));
            auditItem.Id = GenerateAuditId(auditItem);
            await _auditContainer.CreateItemAsync<Audit>(auditItem, ResolveAuditPartitionKey(auditItem.Id));
        }

    }
}

Conclusion¶

At this point, we have covered multiple topics:

• how data is stored in different logical partitions in Cosmos DB
• how data is read from Cosmos DB, including Point Reads, Single-partition Query and Cross-partition Query
• how to design partition keys for better performance
• how partitioning can be implemented in Partitioned Repository Pattern using .NET SDK.

Hopefully this quick article is helpful to your next data modeling and partition key design task. Many thanks for reading!

Sample code in this article is from a GitHub starter project designed to start a ASP.NET Core API and React web application using Cosmos DB with Partitioned Repository Pattern. Additional features like Azure Functions are supported. If you would like to learn more about the start project used in this article, please check out GitHub repo below. You can also check out my articles relevant to the project.

• GitHub repo: Clean Architecture with partitioned repository pattern using Azure Cosmos DB
• Article: Clean Architecture — ASP.NET Core API using Partitioned Repository Pattern and Azure Cosmos DB
• Article: Audit Log Using Partitioned Repository Pattern With Cosmos DB
• Article: Clean Architecture — Azure Functions Using Partitioned Repository with Cosmos DB