Building Instagram with DynamoDB Single Table Designs

Alex DeBrie, author of the DynamoDB Book, created a sample Instagram backend for DynamoDB, as part of a video he did with Marcia Villalba for the FooBar Channel.

In that video, Alex covers the key basics of DynamoDB including data modeling, one-to-many relationships, single-table design and the core DynamoDB API. The sample he used is an excellent introduction to DynamoDB and demonstrates how to organize an app when using DynamoDB.

As a fun exercise, I wanted to see how this sample app would be changed, simplified and potentially made easier to maintain by using the OneTable DynamoDB library.

OneTable is a DynamoDB access library for NodeJS that makes dealing with DynamoDB faster, easier and less error prone. It make DynamoDB single table design patterns simpler and provides a terse, flexible API.

This post will do a side-by-side comparison of OneTable and original native DynamoDB implementations of this Instagram sample.

Disclaimer: The original code is clean, consistent and a well executed sample that is an excellent guide for creating a DynamoDB application. I personally learned several new things while doing this comparison — thank you Alex. If the OneTable design is simpler or cleaner, that is the result of the additional services offered by OneTable and not by any better coding on my part.

The code for the OneTable variation is available at GitHub senseDeep/dynamodb-instagram-blog and you can find the original app at GitHub alexdebrie/dynamodb-instagram.

See the sample README which describes how to checkout the sample from GitHub and how to run locally using the Serverless framework and VS Code.

Getting Started

Before comparing the two implementations, it is worth providing a quick design overview so you understand the sample design and concepts.

Code Overview

This sample is an Instagram backend clone where users can post photos and other users may like a photo or comment on a photo. A user may also choose to follow another user in order to see their recent activity. There is no UI. All interaction is via a REST API.

The sample follows a Single-Table design pattern where all the database entities are stored together in a single DynamoDB table. You can read more in my post Data Modeling for Single Table Designs.

Database Entities

Here is the ERD for the sample database schema with 5 entities: User, Follow, Like, Photo and Comment.

A User represents a person that has signed up for the application. They will be uniquely identified by a user name in the user attribute. A count of followers and people followed by the user is kept in followers and following.

A Photo represents an image uploaded by a particular User. You can browse all Photos for a particular User in reverse-chronological order. Each Photo can be Liked or Commented on (see below). A photo keeps a count of the number of comments and likes. The photo image is referenced by a url. Each photo is uniquely indexed by a photoId and a user name.

A Like represents a specific User liking a specific Photo. A specific Photo may only be liked once by a specific User. When showing a Photo, we will show the total number of Likes for that Photo.

A Comment represents a User commenting on a particular Photo. There is no limit to the number of Comments on a Photo by a given User. When showing a Photo, we will show the total number of Comments for that Photo.

A Follow represents one User choosing to follow another User. By following another User, you will receive updates from that User in your timeline (not implemented in this sample). A Follow is a one-way relationship — a User can follow another User without the second User following in return. For a particular User, we show the number of other Users following them and the number of Users they’re following, as well as the ability to show the lists of Followers and “Followees”.

OneTable Schema

OneTable expresses the ERD as a schema that fully describes each entity and their relationships. The OneTable schema specifies how the entities are stored in the DynamoDB table via their DynamoDB keys. It also describes the indexes, entity keys, attributes and attribute data values and validations.

Our sample schema is comprised of a set of entity models for the Comment, Follow, Like, Photo and User entities. Each model defines the partition and sort keys for the primary index and any secondary indexes for the entity and specifies how those keys are computed via value templates using other attributes values.

The value templates uncouple the entity’s keys from data attributes and provides flexibility should you need to redesign your table in the future.

//  RegularExpressions for validations

const Match = {
  name:    /^[^<>~`!@#$%^&\*(){}\[\]|\\\/:;,?=+]+$/,
  username:  /^[^<>~`!@#$%^&\*(){}\[\]|\\\/:;,?=+]+$/,
  ulid:    /^[0123456789ABCDEFGHJKMNPQRSTVWXYZ]{26}$/,
  url:     /^(https?|ftp):\/\/[^\s/$.?#].[^\s]*$/i,
}

export const Schema = {
  version: '0.1.0',
  format: 'onetable:1.0.0',
  indexes: {
    primary: { hash: 'pk', sort: 'sk' },
    gs1: { hash: 'gs1pk', sort: 'gs1sk', project: 'all' }
  },
  models: {
    Comment: {
      pk:                 { type: String, value: 'Comment#${photoId}' },
      sk:                 { type: String, value: 'Comment#${id}' },
      user:               { type: String, required: true },
      id:                 { type: String, required: true, uuid: 'ulid' },
      text:               { type: String, required: true },
      photoId:            { type: String, required: true, validate: Match.ulid },
    },

    Follow: {
      pk:                { type: String, value: 'follow#${followed}' },
      sk:                { type: String, value: 'follow#${follower}' },
      gs1pk:             { type: String, value: 'follow#${follower}' },
      gs1sk:             { type: String, value: 'follow#${followed}' },
      followed:          { type: String, required: true },
      follower:          { type: String, required: true },
    },

    Like: {
      pk:             { type: String, value: 'Like#${photoId}' },
      sk:             { type: String, value: 'Like#${user}' },
      user:           { type: String, required: true },
      photoId:        { type: String, required: true, validate: Match.ulid },
      gs1pk:          { type: String, value: 'Like#${user}' },
      gs1sk:          { type: String, value: 'Like#${photoId}' },
    },

    Photo: {
      pk:           { type: String, value: 'Photo#${user}' },
      sk:           { type: String, value: 'Photo#${photoId}' },
      comments:     { type: Number, required: true },
      likes:        { type: Number, required: true },
      photoId:      { type: String, required: true, uuid: 'ulid', validate: Match.ulid },
      url:          { type: String, required: true, validate: Match.url },
      user:         { type: String, required: true, validate: Match.username },
    },

    User: {
      pk:           { type: String, value: 'User#${user}' },
      sk:           { type: String, value: 'User#' },
      followers:    { type: Number, required: true, default: 0 },
      following:    { type: Number, required: true, default: 0},
      name:         { type: String, required: true, validate: Match.name },
      username:     { type: String, required: true, validate: Match.username },
    }
  },
  params: {
    "isoDates": true,
    "timestamps": false,
    "typeField": "_type"
  }
}

OneTable Schema Attributes

OneTable model attributes specify their data type via the type property. Using this type, OneTable API operations will fully validate the values written to the database.

OneTable also uses the schema types to dynamically, and auto-magically create TypeScript types for each entity. Using these, TypeScript will generate compile time errors if you misspell a property name.

The attribute’s validate property provides a regular expression to check attribute values. OneTable will only write attributes specified in the schema to the table. This helps greatly to ensure database integrity.

OneTable attributes are controlled and managed by several other properties. The required property designates an attribute that must have a value when the item is created. The default property will assign a default value if one is not provided during create. Alternatively, an attribute may define a value template property that describes how to determine a property value based on other attribute values at run-time. Lastly, attributes may also be designated as unique where the attribute value must be unique in the table.

OneTable has an internal ULID implementation to create sortable unique identifiers. If an attribute is flagged with uuid: 'ulid', an entity item will have a ULID automatically generated when it is created.

Files and Directories

The sample is structured with the source code under three key directories.

The connect directory contains the DynamoDB connection setup and the OneTable DynamoDB table schema.

The data directory contains the data modeling layer which provides APIs for each database entity. The index exports all the entities. There are data source files for each of the entities: Comment, Follow, Like, Photo and User.

The handlers directory contains the REST request handlers. Each handler is a Lambda REST endpoint handler.

The connect directory contains a client.ts which sets up the DynamoDB connection. The connect/index.ts initializes OneTable and exports the OneTable singleton and schema.

Original Like Photo Entity

Let’s first consider the original Like entity class: Like.ts. This file exports a Like class and discrete functions to like photos and list the likes for a photo.

The class communicates directly with DynamoDB and provides a controlled interface for liking photos. It uses transactions when creating a Like and updating the Photo likes counter.

A ConditionExpression is used to ensure this User hasn’t already liked the given Photo.

import { DynamoDB } from "aws-sdk"
import { ulid } from "ulid"

import { Item } from "./base"
import { getClient } from "./client"
import { Photo } from "./photo"
import { executeTransactWrite } from "./utils"

export class Like extends Item {
  user: string
  photoId: string

  constructor(user: string, photoId: string) {
    super()
    this.user = user
    this.photoId = photoId
  }

  static fromItem(item?: DynamoDB.AttributeMap): Like {
    if (!item) throw new Error("No item!")
    return new Like(item.user.S, item.photoId.S)
  }

  //  Encode the partition key
  get pk(): string {
    return `Like#${this.photoId}`
  }

  //  Encode the sort key
  get sk(): string {
    return `Like#${this.user}`
  }

  get gsi1pk(): string {
    return this.pk
  }

  get gsi1sk(): string {
    return `Like#${this.user}`
  }

  toItem(): Record<string, unknown> {
    return {
      ...this.keys(),
      gsi1pk: { S: this.gsi1pk },
      gsi1sk: { S: this.gsi1sk },
      user: { S: this.user },
      photoId: { S: this.photoId },
    }
  }
}

export const likePhoto = async (photo: Photo, user: string): Promise<Like> => {
  const client = getClient()
  const like = new Like(user, photo.photoId)

  try {
    await executeTransactWrite({
      client,
      params: {
        TransactItems: [{
            Put: {
              TableName: process.env.TABLE_NAME,
              Item: like.toItem(),
              ConditionExpression: "attribute_not_exists(PK)"
            }
          }, {
            Update: {
              TableName: process.env.TABLE_NAME,
              Key: photo.keys(),
              ConditionExpression: "attribute_exists(pk)",
              UpdateExpression: "SET #likes = #likes + :inc",
              ExpressionAttributeNames: { "#likes": "likes" },
              ExpressionAttributeValues: { ":inc": { N: "1" } }
            }
          }
        ]
      }
    })
    return like
  } catch (error) {
    console.log(error)
    throw error
  }
}

export const listLikesForPhoto = async (photoId: string): Promise<Like[]> => {
  const client = getClient()
  const like = new Like("", photoId)

  try {
    const resp = await client.query({
        TableName: process.env.TABLE_NAME,
        IndexName: "gsi1",
        KeyConditionExpression: "gsi1pk = :gsi1pk",
        ExpressionAttributeValues: { ":gsi1pk": { S: like.gsi1pk } },
        ScanIndexForward: false
      }).promise()
    return resp.Items.map((item) => Like.fromItem(item))

  } catch (error) {
    console.log(error)
    throw error
  }
}

There are several important facets to this original version of the Like class:

• The class directly encodes the attributes for the User’s primary and secondary partition key and sort key values.
• It directly encodes the entity attributes. This means the table schema is expressed in code and is distributed among the entity classes.
• The code handles translating from JavasScript to DynamoDB data types and vice-versa. This could be simplified by using the DynamoDB DocumentClient wrapper.
• The DynamoDB boilerplate code is quite verbose (see executeTransactWrite) and the code required to create Condition Expressions and Update Expressions is quite finicky to codify. This code for Like is very similar to other entities and the repetition of such code can be error prone.

OneTable Like Model Class

The OneTable based implementation leverages the CRUD capability provided by OneTable. This results in smaller and simpler code.

Model Type

The OneTable version creates a Model type directly from the schema:

import { Entity, Models } from '../connect'
export type Like = Entity<typeof Models.Like>

This creates a type Like with properties for the attributes: user, and photoId. The DynamoDB primary and secondary key properties are computed internally by OneTable via the value template and are thus not needed or present in the Like type.

Like Class

The Like entity class wraps all the logic and methods to interact with the class. It extends the OneTable Model generic to create a class with methods for: create, find, init, get, remove and update. These methods will enforce the Like data type on parameters and results.

export class LikeClass extends Model<Like> { ... }

Database Connection

The database connection is initialized in this sample via the connect/client.ts file. It is wrapped by OneTable and initialized via the connect/init.ts file. The OneTable singleton is passed to the Like class constructor to complete the initialization.

import { Entity, Model, Models, OneTable } from '../connect'
export class LikeClass extends Model<Like> {
    constructor() {
        super(OneTable, 'Like')
    }
}

Transactions

OneTable has a terse API that makes dealing with transactions almost as simple as generic API calls. A transaction object is created and passed to all API calls that contribute to the transaction. The trasact API then enacts the transaction.

async likePhoto(photoId: string, user: string): Promise<Like> {
    const transaction = {}
    const like = await LikeModel.create({user, photoId}, {transaction})
    await PhotoModel.update({username: user}, {
        add: {likes: 1},
        transaction
    })
    await OneTable.transact('write', transaction)
    return like
}

Note also they way the PhotoModel increments the likes via the add operator.

OneTable always employs a ConditionExpression when creating items to ensure it does not already exist.

OneTable Like Class

Here is the complete implementation.

import { Entity, Model, Models, OneTable } from '../connect'
import { PhotoModel } from "./photo"

export type Like = Entity<typeof Models.Like>

export class LikeClass extends Model<Like> {
    constructor() {
        super(OneTable, 'Like')
    }

    async likePhoto(photoId: string, user: string): Promise<Like> {
        const transaction = {}
        const like = await LikeModel.create({user, photoId}, {transaction})
        await PhotoModel.update({username: user}, {
            add: {likes: 1},
            transaction
        })
        await OneTable.transact('write', transaction)
        return like
    }

    async listLikesForPhoto(photoId: string): Promise<Like[]> {
        return LikeModel.find({photoId}, {index: 'gs1', reverse: true})
    }
}
export const LikeModel = new LikeClass()

Some of the key facets of the OneTable implementation are:

• The Like type is dynamically created from the OneTable schema using Entity<typeof Models.Like>.
• The LikeClass extends the OneTable Model generic class that provides full CRUD operations including: create, get, find, init, remove and update operations.
• The OneTable methods will validate data types and values according to the schema.
• OneTable will internally assign default attribute values and computed DynamoDB keys via value templates.
• Public entity methods (likePhoto) are provided as methods on the LikeClass.
• The class exports a singleton that consumers utilize.

Handlers

The REST handlers are only modified slightly in how they create and interact with the data layer entities.

The original likePhoto handler implementation imports likePhoto and Photo references and uses these to mutate the database.

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda"
import { likePhoto } from "../data/like"
import { Photo } from "../data/photo"

export const main: APIGatewayProxyHandler = async (event: APIGatewayProxyEvent) => {
  const { username, photoId } = event.pathParameters
  const photo = new Photo(username, "", photoId)
  const { user } = JSON.parse(event.body)
  const like = await likePhoto(photo, user )
  const response = {
    statusCode: 200,
    body: JSON.stringify({like})
  }
  return response
}

OneTable LikePhoto REST Handler

In the OneTable implementation, likePhoto imports a LikeModel and uses it to create the Like for a photo via the likePhoto method.

import { LikeModel } from "../data"
await LikeModel.likePhoto(photoId, user)

The full OneTable implementation is:

import { APIGatewayProxyEvent, APIGatewayProxyHandler } from "aws-lambda"
import { LikeModel } from "../data"

export const main: APIGatewayProxyHandler = async (event: APIGatewayProxyEvent) => {
  const { photoId } = event.pathParameters
  const { user } = JSON.parse(event.body)
  const response = {
    statusCode: 200,
    body: JSON.stringify({
      like: await LikeModel.likePhoto(photoId, user)
    })
  }
  return response
}

Summary

This comparison highlights the use of DynamoDB design patterns for a simple Instagram like clone. The sample uses many key design patterns of DynamoDB single-table design and demonstrates the effectiveness of OneTable in expressing such patterns.

Thank you to Alex for creating and sharing this excellent sample app.

To learn more about OneTable, read DynamoDB OneTable.

The code for the OneTable variation is available at GitHub senseDeep/dynamodb-instagram. You can find the original app at GitHub alexdebrie/dynamodb-instagram.

Links

• SenseDeep DynamoDB Studio
• OneTable Post