SenseDeep Migration Manager

Introducing the SenseDeep Migration Manager

The SenseDeep Migration manager is part of the SenseDeep DynamoDB Suite and provides a controlled way to manage DynamoDB data migrations. With it, you can easily and reliably evolve your DynamoDB data design to upgrade or downgrade your table data without downtime.

Background

Now that DynamoDB single-table design practices are commonplace, the importance of being able to evolve data designs and schemas is becoming increasingly important. Without migrations, it can be difficult to upgrade code and data when your data design and schemas change. Data migrations are a way of separating data upgrades from code deploys so they can be appropriately coordinated without downtime.

Data migrations are sequenced, reversible pieces of code that mutate your data by applying a changed DynamoDB design (schema) to your data. Using migrations expressed as code and committed to your code repository, you can confidently evolve your DynamoDB design knowing that you can easily and quickly reverse a change.

Naming and Versioning

Migrations are named and versioned using Semantic Versioning to indicate and control data compatibility. Migrations thus form a sequence of version numbers through which you can step to apply or reverse migrations.

Each migration contains upgrade and downgrade functions that will apply or reverse a migration to your table. After a migration is applied, the data table is said to be at the version of that migration.

For example, here is a simple migration with up and down functions.:

import Schema from './schema-0.1.6'

const migrate = new Migrate(OneTableParams, {
    migrations: [
        {
            version: '0.1.6',
            description: 'Create a status record',
            schema: Schema,
            async up(db, migrate) {
                await db.create('Status', {active: true})
            },
            async down(db, migrate) {
                await db.remove('Status', {})
            }
        }
    ]
})

Migrations are optionally (and advisably) paired with a specific single-table schema version that describes the data model at that migration version. Schemas are created via the SenseDeep Single Table Designer or via One Table.

The SenseDeep database migration manager displays what migrations have been applied to your data and what migrations are outstanding. It shows what is the migration version for your current data and schema.

Upgrades and Downgrades

You can run outstanding migrations by clicking “up” for each migration. You can downgrade one step by clicking “down”.

The “Repeat” button will run the last migration again. The “Reset” button will run the special “latest” migration which is useful for dev environments where you want to reset the database to a good known state.

After applying a migration, the migration manager will create an item in the table to contain the current schema version. This means your table is self-describing and always contains the appropriate schema to interpret the data.

Setup

SenseDeep can orchestrate DynamoDB migrations controlled by the OneTable Migration library. You can use this library even if you are not using OneTable for your apps. Check out the OneTable Controller sample OneTable Controller in GitHub and deploy to host and manage your data migrations. Read OneTable Migrate for more information and setup.

Once you have deployed the OneTable Controller, enter the Lambda Arn for the controller in the Table Edit slide out panel under the monitoring tab.

Migration Design Patterns

There are several important design techniques that will make creating discrete migrations easier.

• Subdivide migrations into small, stand-alone mutations to lessen the risk of bugs, data loss or corruption. It is better to have a series of small, non-compatibility breaking mutations than one big, complex migration.
• Create reversible migrations wherever possible. If something goes wrong, you may need to quickly and easily revert the database.
• Create idempotent migrations wherever possible. If a migration fails to complete for any reason, it can then be resumed. This also enables the use of canary migrations where say 5% of the data is migrated and tested before migrating the entire database.
• Observe the make then break principle where migrations never leave the database in a partially upgraded or available state. This means adding new data or capabilities while maintaining prior data until after the migration is complete and new code is deployed and fully tested.
• Before deploying a database migration, the currently deployed code must fully work with the database before and after the migration is deployed. This often means fallback code to handle new and old data entities, attributes and relationships.
• Uncoupled our keys and attributes from the physical table keys and attributes. This gives you freedom to change attributes without having to re-create items due to keys changing.
• For secondary indexes, use attribute packing to project attributes from multiple entities into a single GSI data attribute. This further uncouples index design from the actual data schema.
• Use migrations to perform maintenance tasks such as finding and removing orphaned items and old, unused attributes. LINK (gist).

Summary

Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.

Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.

SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.

More?

Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.

You may also like to read:

• Evolving DynamoDB Designs with Migrations
• SenseDeep DynamoDB Data Browser
• SenseDeep Single Table Designer
• SenseDeep Provisioning Planner
• SenseDeep DynamoDB Metrics
• SenseDeep DynamoDB Studio
• DynamoDB OneTable