A new log engine is coming soon to SenseDeep (Nov 3rd). This post is a preview of what is to come and how the new release will be deployed.
The new log engine offers enhanced speed and reduced cost of ingesting log data on your AWS bill. The current SenseDeep release uses the DynamoDB database to store recent log data and while this can be ideal for Serverless, DynamoDB imposes a non-trivial cost for writing log data and for larger sites, this cost can be significant. These costs impact your AWS bill.
The new engine uses the AWS Elastic File System (EFS) to cache recent log data. EFS is a low-cost, highly scalable, persistent filesystem. SenseDeep has created a log database that uses EFS to cache recent log data and this has greatly reduced the cost of ingesting log data.
With the new release, you get the same easy access to your most recent log data, but at a fraction of the price on your AWS bill.
Scheduled Maintenance Window to Upgrade SenseDeep: November 3rd 11:00 PM and 3:00 AM PST.
The current release of SenseDeep uses DynamoDB to cache recent log data and preserves the master copy of log data in CloudWatch Logs. DynamoDB is an AWS serverless database that offers almost limitless scalability and can ingest vast amounts of data.
DynamoDB has several features that make it ideal for storing log data. It can scale to accommodate heavy simultaneous updating when a burst of log data needs to be ingested and stored. Time-series log data can also be effectively mapped onto the DynamoDB indexing schema.
DynamoDB is highly reliable and stores 6 replicas of your data, however, this resiliency comes with a cost. For log data, extra copies are not required as a master copy of the log data is always available from CloudWatch logs. The extra replicas do however, impose a significant cost to your AWS bill.
The AWS Elastic File System (EFS) is a simple, fast, serverless, fully managed, elastic file system. It offers up to almost 8 Exabytes of storage with a variety of redundancy configurations.
As a storage medium for log data, it offers extremely low cost with fast storage and retrieval times.
SenseDeep has created a high-performance log database that leverages EFS and offers the following features:
The change to SenseDeep for the new logging engine is entirely “under the hood”. The SenseDeep App user interface is not changing with the exception of removing log data caps from the Cloud > Edit panel.
The rollout for the new release is scheduled for Nov 3rd between 11:00 PM and 3:00 AM PST.
Following the deployment of the release, it will be necessary for you to recreate your cloud connections in SenseDeep to access log data. Please be aware that your recent log data cache will be cleared, but there is no need to worry, as your historical CloudWatch logs data will remain completely unaffected.
To update your cloud connections, follow these steps:
If you have any difficulties, please contact us at support@sensedeep.com.
After updating your cloud connection, SenseDeep will start caching your log data in the new log database. If you require past log data, you can select backfill from the Log Edit page to backfill the SenseDeep log cache from CloudWatch logs.
In addition to the new logging engine, the new release splits the SenseDeepWatcher lambda into a separate SenseDeepLogger. Previously the Watcher was responsible for ingesting log data and for synchronizing SenseDeep’s map of your AWS resources.
The SenseDeepLogger is now solely responsible for ingesting log data and for running log alarms to check for any alert conditions in your log data. An AWS EFS must run in a VPC, so the Logger lambda runs in its own private, isolated VPC disconnected from the public internet.
As a result of switching to EFS, SenseDeep no longer needs to impose logging data caps. Due to the capacity and scale of EFS, log data caps are not necessary and will be removed in this release.
If you have any questions prior to the rollout. Please contact us at support@sensedeep.com.
]]>If you have not properly prepared, debugging some requests may be impossible with your current level of instrumentation and tooling.
So what is the best way to debug serverless apps and serverless requests?
The blog post describes how we debug the serverless backend for our SenseDeep Serverless Developer Studio service and what tools we use with our NodeJS serverless apps.
Some of the suggestions here may not be suitable for all sites, but I hope they give you ideas to improve your ability to debug your serverless apps.
But first, some background.
The way enterprises design, debug and ship applications changed forever when serverless computing arrived on the scene. The introduction of serverless allowed developers to build and ship much faster, which in turn allows them to concentrate on coding rather than maintenance, auto-scaling, and server provisioning.
Serverless is now mainstream and it offers many compelling advantages, but debugging serverless apps is still a tough problem as the required tools have not kept pace.
Debugging a server-based app (monolith) is well understood and assisted by a suite of refined tools that have been created over decades of evolution. Debugging can be performed using IDEs either locally or remotely over SSH and other protocols. Importantly, server state is typically persisted after requests to permit live or after-the-fact inspection and debugging.
Serverless is not so lucky
Serverless is different. It is often more complex with multiple interacting, loosely coupled components. Instead of a monolith, it is more like a spider’s web and consequently harder to follow a single request with no ability for live debugging or setting breakpoints to intercept code execution.
Debugging serverless via remote shell or execution is not possible. Serverless requests are ephemeral and request state is not available after execution. Once a request has been completed, there is no state left behind to examine.
So when a serverless request errors, it often fails silently or any record is buried in a mountain of log data in CloudWatch. You will not be proactively notified and if you are informed by the end-user, finding the failed request is a “needle in a haystack” type of problem.
Furthermore, comprehensive local emulation of cloud environments is either not possible or has severe limitations. As cloud services become more evolved, local emulation is becoming increasingly limited in scope and application.
So we need a different technique to debug serverless compared to server-based app debugging.
The primary means of debugging serverless apps is via detailed, intelligent request and state logging, paired with log management to correlate log data and quickly query to locate events of interest.
Whether your code is under development, is being tested as part of unit or integration tests, or in production, detailed logging is the foundation for you to see and understand what is actually happening in your code for each request.
Detailed, intelligent logging is the foundation for serverless observability.
Observability is the ability to understand your system’s unknown unknowns. i.e. the ability to understand not just expected errors, but also unforeseeable conditions. It is not enough to just log simple, plain text error messages.
Our understanding of all the possible the failure modes of serverless apps is limited by the extent of our knowledge today. Serverless apps can fail in many surprising ways that we may not yet understand and cannot foresee.
So to meet the debugging needs of tomorrow, you need to log much more information and context than you might realize. Your app must emit sufficient state and information to permit debugging a wide range of unexpected conditions in the future without having to stop and redeploy code.
Fortunately, this is easy to do.
Often developers add diagnostic logging as an after thought via sprinkled console.log
messages.
console.log("Oops, bad thing happened");
However, using console.log
to emit simple messages is almost useless in achieving true Observability. Instead, what is required is the ability to emit log messages in a structured way with extra context that captures detailed request state.
Every time a developer uses
console.log
an angel cries in heaven.
In addition to the log message, log events should emit additional context information. To achieve this, the output format must be structured. JSON is an ideal format as most log viewers can effectively parse JSON and understand the context emitted.
For example:
log.error(`Cannot authenticate user`, {user, auth, req});
This would emit something like:
{
"timestamp": "2022-03-26T06:41:59.216ZZ",
"message": "Cannot authenticate user",
"type": "error",
"user": { ... },
"auth": { ... },
"req": { ... },
"stack": [ /* stack trace */ ]
}
There are many log libraries that are capable of emitting structured log context data. For SenseDeep, we use the SenseLogs library which is an exceptionally fast logging library designed for serverless. It has a flexible, simple syntax that makes adding detailed log events easy.
SenseLogs emits log messages with context and stack traces in JSON. It can also be dynamically controlled as to which log messages are emitted at runtime without having to redeploy code.
Here is a mock Lambda which initializes SenseLogs and emits various sample log messages:
const log = new SenseLogs()
exports.handler = async (event, context) => {
log.addTraceIds(event, context)
let body = JSON.parse(event.body)
log.info('Request start', {body, event, context})
log.info('New user login', {user, auth})
log.debug('Queue Stats', {q, backlog})
log.fatal('Unexpected exception with cart', {err, cart})
log.emit('Beta', 'Performance metrics for beta release features', {metrics})
}
SenseLogs emits messages into channels such as info
, debug
, error
. These channels can be enabled or disabled via Lambda environment variables. In this way, SenseLogs can dynamically scale up or down the volume of log data according.
It is very important to catch all errors and ensure they are reported. Lambdas should catch all exceptions and not rely on the default Lambda and language runtimes to catch errors. Employing a top-level catch can ensure that the exception report includes request state and correlation IDs.
All log data has a cost and extensive logging can significantly impact the performance of your Lambdas and excessive log storage can cause rude billing surprises. Logging extensive state for all requests is typically prohibitively expensive.
A better approach is to enable a solid baseline of log data and to scale up and/or focus logging when required without redeploying code. This is called Dynamic Logging and it is important to be able to do this without the risk of redeploying modified code.
SenseLogs implements Dynamic Logging by listening to a Lambda LOG_FILTER
environment variable. This variable may be set to a list of log channels to enable. By default, it is set to enable the following channels:
LOG_FILTER=fatal,error,info,metrics,warn
If the Lambda LOG_FILTER
variable is modified, the next and subsequent Lambda invocations will use the adjusted channel filter settings. In this manner, you can scale up and down the volume and focus of your log messages without redeploying code.
You can use custom channels via log.emit
to provide log coverage for specific paths in your code and then enable those channels on demand.
log.emit('feature-1', 'Unexpected condition, queue is empty', { /* state objects */ })
SenseLogs has two other environment variables which give more control over the log messages that are emitted.
LOG_OVERRIDE is a convenient way to temporarily boost log messages that will automatically revert to the base level of logging when the duration completes. LOG_OVERRIDE is set to a Unix epoch timestamp (seconds since Jan 1 1970) when the override will expire, followed by a list of channels.
LOG_OVERRIDE=1626409530045:data,trace
LOG_SAMPLE is set to a percentage of requests to sample followed by a list of channels to add to the list of LOG_FILTER for sampled requests.
LOG_SAMPLE=1%:trace
See the SenseLogs Documentation for more details.
You can modify environment variables with the AWS console:
But our SenseDeep serverless developer studio provides a much more convenient interface to manage Lambda LOG_FILTER settings for a single Lambda or for multiple Lambdas.
Your serverless apps should emit a good base of request state for all requests.
This should include the request parameters, request body and other high priority state information. The volume of log data should be low enough so that the cost of log ingestion and storage is not a significant burden.
For critical code modules or less mature code, we include additional logging on custom channels that correspond to each module or feature. This can then be enabled via LOG_FILTER
or LOG_OVERRIDE
.
LOG_FILTER=fatal,error,info,metrics,warn,feature-1
or
LOG_OVERRIDE=1626409530045:feature-1
For a small percentage of requests, we log the full request state so that we always have some record of all the state values.
We use a custom SenseLogs channel called complete
and we sample those requests at a low percentage rate.
log.emit('complete', { /* additional state objects */ })
LOG_SAMPLE=1%:complete
Serverless apps often have multiple Lambda functions or services that cooperate to respond to a single client request. Requests that originate with a single client request may traverse through many services and these services may be in different AWS regions or accounts.
A single request will often traverse through multiple AWS services such as: API Gateway, one more more Lambda functions, Kinesis streams, SQS queues, SNS messages, EventBridge events and other AWS services. The request may fan out to multiple Lambdas and results may be combined back into a single response.
To trace and debug an individual requests, we add a request ID to all our log events. This way we can filter and view a complete client request that flows though multiple Lambda services.
We add the trace IDs by using the SenseLogs API:
log.addTraceIds(event, context)
SenseLogs will extract the API Gateway request ID, Lambda request ID and X-Ray trace ID. SenseLogs will map these IDs to x-correlation-NAME SenseLogs context variables suitable for logging. The following variables are automatically detected and mapped:
SenseLogs will define a special variable ‘x-correlation-id’ that can be used as a stable request ID. It will be initialized to the value of the X-Correlation-ID header and if not defined, SenseLogs will use (in order) the API Gateway request or X-Ray trace ID.
SenseLogs also supports adding context state to the logger so you don’t have to specify it on each logging API call. Thereafter, each log call will add the additional context to the logged event output.
Finally we define a client ID that is passed from the end-user so we can track a “user request” through all AWS services that are invoked.
log.addContext({'x-correlation-client': body?.options.clientId})
Once you have added detailed logging with correlation IDs to your Lambdas, you need an effective log viewer to display and query log events.
CloudWatch just won’t cut it.
For our logging needs, we need a log viewer that can:
Our product, SenseDeep has such a log viewer that we use to monitor and debug the SenseDeep backend service.
Manually monitoring logs for app errors and signs of trouble is not scalable or reliable. An automated alarm mechanism when paired with detailed logging is the ideal platform to provide 24x7 oversight of your serverless apps.
We use such an automated approach. We configure multiple SenseDeep alarms to search for specific flags in log data that indicate errors or potential issues. If an alarm is triggered, an alert will immediately notify the relevant developer with the full context of the app event.
We instrument our code to not only capture and log errors, but also with “asserts” that test expected conditions and will emit log data that will trip alarms if unexpected conditions arise.
To reliably trigger alarms, it is necessary to have a unique property value that the alarm mechanism can detect.
The SenseLogs logging library supports a flagging mechanism where log events to specific log channels can add a unique matchable “flag” to the log data. Alarms can then reliably test for the presence of such a flag in log data.
We use SenseLogs to add property flags for errors and asserts. By default, SenseLogs will add a FLAG_ERROR
property for log events error()
or fatal()
.
For other log channels, the flag option can be set to an map of channels, and the nominated channels will be flagged with the associated value string. For example:
new SenseLogs({flag: {warn: 'FLAG_WARN', error: 'FLAG_ERROR', custom: 'FLAG_CUSTOM'})
log.warn('Storm front coming')
This will emit:
{
"message": "Storm front coming",
"FLAG_WARN": true
}
We then create a SenseDeep alarm to match FLAG_ERROR
and other FLAG_*
patterns.
We configure alarms for generic errors and exceptions as well as specific asserts and state conditions.
CloudWatch Alarms do not have the ability to efficiently monitor log data for specific log data patterns.
Fortunately, SenseDeep was built for this purpose.
SenseDeep developers create alarms to detect flagged errors when errors and unexpected conditions are encountered.
As SenseDeep ingests log data, it runs configured alarms to match app errors in real-time.
When an alarm triggers it generates an alert to notify the developer via email, SMS or other notification means.
The alert contains the full details of the flagged condition.
Clicking on the notification link will launch SenseDeep and display full details of the alert that triggered the alarm.
To see the log entries for the invocation, click on Goto Invocation
.
If you want to see the preceding or subsequent log entries, click All Logs
which will launch the log viewer homed to the invocation log data.
Using this serverless debugging pattern, we can detect problems early, typically before users are even aware of an issue.
The automated monitoring offloads the burden of “baby-sitting” a service and gives developers tools needed to ensure their apps are performing correctly.
You may wonder if you can use the native AWS services: CloudWatch and Xray to implement this serverless debugging pattern.
Short answer, you cannot.
CloudWatch does not aggregate log streams or groups. It is slow and only offers primitive query capabilities.
CloudWatch Insights does have the ability to search for correlation IDs in logs in a single region. But it is very, very slow and you cannot then see the request in context with the requests before or after the event. CloudWatch Insights is most useful for custom one-off queries, but not for repeated debugging efforts.
CloudWatch alarms cannot trigger alerts based on log data patterns. You can pair it with CloudWatch Insights, but this is a slow one-off solution and does not scale to process all log data for matching events.
Similarly, X-Ray has many good use cases, but because it only samples 1 request per second and 5% of additional requests, it cannot be used to consistently debug serverless applications. It is great for observing complete outages when an entire service or component is offline, but not for failing individual requests.
SenseDeep is a development studio for developers creating and evolving serverless apps. It offers fast, in-depth serverless design, troubleshooting and monitoring so that developers can quickly create and debug serverless applications. It provides AWS developers with critical tools to efficiently create, debug, deliver and maintain serverless applications.
SenseDeep includes an integrated suite of design-time DynamoDB tools including: table manager, single-table aware data browser, single-table designer, migration manager, provisioning planner and entity-level metrics.
SenseDeep provides request invocation traces, metrics, logs, alarms, alerts and notifications and watches over your services 24x7.
SenseDeep is unique as the only open architecture solution where your serverless data never leave your account and you have full access and control over your data.
SenseDeep is not for “Netflix” scale production sites. Rather it is for development teams that need insight to help accelerate their delivery as they develop and evolve their serverless apps.
SenseDeep organizes and aggregates essential AWS lambda metrics and CloudWatch logs data into a unified view.
You can visualize:
Integrated suite of DynamoDB tools to create and manage single-table designs.
Smart Data Browser to make sense of your single-table designs.
Single Table Designer to create and manage single-table schemas and models.
Migration Manager to orchestrate database migrations for your table.
Provisioning Planner to calculate and monitor your table provisioning and billing.
Detailed Metrics down to the single-table entity level.
Table Manager for quick/easy development time creation of tables and indexes.
One click on a function event brings up the full details of that invocation.
The inline log data for that invocation shows the precise invocation details. Log data is formatted and JSON color-coded.
Errors are highlighted so you can easily see the cause of any failures.
If required, you can jump to the full log and zoom to that invocation in context.
Organize Lambdas into applications for service level monitoring and management.
Unified dashboard with fully customizable dashboards with graphical and numeric widgets.
Display metrics for every AWS service and for custom application metrics.
Dashboards can display widgets for multiple AWS accounts in different regions.
Save and share dashboard views with team members.
Automate your error and performance issue detection with powerful alarm capabilities.
Create alarms for any application log event or EventBridge event, not just metrics.
Alarms can trigger on log event patterns and exclude noisy conditions for more effective alerts.
Create event-based alarms based on matching EventBridge events.
Dynamically subscribe new lambdas to alarms by tags or matching regular expression names.
Trigger notification alerts by email or SMS.
Smart notification dampening so you don’t get swamped by alerts.
Improve service performance with guided insights and recommendations.
Tune your function’s memory size, timeout limits, throttling, cold starts and function cost.
The SenseDeep log viewer is blazing fast and a joy to use.
It has infinite, buttery smooth scrolling.
Real-time, live-tail automatically displays new log data without clicking reload.
Powerful search and query to instantly locate important events.
Create and share multiple “views” of your log with custom filters and time ranges.
Search for important log events via full text queries.
Use regular expressions or key/value searches using a familiar Javascript expression language.
Exclude service related events to focus on application events of interest.
Add custom exclusion patterns to focus on events of interest.
SenseDeep will extract meaning from your log event structure and automatically map fields to columns for display.
Understands: JSON, Anchored patterns, CSV, TSV, Delimited, Key/Value pairs, Syslog and custom formatted logs.
Connect multiple AWS accounts for a single over-arching view of your service.
Connect multiple AWS regions.
Connect to AWS via a secure IAM role.
Seamlessly invokes CloudFormation template to setup your account.
With the SenseDeep Team plan, you can invite team members to share the same management configuration and views.
Invite outside users and consultants to be part of your SenseDeep organization.
Control access with detailed permissions. Grant view or admin access as required.
Easy revoking of access if needed.
Configure alerts to be resolved by specific team members.
SenseDeep has a unique open architecture design where your serverless and log data are stored within your account. Your data never leaves your account.
SenseDeep has an open, published data schema so you can fully access and utilize your data for custom needs.
Log data captured by a high-performance lambda in your account called the Watcher.
Recent data cached in a DynamoDB table in your account for secure, high-performance retrieval.
More secure as your logging identifiers and secrets are never sent over the wire to another service.
SenseDeep leverages the best of AWS without duplication. Our unique open transparent design utilizes the best of AWS services to give you complete access and control of your data.
]]>AWS CloudWatch offers metrics for monitoring specific aspects of your applications. However, AWS custom metrics can become costly when updated or queried frequently, with each custom metric costing up to $3.60 per metric per year, along with additional expenses for querying. If you have a significant number of metrics or high dimensionality in your metrics, this could result in a substantial CloudWatch Metrics bill.
On the other hand, CustomMetrics presents a cost-effective alternative metrics API that is considerably more budget-friendly and efficient compared to standard CloudWatch metrics.
The cost savings achieved by CustomMetrics are primarily due to its focus on providing only the most recent period metrics, such as those from the last day, last month, last hour, last 5 minutes, and so on. These metric timespans are also fully configurable. This approach ensures that each metric can be saved, stored, and queried with minimal cost.
Frequently, users complain that CloudWatch is one of the most expensive parts of their AWS bill. For those users that employ AWS metrics with high update frequency or high dimensionality, this can quickly translate into a large bill.
CloudWatch charges based on the number of metrics sent to the service and the frequency of updating or querying metrics. Your bill will increase as you send more metrics to CloudWatch and make API calls more frequently. For a regularly updated or queried metric, you will pay $0.30 per metric per month for the first $3,000 per month.
AWS metrics are expensive because they store metrics with arbitrary data spans. You can query metrics for any desired period (with decreasing granularity as the metrics age). On the positive side of the ledger, you can use CloudWatch EMF log format to emit metrics from Lambda’s without invoking an API. But you still pay for maintenance of the metric.
CustomMetrics foregoes the option for arbitrary date queries and provides “latest” period metrics only. When using CustomMetrics, you request data for specific recent time spans such as the last “5 minutes”, “hour”, “day”, “week”, “month”, or “year”. You have the flexibility to configure these “last” time spans according to your preferences, but they are always based on the current time. For example, you could record metrics for the last “minute” and “15 minutes”.
In exchange for its exclusive emphasis on the most recent metrics, CustomMetrics can store and retrieve metrics at a significantly lower cost compared to CloudWatch custom metrics.
CustomMetrics is a NodeJS library designed to emit and query custom metrics for AWS applications. Is offers the following features:
CustomMetrics stores each metric in a single, compressed DynamoDB item. Each metric stores the optimized data points for the metric’s timespans. The default spans are 5 mins, 1 hour, 1 day, 1 week, 1 month and 1 year. But these can be configured for each CustomMetric instance.
Emitting a metric via the emit
API will write the metric via a DynamoDB item update. Multiple simulataneous clients can update the same metrics, and CustomMetrics will ensure no data is lost.
If optimized metric buffering is enabled, metric updates may be aggregated according to your buffering policy to minimize the database write load.
Here is a quick tour of CustomMetrics demonstrating how to install, configure and use it in your apps.
First install the library using npm:
npm i custom-metrics
Import the CustomMetrics library. If you are not using ES modules or TypeScript, use require
to import the library.
import {CustomMetrics} from 'CustomMetrics'
Next create and configure the CustomMetrics instance by nominating the DynamoDB table and key structure to hold your metrics.
const metrics = new CustomMetrics({
table: 'MyTable',
region: 'us-east-1',
primaryKey: 'pk',
sortKey: 'sk',
})
Metrics are stored in the DynamoDB database referenced by the table name in the desired region. This table can be your existing application DynamoDB table and metrics can safely coexist with your app data.
The primaryKey and sortKey are the primary and sort keys for the main table index. These default to ‘pk’ and ‘sk’ respectively. CustomMetrics does not support tables without a sort key.
If you have an existing AWS SDK V3 DynamoDB client instance, you can use that with the CustomMetrics constructor. This will have slightly faster initialization time than simply providing the table name.
import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
const dynamoDbClient = new DynamoDBClient()
const metrics = new CustomMetrics({
client: myDynamoDbClient,
table: 'MyTable',
region: 'us-east-1',
primaryKey: 'pk',
sortKey: 'sk',
})
You can emit metrics via the emit
API:
await metrics.emit('Acme/Metrics', 'launches', 10)
This will emit the launches
metric in the Acme/Metrics
namespace with the value of 10.
A metric can have dimensions that are unique metric values for specific instances. For example, we may want to count the number of launches for a specific rocket.
await metrics.emit('Acme/Metrics', 'launches', 10, [
{rocket: 'saturnV'}
])
The metric will be emitted for each dimension provided. A dimension may have one or more properties. A metric can also be emitted for multiple dimensions.
If you want to emit a metric over all dimensions, you can add {}. For example:
await metrics.emit('Acme/Metrics', 'launches', 10, [
{},
{rocket: 'saturnV'}
])
await metrics.emit('Acme/Metrics', 'launches', 10, [
{},
{rocket: 'falcon9'}
])
This will emit a metric that is a total of all launches for all rocket types.
To query a metric, use the query
method:
let results = await metrics.query('Acme/Metrics', 'speed', {
rocket: 'saturnV'
}, 86400, 'max')
This will retrieve the speed
metric from the Acme/Metrics
namespace for the {rocket == 'saturnV'}
dimension. The data points returned will be the maximum speed measured over the day’s launches (86400 seconds).
This will return data like this:
{
"namespace": "Acme/Metrics",
"metric": "launches",
"dimensions": {"rocket": "saturnV"},
"spans": [{
"end": 946648800,
"period": 300,
"samples": 10,
"points": [
{ "sum": 24000, "count": 19, "min": 1000, "max": 5000 },
]
}]
}
If you want to query the results as a single value over the entire period (instead of as a set of data points), set the accumulate
options to true.
let results = await metrics.query('Acme/Metrics', 'speed', {
rocket: 'saturnV'
}, 86400, 'max', {accumulate: true})
This will return a single maximum speed over the last day.
To obtain a list of metrics, use the getMetricList
method:
let list: MetricList = await metrics.getMetricList()
This will return an array of available namespaces in list.namespaces.
To get a list of the metrics available for a given namespace, pass the namespace as the first argument.
let list: MetricList = await metrics.getMetricList('Acme/Metrics')
This will return a list of metrics in list.metrics. Note: this will return the namespaces and metrics for any namespace that begins with the given namespace. Consequently, all namespaces should be unique and not be substrings of another namespace.
To get a list of the dimensions available for a metric, pass in a namespace and metric.
let list: MetricList = await metrics.getMetricList('Acme/Metrics', 'speed')
This will also return a list of dimensions in list.dimensions.
You can scope metrics by chosing unique namespaces for different applications or services, or by using various dimensions for applications/services. This is the preferred design pattern.
You can also scope metrics by selecting a unique owner
property via the CustomMetrics constructor. This property is used, in the primary key of metric items. This owner defaults to ‘default’.
const cartMetrics = new CustomMetrics({
owner: 'cart',
table: 'MyTable',
primaryKey: 'pk',
sortKey: 'sk',
})
All feedback, discussion, contributions and bug reports are very welcome.
SenseDeep can be used to view CustomMetrics graphs and data. You can also create alarms and receive alert notifications based on CustomMetric data expressions.
You can contact me (Michael O’Brien) on Twitter at: @mobstream, and read my Blog.
]]>This video covers:
If you prefer a read instead, checkout Getting Started with SenseDeep.
I’m happy to announce SenseDeep 3.9 with support for CustomMetrics.
This release eliminates any SenseDeep CloudWatch Metrics costs and offers faster, low cost metrics for your apps.
Previous SenseDeep releases used CloudWatch Metrics for extended Lambda and DynamoDB metrics.
However, CloudWatch charges based on the number of metrics sent to the service and the frequency of updating or querying metrics. So the use of these CloudWatch metrics became a significant cost for users with a large number of Lambdas.
This release replaces the use of CloudWatch Metrics with [CustomMetrics](https://npmjs.com/package/custom-metrics] for Lambda, DynamoDB and for metrics for your apps.
CustomMetrics presents a cost-effective alternative metrics API that is considerably more budget-friendly and efficient compared to standard CloudWatch metrics.
The cost savings achieved by CustomMetrics are due to its focus on providing only the most recent period metrics, such as those from the last day, last month, last hour, last 5 minutes, and so on.
We still support CloudWatch metrics of course, but internally, SenseDeep has eliminated the use of CloudWatch Metrics to offer a more cost effective solution.
The SenseDeep dashboard and widgets have been updated to support data from CustomMetrics and AWS CloudWatch metrics.
When configuring widgets to display CustomMetric data, you select the DynamoDB table containing the metrics (and similarly to CloudWatch metrics) the namespace, metric name, statistic and resource dimensions.
When configuring AWS CloudWatch metrics, you enter “AWS CloudWatch” in the Origin field, then select the SenseDeep Cloud, namespace, metric name, statistic and ressource dimensions. To avoid confusion, AWS CloudWatch metric namespaces are now prefixed with “@”. For example: “@AWS/Billing”.
The Lambda Studio, and App Studio are now faster to gather and render Lambda metrics due to CustomMetrics. But otherwise, the functionality is unchanged.
The DynamoDB Studio supports detailed single table metrics. Previously, the cost of single table metrics could be very large due to the high dimensionality of the OneTable metrics. OneTable now also supports CustomMetrics and so the cost of these metrics is dramatically reduced.
Here is some background on CustomMetrics. You can read all about them in the blog post: Custom Metrics – Simple, Cost-Effective Metrics for AWS.
CustomMetrics is a NodeJS library designed to emit and query custom metrics for AWS applications. Is offers the following features:
SenseLogs is a simple, flexible, dynamic, blazing fast log library designed exclusively for serverless apps using NodeJS.
While there are many other good logging libraries that claim to be flexible and fast, they were not designed for serverless. They are thus bigger and slower than necessary and don’t meet the unique logging needs of serverless.
SenseLogs is more than 6x faster than the Pino logging library.
Serverless apps have special requirements like minimizing cold-start time and being exceptionally efficient due short execution lifespans of most Lambda invocations. Furthermore, the ephemeral nature of serverless makes it difficult to remotely control the volume and type of emitted log messages without redeploying code.
SenseLogs was designed to work in this environment and offers a fast, dynamic logging library exclusively for serverless. SenseLogs offers a flexible API to simply log data with rich context in JSON and then dynamically control which messages are emitted at run-time without redeploying your functions.
Here are some of the key features of SenseLogs:
Install the library using npm or yarn.
npm i senselogs
Import the SenseLogs library. If you are not using ES modules or TypeScript, use require
to import the library.
import SenseLogs from 'senselogs'
Create and configure a logger.
const log = new SenseLogs()
Then log with a message:
log.info('Simple messsage')
log.error(`Request error for user: ${user.email}`)
You can also supply addition data to log via a map of additional context information.
log.error('Another log error message with context', {
requestId: 1234,
userId: '58b23f29-3f84-43ff-a767-18d83500dbd3'
})
This will emit
{
"message": "Another log error message with context",
"requestId": 1234,
"userId": "58b23f29-3f84-43ff-a767-18d83500dbd3"
}
To implement observability for your serverless apps, you need to proactively embed logging that captures rich and complete request state. This logging needs to impose little to no overhead and ideally, should be remotely manageable to activate without needing to redeploy code.
SenseLogs achieves this via log channels and log filters. SenseLogs organizes log messages via channels which are names given to classify log message types. You can then select log messages to emit by filtering on channel names.
SenseLogs provides standard channels like: debug, error, fatal, warn and info.
log.error('Bad things happen sometimes')
log.error(new Error('Invalid request state'), {requestId})
log.debug('The queue was empty')
log.trace('Database request', {request})
You can extend upon this basic set of channels and use your own custom channels via the emit
API. For example:
log.emit('custom-channel', 'My custom channel')
SenseLogs selects log messages to emit depending on whether the log channel is enabled in the log filter. The log filter is a set of channel names that are enabled for output.
The default log filter will emit messages for the fatal, error, metrics, info and warn, channels. The data, debug and trace channels will be hidden by default.
You can change the filter set at any time to add or remove filters. For example, to enable the output of log messages for the data and debug channels.
log.addFilter(['data', 'debug'])
It is highly desirable to emit detailed request information in each logging call such as X-Ray IDs, API Gateway request IDs and other critical context information. It is cumbersome to encode this information explicitly in each logging call. It is much better to define once and have it inherited by each logging call.
SenseLogs supports this by providing additional log information via contexts. These can be defined and updated at any point and will be included in the logged data by each logging call.
log.addContext({ userId })
log.info('Basic message', {
event: "Login event",
})
This will emit the logged message with the additional contexts:
{
"message": "Basic Message",
"event": "Login event",
"userId": "58b23f29-3f84-43ff-a767-18d83500dbd3"
}
SenseLogs manages log filtering via environment variables that determine the log filter sets. If you change these environment variables, the next time your Lambda function is invoked, it will be loaded with the new environment variable values. In this manner, you can dynamically and immediately control logging output without modifying code or redeploying.
SenseLogs keeps three log filter sets:
The default
set defines the base set of log channels that are enabled for output. The override
set is added to the default set for a limited time duration. The sample
set is added to the default set for a percentage of log requests.
You can change these environment values via the AWS console or via the AWS CLI.
Better still, the SenseDeep Serverless Studio provides an integrated, high-level way to manage these filter settings.
AWS CloudWatch can receive custom metrics via log data according to the Embedded Metric Format (EMF).
SenseLogs makes it easy to emit your application metrics via the SenseLogs metrics
log channel. For example:
log.metrics('info', 'Acme/Rockets', {Launches: 1})
Once emitted, you can view and track your metrics in the AWS console or in the SenseLogs dashboard.
Because SenseLogs was designed exclusively for serverless, it does not carry unnecessary enterprise baggage and is blazing fast for serverless logging tasks.
Here are the results of benchmarks against the self-claimed fastest logger Pino.
SenseLogs 6.5 times faster than the best alternative.
Logger | Time | Code Size |
---|---|---|
SenseLogs | 477 ms | 478 lines (commented) |
Pino | 3,269 ms | 1281 lines (comments stripped) |
SenseLogs is provided open source (MIT license) from GitHub SenseLogs or NPM SenseLogs.
You can read more in the detailed documentation including the full API at:
]]>CloudWatch has many good services, but they are hidden behind a challenged UI.
SenseDeep uses the underlying CloudWatch services, but provides an easier user interface that makes spelunking over your logs much less painful. And with the SenseDeep Developer plan its FREE.
AWS CloudWatch is a convenient, unified logging solution so you can collect and access all your operational and performance data in one place. However, while the CloudWatch capture and storage facilities are excellent, the viewing and querying options are slow and limited.
I’ve tried a number of offerings that build upon the CloudWatch Logs service, but never found the experience I was looking for.
As a developer, I wanted:
The AWS log console, while usable, is scoped to a single AWS account and region and is too slow to render and query log events. It takes up to 2-4 seconds to load page by page of log events. The CloudWatch Insights product is more powerful for queries, but can take over a minute for a single query.
To address these needs, and build upon the CloudWatch foundation, we designed the SenseDeep log viewer.
Here is a quick comparison of CloudWatch Logs and how SenseDeep augments these capabilities to offer a more complete logging solution.
CloudWatch Logs | SenseDeep | |
---|---|---|
Capture Log Data | ||
Retain Log Data | ||
Manage Retention Policies | ||
Log Viewer | basic | advanced |
Time to show next page | > 1-2 seconds | < 1/100th second |
Time to page through 1000 events | >20 seconds | <1/2 second |
Automatically Aggregate Log Streams | ||
Correlate and combine multiple log groups | ||
Combine logs across AWS accounts and regions | ||
Transparent Event Pre-Cache | ||
Live Refresh and Live Tail | ||
Infinite Smooth Scroll | ||
Understand Structured Events | ||
Filter and Query Events | Separate CloudWatch Insights | Integrated Javascript expressions |
Save Log Views and View Formatting |
SenseDeep is dramatically faster than other solutions because it has a unique architecture. SenseDeep runs a lambda in your account called the SenseDeepWatcher
which ingests log data and caches recent data in a high-performance logging database in your account. This means your log data is never shipped over the public internet to a 3rd party storage facility.
SenseDeep also intelligently pre-fetches log data to anticipate your needs. SenseDeep fetches log data in the background and caches it locally. As you scroll forward or backward, SenseDeep transparently fetches new log events to anticipate your need. With SenseDeep, events can be displayed, queried, and scrolled with lightning speed.
Serverless apps are an immensely powerful way to build highly scalable and available services. But, serverless apps are also uniquely difficult to monitor, debug and manage due to their distributed components, stateless nature, short execution lifespans and limited access to configuration.
Observability has been popularized as the solution to this dilemma by “instrumenting your functions verbosely” and “wrapping all service boundaries with copious logging”. And … I’m a big fan of real Observability.
However, this can degrade critical production performance and send logging and metric costs spiraling.
So what is the solution?
This post explores a simple, yet effective solution via Dynamic Log Control without compromising Observability.
Distributed serverless systems need to be easily monitored for current and future failures. To achieve this, such systems need to be “Observable”.
Observability aims to provide granular insights into the behavior of a system with rich context to diagnose current errors and anticipate potential failures. Observable systems integrate telemetry to capture detailed information about the behavior of the system.
Easy you say, just add lots of logging and metrics.
However the painful tradeoff with observability is the hard cost of the telemetry vs the potential benefit of that observability in the future. Each line of instrumentation costs in terms of CPU and log storage. Copious logging can lead to rude performance and cost surprises.
What is needed is a way to minimize the cost of telemetry and to be able to scale it up and down quickly without redeploying.
Of course, you need a strong baseline of logging and metrics so you can have an accurate assessment of your services. But you often need greater insight when digging deep into an issue.
Serverless apps pose challenges for the dynamic scaling up and down of logging.
Serverless functions are ephemeral. They are not like an EC2 server or long lived docker applications. They often last only milliseconds before terminating. They keep little or no state and fetching state from a database may be too great an overhead to incur on every function invocation.
However, there is a technique that can be adapted for dynamic log control that is well proven and understood: Good Old Environment Variables.
When AWS Lambda loads a function, it provides a set of environment variables that can be read by the function with zero delay. Via these variables, you can provide log control instructions to vary the amount and focus of your logging.
Here is the AWS console’s Lambda environment configuration page.
If you change the environment variables (without redeploying or modifying your function), your next invocation will run with the new environment values.
So using special LOG environment variables, we can communicate our desired log level, filtering and sampling, and have our functions respond to these settings without modifying code or redeploying. Sweet!
What are the downsides: Changing environment variables will incur a cold-start the next time the functions are invoked, but that is typically a short, one-off delay.
To control our logging, we propose three log environment variables:
Depending on your log library (SenseLogs, Pino, Winston, …) you specify your log level or channel via these variables.
The LOG_FILTER defines log level. Enabled levels will emit output. Disabled levels will be silent. For example:
LOG_FILTER=5
or if you are using SenseLogs you specify a set of log channels.
LOG_FILTER=error,warn
This will enable logging output in your Lambdas for log.error()
and log.warn()
but disable output for calls to trace
or debug
.
The LOG_OVERRIDE will replace the LOG_FILTER for a given period of time before reverting to the LOG_FILTER settings. LOG_OVERRIDE is prefixed with an expiry time expressed as a Unix epoch in milliseconds since Jan 1 1970.
For example:
LOG_OVERRIDE=1629806612164:5
or
LOG_OVERRIDE=1629806612164:debug,trace
The LOG_SAMPLE is a sampling filter that applies to a given percentage of requests. The list of levels is is prefixed with a percentage.
For example, this will trace 5% of requests.
LOG_SAMPLE=5%:trace
This technique will work with Node, Python and Java Lambda functions. It is easy to configure with most popular log libraries including: Bunyan, Debug, Pino, SenseLogs or Winston.
An ideal logging library will permit custom levels so that latent logging code can be embedded in your function without incurring a run-time overhead for most invocations. When required, you can add that custom level to your LOG_FILTER or LOG_OVERRIDE and turn on that custom log output. Custom levels should be able to be enabled and disabled without impacting other levels.
It is very important that your logging library has extremely low overhead for disabled log levels. Otherwise, the benefit of dynamically scaling is lost.
I’ve included below some examples for NodeJS using the SenseLogs and Pino logging libraries.
To see code samples for Bunyan, Debug, Winston or Python please checkout our Dynamic Logging Samples which has detailed code samples for each library and Python using these techniques.
Also available via GitHub gists at: Dynamic Log Control Gist
Here is a sample of how to use this technique with SenseLogs which has builtin support to respond to the LOG_FILTER, LOG_OVERRIDE and LOG_SAMPLE environment variables. SenseLogs is an extremely fast serverless logger that supports custom log channels. It has almost zero cost for disabled log levels.
import SenseLogs from 'senselogs'
let log = new SenseLogs()
exports.handler = async (event, context) => {
// This will be emitted if LOG_FILTER contains 'debug' as a log level
log.debug('Hello world')
// EMF metrics can also be dynamically controlled
log.metrics('trace', 'AcmeRockets/CriticalLaunchFailure', {explosion: 1})
}
Learn more at Serverless Logging with SenseLogs.
Here is a sample of how to use the popular Pino general purpose logger.
This sample uses a small amount of code to parse the LOG environment variables and setup the Pino logger.
import Pino from 'pino'
let {LOG_FILTER, LOG_OVERRIDE, LOG_SAMPLE} = process.env
if (LOG_OVERRIDE != null) {
let [expire, level] = LOG_OVERRIDE.split(':')
if (level && Date.now() < expire) {
LOG_FILTER = level
}
}
let sample = 0, sampleRate, sampleLevels
if (LOG_SAMPLE != null) {
[sampleRate, sampleLevel]s = LOG_SAMPLE.split(':')
}
const pino = Pino({name: 'pino', level: LOG_FILTER})
exports.handler = async (event, context) => {
if (sampleRate && (sample++ % (100 / sampleRate)) == 0) {
// Apply the sample levels
pino.level = sampleLevel
}
pino.debug('Debug message')
}
When you have an important issue to diagnose, you can now scale up your logging by setting the LOG_OVERRIDE environment variable to increase the log level for a period of time. Depending on your log library, you may also be able to focus your logging on a specific module by using custom levels.
You can modify your environment configuration via API, the AWS Console, the AWS SDK or using the SenseDeep Developer Studio.
The AWS CLI has support to update the function configuration using update-function-configuration.
For example:
EXPIRES=$((($(date +%s) + 3600) * 1000))
aws lambda update-function-configuration --function-name MyFunction \
--environment "Variables={LOG_OVERRIDE=${EXPIRES}:error,info,debug,trace,commerce}"
This calculates an expiry time and overrides the log levels for one hour to include the error, info, debug and trace levels and the custom commerce
level.
You can also change these environment values via the AWS console.
Simply navigate to your function, select Configuration
then Environment Variables
then Edit
and modify then save.
Better still, the SenseDeep Serverless Developer Studio provides an integrated, high-level way to manage these filter settings.
Using this technique, you can easily and quickly scale up and down and focus your logging as your needs vary. You can keep a good base level of logging and metrics without breaking the bank, and then increase logging on-demand.
The logging library code samples are available at Log Control Samples.
SenseLogs is available from GitHub SenseLogs or NPM SenseLogs.
And you can get a free developer license for SenseDeep at SenseDeep App or learn more at https://www.sensedeep.com.
]]>If you want improved Observability, but the cost of logging in production is prohibitive, then SenseDeep dynamic logging may be your solution.
SenseDeep supports on-demand Dynamic Log Control to scale up or down the volume and focus of your serverless logging.
You can use this to include a base level of log telemetry in your application and then increase or focus logging when the need arises. This improves the Observability of your serverless functions without having a heavy constant price due to excessive logging.
The entire concept of Dynamic Log Control is best explained in this post Dynamic Log Control and you may want to read that first.
To control log output, SenseDeep manages three environment variables:
These variables control which log levels or channels are enabled for output. Together, they control the base logging level, a time-limited override and background sampling of logs for a percentage of requests.
SenseDeep log control supports a wide selection of platforms and log libraries including: Node, Python and Java, and the Node log libraries Bunyan, Debug, Pino, SenseLogs or Winston.
See Dynamic Log Control for Log Libraries for code samples for each platform and library.
The LOG_FILTER defines the set of enabled log levels or channels. Enabled levels will emit output. Disabled levels will be silent. For example:
LOG_FILTER=error,warn
This will enable logging output in your Lambdas for log.error()
and log.warn()
but disable output for calls to trace
or debug
.
The LOG_OVERRIDE will replace the LOG_FILTER for a given period of time before reverting to the LOG_FILTER settings. LOG_OVERRIDE is prefixed with an expiry time expressed as a Unix epoch in milliseconds since Jan 1 1970.
For example:
LOG_OVERRIDE=1629806612164:debug,trace
The LOG_SAMPLE is a sampling filter that applies to a given percentage of requests. The list of levels is is prefixed with a percentage.
For example, this will trace 5% of requests.
LOG_SAMPLE=5%:trace
SenseDeep provides a convenient one-stop interface to control your logging levels via the LOG* parameters.
When you view a Lambda function, click on Edit
to display the Lambda Logging slide-out panel. From here, you can define the LOG_FILTER which specifies the log level or channels to enable. If your log library uses numeric levels, enter the level here. If your log library uses alphabetical levels such as info
or error
enter those. Some libraries such as SenseLogs support a comma separate set of channels.
If you need to diagnose an issue or debug a Lambda function, enter increased log levels in the LOG_OVERRIDE. Enter a time period after which the override will be lifted.
It is often desirable to fully log a small percentage of requests so that you always have a complete request trace. To do this, enter a LOG_SAMPLE and the percentage rate that you wish to use increased sample logging.
If you use the SenseLogs logging library, you can emit CloudWatch custom metrics via log commands.
SenseLogs emits AWS CloudWatch custom metrics via the Embedded Metric Format (EMF). For example:
log.metrics('metrics', 'Acme/Rockets', {Launches: 1})
Using the SenseDeep log control, you can enable or disable these metrics by enabling or disabling the metrics
log level.
Once emitted, you can view and track your metrics using the SenseLogs dashboard and create Alarms and Alerts.
SenseDeep makes it easy to instantly scale up or down your logging as your needs vary. You can keep a good base level of logging and metrics without breaking the bank, and then increase logging on-demand.
The logging library code samples are available at Log Control Samples and SenseLogs is available from NPM SenseLogs.
If you are using serverless micro-services and AWS Lambda and you need to trace and debug requests, then you may have struggled to track requests as they progress across multiple services perhaps traversing different AWS regions and accounts.
If so, then SenseDeep correlated logs may be the solution for you.
Serverless applications often have multiple Lambda functions or services that cooperate to respond to a single client request. Requests that originate with a single client request may traverse through many services and these services may be in different AWS regions or accounts.
A single request will often traverse through multiple AWS services such as: API Gateway, one more more Lambda functions, Kinesis streams, SQS queues, SNS messages, EventBridge events and other AWS services. The request may fan out to multiple Lambdas and results may be combined back into a single response.
Unfortunately, this means that the log data for a single request is scattered over multiple AWS CloudWatch log groups that are potentially in different regions or accounts.
Consequently, diagnosing a single request can be like searching for a needle in a haystack. Tracing a request over multiple log groups can be difficult and time consuming.
SenseDeep addresses this issue via correlated (meta) logs that combine multiple AWS CloudWatch log groups in real-time into a unified correlated log view.
Using CloudWatch Insights, you can search for a single request, however this method has some key limitations – it will only work within a single AWS region or account.
For example, you can use the following CloudWatch query to retrieve log events from multiple log groups. First you select each of the logs for the query, then filter the log entries that match the request ID.
fields @timestamp, @log, @message, x-correlation-id
| filter @message like /MENkHjqOIAMESfg=/
This extracts the log messages that contain the given message pattern string. While this manual method works, is slow to setup, isn’t very scalable and has a few major limitations.
First and foremost, it can be slow, … very slow. CloudWatch insights often takes 20+ seconds to fetch results and can take up to 15 minutes if the event you are searching for an event in the past that did not happen very recently.
Second, after finding the matching log events, you cannot see the logs either just before or after the event. If the root cause of the request failure is in the log events just prior to the request, you cannot see that failure.
Third, the logs must be in the same account and region. You cannot correlate requests across different AWS accounts or regions. If your services are delivered from different AWS accounts or regions, you are out of luck.
Another potential solution is AWS X-Ray. X-Ray will trace requests across AWS services, but it too has some critical limitations.
X-Ray only samples 1 request per second and 5% of additional requests. So the percentage of sampled requests is low and the probablity that your request error is not sampled and monitored is high. X-Ray is useful for diagnosing complete service failures or repeated service failures, but not single request issues.
X-Ray also requires a lot of additional code that bloat your lambdas. Furthermore your code must be modified to initialize and configure X-Ray. This is intrusive and impacts the performance of your lambdas.
SenseDeep addresses these issues via correlated meta Logs and does not suffer from these limitations.
A SenseDeep meta log is a correlated view over multiple CloudWatch log groups that can be in any AWS region or account. From a meta log, you can view log events ordered in sequence regardless of the AWS service or log group. This log view behaves like any other log view, except it combines log groups and streams from multiple sources regardless of AWS account or region.
From the meta log, you can immediately locate and isolate any specific request by searching for a request ID or pattern of your choosing to isolate the complete request trace.
To create a meta log, select “Logs” from the side menu and then click “Add”. Enter your meta log name and select the logs to combine.
You can explicitly select the individual contributing logs by entering a regular expression pattern to dynamically match contributing logs. Using a regular expression pattern is preferable if you have a changing set of log group names.
Alternatively, you can select specific logs via the log list combo box.
Once created, you can use the SenseDeep log viewer to display a unified view of the logs for the selected services and events.
Here is a sample view of a request that flows through two lambdas, an EventBridge bus and a final lambda.
From the log viewer, you can filter by a correlation ID by double clicking on that column which will display the view configuration panel.
The correlation ID will be automatically extracted from the log event and be pasted into the Match Events
form field. Click Run
and the specified request will be selected and all other events will filtered out.
To get the most out of meta logs, you should utilize a unique request ID that is passed to your Lambda functions and propagated to any downstream AWS services. This correlation ID should be emitted in all log events. This is called a High Cardinality ID. Using this ID, you can filter log events using this ID and display only those events for that request ID.
We use the use the SenseLogs library to manage correlation IDs. SenseLogs will extract trace IDs including the Lambda and X-Ray trace IDs from the request context. It will map and emit these as a x-correlation-id
log property.
For example:
const log = new SenseLogs()
exports.handler = async (event, context) => {
log.addTraceIds(event, context)
log.info('Request start', {body, event, context})
}
For more details about SenseLogs, see:
For effective serverless development and debugging, you need to be able to quickly correlate and isolate requests across multiple cooperating AWS services such as API Gateway, Lambda, SQS and SNS. SenseDeep provides a real-time, fast meta log facility to create a unified view of a request as it passes through multiple AWS services.
Try the SenseDeep Serverless studio with a free developer license at SenseDeep App.
SenseDeep has a complete DynamoDB developer studio to support your DynamoDB designs and single-table development.
It includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics.
SenseDeep DynamoDB studio is a comprehensive set of DynamoDB tools that are single-table “aware”. This means SenseDeep can understand your single-table designs and application entity data and can guide your design, queries and monitoring based on this deeper understanding.
If you prefer a video instead, watch SenseDeep DynamoDB Studio.
DynamoDB best practices are evolving quickly as developers are realizing how to exploit the power behind the deceptively simple DynamoDB design. Designs with single-table design patterns, key overriding and composition, sparse indexes, query optimization and more powerful single-table access libraries such as OneTable are becoming common place.
However, managing single-table data and performance can often feel like you are peering at Assembly Language. Packing disparate data items into a single-table can make navigating, organizing and viewing data difficult. Furthermore, single-table design techniques such as prefixed and mapped attribute names exacerbate this problem and can make interpreting keys tough.
New tools are needed that understand the single-table schema and its relationships. These tools should support schema creating and be able to present and organize your data logically according to your application entities.
The SenseDeep data browser can query, manage and modify your table data. It supports browsing by scan, query or by single-table entities.
While the data browser can be used to browse and manage any DynamoDB table, it is turbo-charged when a schema describing the data is imported (such as from a OneTable schema) or defined using the SenseDeep single-table designer. When using a data schema, SenseDeep is able to “understand” your data and organize and present data items as application entities instead of raw, encoded DynamoDB items.
The SenseDeep data browser intelligently displays and formats data according to your single-table schema. This not only makes your encoded single-table data easier to understand, but it guides and validates your changes and prevents schema-breaking and application breaking data changes.
SenseDeep supports accessing your data via DynamoDB scans, queries or queries by application entity.
When scanning, you select the index to scan and provide optional additional filtering attributes. Attributes can be combined in an AND
or OR
expression.
When querying, you specify the index and partition key with optional sort key value. Additional filters may also be specified.
When querying by Entity, you specify the index and application entity model name that is defined in the schema. SenseDeep then intelligently prompts you for the required attributes that comprise the keys to retrieve the data items.
Queries may be saved to your database table where they are persisted in the schema. You can load and delete queries using the Queries
button.
SenseDeep groups, organizes and color-codes query results for maximum clarity.
Columns are ordered with the index partition and sort key first, followed by the schema entity type attribute (if defined). After that, columns for all items are ordered alphabetically. If an item has many attributes, click the edit icon to display the item attributes vertically.
Cells are color coded:
If you control-click (or Cmd-Click) on a hot-link, you can quickly traverse related items. SenseDeep determines the relevant query to locate the item and automatically fills the index, key and filter attributes accordingly. You can save these queries using Queries -> Save
. Queries are entered into your browser history so you can click the browser Back
button to easily jump backwards to the original query.
A useful single-table design technique is to compose key values using templates that combine the values of other attributes at runtime to calculate the key values. It is often useful to view the value templates vs the calculated key values. The Templates
toggle above the table switches between displaying the template values vs the actual data values.
Similarly, there may be some attributes that are designated as hidden
in the schema. Changing the hidden toggle will display or hide these attributes.
On data cells, you can right click to display the context menu. The options are:
Copy
to copy the current item to the clipboard in JSON format.Design Schema
will jump to the SenseDeep single-table designer for the entity on which the current item is based.Edit
will open the editor slide-out panel for easy editing of the whole item.Follow Reference
behaves the same as Cmd-Click to follow a hot link reference.Generate Mock Data
option can be used when developing to generate sample data, such as email addresses or phone numbers.You can modify data inline by clicking on any green
cell. Once modified, the Save
button will be displayed above the items to persist the changes to the table.
To add a new item, click the Add Item
button. You can select the desired entity model (if a schema is present) and it will intelligently prompt you for the appropriate attributes.
You can also edit by clicking on the Edit
pencil icon at the start of each item. This will display a slide out editor panel that will display only the attributes of the item, organized vertically. Click on a green cell to modify the contents. The red cells have their value derived from other attributes and cannot be edited. You can modify the value template in the schema via the SenseDeep single-table designer.
If you change the partition key values either directly or indirectly be changing attributes that are used in a key value template, SenseDeep will atomically remove the old item and create the new item via a DynamoDB transaction.
When you click save, the changes are accepted, but you still must click the Save button on the query page to persist results to the table.
No man is an island and your data must be easy to export or import. SenseDeep provides and Export to AWS Workbench option. This will export your schema and data items into a WorkBench model. This model can also be imported by the Dynobase tool.
When exporting, you should limit the amount of data you export. WorkBench models are for development and are designed for limited data sets. You can also export in a JSON backup format.
When importing, you can import a model to an existing (empty) table, or you can dynamically create a new table to hold the imported model.
The SenseDeep Single Table Designer provides an easy, intuitive interface to create, manage and modify your single-table schemas.
Single-table schemas control how your application data are stored in DynamoDB tables. Schemas define your application entities, table indexes and key parameters. Via schemas, complex mapped table items can be more clearly and reliably accessed, modified and presented.
The designer stores your single-table schemas in your DynamoDB table. In this manner, your table is self-describing as to how table data should be interpreted. You can export schemas and generate JSON, JavaScript or TypeScript data/code to import into your apps.
The SenseDeep single-table designer works best with the OneTable library, but should work with any consistent single-table design.
Schemas are named and versioned. The default schema is called the Current
schema. Other schemas can have any name of your choosing. For example “Prototype”.
Schemas have version numbers so that data migrations can utilize the correct versioned schema when upgrading or downgrading data items in your table. The migration manager will select and apply the correct versioned schema when data migrations are run.
Version numbers use Semantic Versioning to indicate and control data compatibility.
An entity model defines the valid set of attributes for an application entity. Each attribute has a defined name, type and a set of properties that may be utilized by your DynamoDB access library such as OneTable or an ORM of your choosing.
From the list of entity models, you can click on an entity to modify, click Add Model
to add a new entity model.
Via the Edit Schema
button you can modify the schema name or version. If the schema is a non-current schema, you can also click Apply to Current
to apply the contents of the displayed schema to the saved Current
schema in the table. This overwrites the previous Current
schema.
Clicking on an entity attribute will display a slide out panel to edit the properties of that attribute.
You can modify the type and any other properties of the type. These properties are defined in the schema and may (or may not) be implemented by your DynamoDB access library. All these are implemented by OneTable.
The properties include:
You can also export your schema in JavaScript, TypeScript or JSON formats via the Export Schema
button from the Models list. You can utilize the exported file directly in your OneTable apps to control your database interactions.
The DynamoDB Provisioning page displays your current and projected provisioning, utilization and costs.
While there are various DynamoDB calculators, they are “theoretical” and not based on actual data usage. The SenseDeep provisioning planner displays displays the actual costs of your current billing plan based on real, live data. It then compares this usage with the the cost of an alternate plan if you were to switch your billing plan from OnDemand to Provisioned or vice-versa.
The SenseDeep database migration manager provide a controlled way to upgrade or downgrade your table data.
It displays what migrations have been applied to your data and what migrations are outstanding. It shows what is the migration version for your current data and schema. You can upgrade by running outstanding migrations or downgrade by reversing migrations.
SenseDeep can orchestrate DynamoDB migrations controlled by the OneTable Migration library. You can use this library even if you are not using OneTable for your apps. Check out the OneTable Controller sample OneTable Controller in GitHub and deploy to host and manage your data migrations.
Read OneTable Migrate for more information and setup.
SenseDeep provides both standard AWS DynamoDB metrics and enhanced single-table metrics.
The DynamoDB standard metrics displays overview metrics and operation details for a whole DynamoDB table. These metrics come from the standard AWS CloudWatch DynamoDB metrics.
DynamoDB Single-table designs are more complex and require per entity/model performance monitoring and metrics. Traditional monitoring covers table level and operation level metrics only. What is missing is the ability to see single-table entities and their performance and load on the database.
SenseDeep provides single-table metrics for apps using DynamoDB OneTable or for any JavaScript app that utilizes DynamoDB Metrics.
SenseDeep can manage the DynamoDB tables for your enabled clouds. You can create and delete tables and indexes for your tables.
SenseDeep will automatically discover your tables and will dynamically update this list as new tables are created or destroyed. The table list includes the table size, number of items, billing scheme and provisioned capacity.
The SenseDeep table management is not meant to replace appropriate “infrastructure-as-code” deployment of tables to production. Rather, it intends to provide a quick and easy way to create and manage tables and indexes while developing your DynamoDB applications.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Dig into each part of the studio with:
You may also like to read:
]]>SenseDeep has a complete DynamoDB developer studio to support your DynamoDB designs and single-table development.
It includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics.
The SenseDeep DynamoDB studio is single-table “aware”. This means SenseDeep can understand your single-table designs and application entity data and can guide your design, queries and monitoring based on this deeper understanding.
Make sense of your encoded single-table data with the SenseDeep DynamoDB table data browser that is fully single-table aware. It will streamline your queries and intuitively understand your application entities when presenting item data.
Guided query or scan using single-table schema that understands your app entities.
Data presented as color-coded item collections and application entities.
Inline edit and load and save queries.
Navigate by single click to follow application entity relationship hot-links.
Import and export table data to Workbench or JSON files.
The Single Table Designer assists you to create your single-table entity schema. By defining your application schema first, your data access and queries become much simpler and easier to express and view.
Create single-table schemas and application entities.
Define entity field attributes, validations and other properties.
Define entity relationships for hot-linking.
Save schemas in-table for self-describing tables.
Integrated with Data Browser and Migration manager.
Export schema to OneTable.
The SenseDeep Migration Manager provides a controlled way to transform your DynamoDB data and manage data migrations. With it, you can easily and reliabily evolve your DynamoDB data design to upgrade or downgrade your table data without downtime.
Manage and easily evolve DynamoDB data with sequenced data migrations.
Upgrade and downgrade via coded migrations.
Repeat migrations for service related operations like removing old attributes.
Reset table data to known good state for development tables.
Graphical migration control panel showing current version with applied and outstanding migrations.
The SenseDeep provisioning planner displays displays the actual costs of your current billing plan based on real, historical and live data. It then compares this usage with the the cost of an alternate plan if you were to switch your billing plan from OnDemand to Provisioned or vice-versa.
Display current and projected provisioning, utilization and costs.
Graphically present read and write utilization vs provisioned capacity over time.
Displays actual historical costs vs estimated alternate billing plans.
Smart recommendations for provisioned capacity and billing plan.
SenseDeep DynamoDB metrics is a suite of in-depth performance metrics for DynamoDB. It exposes performance metrics at a granular level, down to each application entity or API invocation.
AWS account and table utilization metrics.
Single-table metrics down to the application entity level.
Track usage by tenant, source application or entity.
Drill-down for metrics on table, tenant, source, index, model and operation.
Custom query profiling for individual queries.
The SenseDeep table management provides a quick and easy way to create and manage tables and indexes while developing your DynamoDB applications.
Manage DynamoDB tables over multiple AWS accounts and regions.
Quickly create and delete tables and secondary indexes.
Monitor table size and billing mode.
]]>The SenseDeep Data browser is a DynamoDB data browser and editor that fully understands your single-table designs and schema.
It can query, manage and modify your DynamoDB data at the application entity level and make sense of complex single-table keys.
Developers are adopting DynamoDB single-table design patterns as the preferred design model where all application data is packed into a single-table. Combing disparate items with different attributes into one table can make browsing, navigating, organizing and viewing data obscure and difficult.
Managing single-table data and performance can often feel like you are peering at Assembly Language as it is hard to decode overridden keys and attributes manually. A new generation of tools is required.
SenseDeep can understand your single-table designs and make sense of your data and present items as intuitive application entities instead of raw data. The SenseDeep data browser is single-table “aware”. This means SenseDeep can transform raw data items to present as application entities and fields.
The SenseDeep data browser can query, manage and modify your table data. It supports browsing by scan, query or by single-table entities.
While the data browser can be used to browse and manage any DynamoDB table, it is turbo-charged when a schema describing your data is applied.
A scheme describes your application entities and their attributes. It specifies exactly how your data should be interpreted and what data is valid to store in the database table. You can import a schema from a JSON file (such as a OneTable schema) or you can define a schema using the SenseDeep single-table designer.
When armed with a schema, SenseDeep is able to “understand” your data and organize and present data items as application entities instead of raw, encoded DynamoDB items. The SenseDeep data browser intelligently displays and formats data according to your single-table schema. This not only makes your encoded single-table data easier to understand, but it guides and validates your changes and prevents schema-breaking and application breaking data changes.
The data browser supports DynamoDB scans, native queries or queries by application entity. Queries by application entity are normal DynamoDB queries but where SenseDeep understands which attributes are required to select and filter your data items.
When scanning, select the index to scan and provide optional additional filtering attributes. Attributes can be combined using AND
or OR
operators. SenseDeep translates these instructions into DynamoDB scan commands. You can click on the Command
button to see the generated DynamoDB command.
When querying, specify the index and partition key with optional sort key value. Additional filters may also be specified.
You can use sort key operations to select a single matching item with “Equal” or one or more items with the other Sort Key Operators from the pull down menu.
When querying by Entity, you select the desired entity model and SenseDeep will select the appropriate attribute filters for that model.
SenseDeep understands the schema for the selected model and what are the required attributes to retrieve specific entity items. Sensedeep will guide you with the list of attributes for that entity model.
Queries may be saved to your database table where they are persisted in the schema. The schema, saved in the database table, contains the entity definitions, saved queries and modeling data. This makes your table self-describing for 3rd party tools.
You can load and delete queries using the Queries
button.
SenseDeep groups, organizes and color-codes query results for maximum clarity.
Cells are color coded:
Columns are ordered with the index partition and sort key first, followed by the schema entity type attribute (if defined). After that, columns for all items are ordered alphabetically. If an item has many attributes, click the edit icon to display the item attributes vertically.
Databases like DynamoDB have relationships between items. Just because it is a NoSQL database, does not mean there are no relationships. It just means there is little support in terms of joining tables, enforced foreign keys and data relationship integrity.
SenseDeep and OneTable bridge this gap by defining relationships between entities in the schema. SenseDeep will interpret these relationship links and highlight those in the browser with blue underlining.
You can quickly traverse related items by clicking on a blue hot link in the query results.
SenseDeep defines relationships between entity items using the schema “Reference” field property. This property specifies that a field refers to another entity item and which attributes are required to uniquely identify the target item.
The format of the “Reference” field is:
Model:index:attribute-1=source-attribute-1,...
For example, the following Reference defines a link to an Account item:
Account:primary:id=accountId
This means: select the Account entity using the “primary” index and the “Account.id” attribute using the value of “accountId” from this item.
If you control-click (or Cmd-Click) on a hot-link SenseDeep uses the reference and determines the relevant query to locate the item and automatically fills the index, key and filter attributes accordingly. Queries are entered into your browser history so you can click the browser Back
button to easily jump backwards to the original item.
You can also save these hot-linked queries using Queries -> Save
.
An essential single-table design technique is decouple your keys from regular data attributes. This greatly enhances your ability to evolve your DynamoDB data going forward. SenseDeep and OneTable support this technique via value templates
where key values are composed using templates that combine the values of other attributes at runtime.
It is often useful to view the value templates vs the calculated key values. The Templates
toggle above the table switches between displaying the template values vs the actual data values.
Similarly, there may be some attributes that are designated as hidden
. Changing the hidden toggle will display or hide these attributes.
In any data cell, you can right click to display the context menu for additional command options.
The options are:
Copy
to copy the current item to the clipboard in JSON format.Design Schema
will jump to the SenseDeep single-table designer for the entity on which the current item is based.Edit
will open the editor slide-out panel for easy editing of the whole item.Follow Reference
behaves the same as Cmd-Click to follow a hot link reference.Generate Mock Data
option can be used when developing to generate sample data, such as email addresses or phone numbers.You can modify data inline by clicking on any green
cell. Once modified, the Save
button will be displayed above the items to persist the changes to the table.
To add a new item, click the Add Item
button. You can select the desired entity model (if a schema is present) and it will intelligently prompt you for the appropriate attributes.
You can also edit by clicking on the Edit
pencil icon at the start of each item. This will display a slide out editor panel that will display only the attributes of the item, organized vertically.
The red cells have their value derived from other attributes and cannot be edited. You can modify the value template in the schema via the SenseDeep single-table designer.
If you change the partition key values either directly or indirectly be changing attributes that are used in a key value template, SenseDeep will atomically remove the old item and create the new item via a DynamoDB transaction.
When you click save, the changes are accepted, but you still must click the Save button on the query page to persist results to the table.
No man is an island and your data must be easy to export or import. SenseDeep provides an “Export to AWS Workbench” option. This will export your schema and data items into a WorkBench model. This model can also be imported by the Dynobase app.
When exporting, you should limit the amount of data you export. WorkBench models are for development and are designed for limited data sets. You can also export in a JSON backup format.
When importing, you can import a model to an existing (empty) table, or you can dynamically create a new table to hold the imported model.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>The SenseDeep Single Table Designer is a DynamoDB design tool that can be used to create effective schema-driven, single-table design patterns.
Creating single-table designs has to date been an iterative voyage of exploration. There has been a paucity of design tools and the existing tools prototype data models via an item-by-item incremental fashion. While this demos well, it can result in inconsistent, brittle data models that are not rigorous in defining all application entities, attributes and relationships exactly.
The SenseDeep Designer uses a schema-first design methodology where you define your entities first and then test incrementally with data and queries.
The Designer is both single-table and schema “aware” which means it can validate your data model, data items and queries against your schema.
Going forward, armed with a well designed schema, you DynamoDB design is well equipped to evolve and manage your data in a safe, reliable and predictable manner.
The SenseDeep Single Table Designer provides an easy, intuitive interface to create, manage and modify your single-table schemas.
Single-table schemas control how your application data are stored in DynamoDB tables. Schemas define your application entities, table indexes and key parameters. Via schemas, complex mapped table items can be more clearly and reliably accessed, modified and presented.
Once created, your schema can be used with the SenseDeep data browser and the OneTable access library. It can also be imported into other tools such as the AWS WorkBench.
The Designer stores your single-table schemas in your DynamoDB table. In this manner, your table is self-describing as to how table data should be interpreted. You can export schemas and generate JSON, JavaScript or TypeScript data/code to import into your apps.
The SenseDeep single-table designer works best with the OneTable library, but should work with any consistent single-table design.
Schemas are named and versioned. The default schema is called the Current
schema. Other schemas can have any name of your choosing. For example “Prototype”.
Schemas have version numbers so that data migrations can utilize the correct versioned schema when upgrading or downgrading data items in your table. The migration manager will select and apply the correct versioned schema when data migrations are run.
Version numbers use Semantic Versioning to indicate and control data compatibility.
The SenseDeep / OneTable schema is documented and specified at Schema Specification.
An entity model defines the valid set of attributes for an application entity. Each attribute has a defined name, type and a set of properties that may be utilized by your DynamoDB access library such as OneTable or an ORM of your choosing.
From the list of entity models, you can click on an entity to modify, click Add Model
to add a new entity model.
Via the Edit Schema
button you can modify the schema name or version. If the schema is a non-current schema, you can also click Apply to Current
to apply the contents of the displayed schema to the saved Current
schema in the table. This overwrites the previous Current
schema.
Clicking on an entity attribute will display a slide out panel to edit the properties of that attribute.
You can modify the type and any other properties of the type. These properties are defined in the schema and may (or may not) be implemented by your DynamoDB access library. All these are implemented by OneTable.
The properties include:
You can also export your schema in JavaScript, TypeScript or JSON formats via the Export Schema
button from the Models list. You can utilize the exported file directly in your OneTable apps to control your database interactions.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>The DynamoDB database provides an easy to configure, high-performance, NoSQL database with low operational overhead and extreme scalability. It appeals to developers with OLTP applications requiring a simple serverless database or those needing the utmost in scalability.
More recently, best practices have evolved around DynamoDB single-table design patterns where one database table serves the entire application and holds multiple different application entities. This design pattern offers greater performance by reducing the number of requests required to retrieve information and lowers operational overhead. It also greatly simplifies the changing and evolving of your DynamoDB designs by uncoupling the entity key fields and attributes from the physical table structure.
The recent rise of DynamoDB single-table designs is due to the tireless work of educators like Alex DeBrie and Rick Houlihan to help get a better understanding of how to model DynamoDB data in a single-table design and the availability of modeling tools like the SenseDeep DynamoDB Studio.
DynamoDB is a great NoSQL database that comes with a steep learning curve, especially with single-table designs. However, developers find that modeling, defining and expressing DynamoDB single-table designs can be difficult at first.
The OneTable library builds upon recent modeling tools and provides a more natural way to define DynamoDB single-table designs and entity definitions without obscuring any of the power of DynamoDB itself.
So how does the process of using OneTable for single-table designs differ from the traditional approach?
The first key difference is that OneTable uses a schema to define the application entities, keys, attributes and table indexes. Having your application indexes, entities and keys defined in one place is much better than scattering these definitions throughout your application.
For example, here is a schema that defines two entities: Account and User and the primary index and one GSI.
const MySchema = {
indexes: { primary: { hash: 'pk', sort: 'sk' } },
models: {
Account: {
pk: { value: 'account#${name}' },
sk: { value: 'account#' },
name: { type: String, required: true },
},
User: {
pk: { value: 'account#${accountName}' },
sk: { value: 'user#${email}', validate: EmailRegExp },
accountName: { type: String },
email: { type: String, required: true },
},
Post: {
pk: { value: 'post#${id}' },
sk: { value: 'user#${email}', validate: EmailRegExp },
id: { type: String },
message: { type: String },
email: { type: String },
}
},
version: '0.1.0',
format: 'onetable:1.0.0'
}
Single-table designs overload the partition and sort key values by using key prefix labels. In this way, multiple entities can be safely stored and reliability retrieved via a single-table.
OneTable centralizes key management for your queries and database operations. The entity partition and sort keys in OneTable can be ordinary attributes. However, it better to uncouple your keys by defining them as template strings that are calculated at run-time based on the values of other attributes. OneTable uses the value
property to specify the template which operates just like a JavaScript string template.
These OneTable techniques effectively uncouple the logical entity keys from the physical table key names and make changing and evolving your single-table design much easier.
The single-table design process is iterative and consists of the following steps:
Determine the application entities and relationships.
Determine all the access patterns.
Determine the key structure and entity key values.
Codify the design as a OneTable schema.
Create OneTable migration scripts to populate the table with test data.
Prototype queries to satisfy each of the access patterns.
Repeat and refine.
Good key design will support item collections where a single request can be used to retrieve multiple related items and thus in practice implement a “join” between different entities. Consequently, work hard to create item collections wherever possible and avoid “joining” entities in your application code after multiple requests to retrieve the data.
Before even thinking of writing code or creating a database table, ensure you have determined all your application data entities and their relationships. You should document this as an Entity Relationship Diagram (ERD).
For example: an ERD for a trivial blog application with entities for Accounts, Users and Blog Posts.
Your ERD should identify all essential entities and their constituent attributes and relationships.
You need to classify the relationships between your entities as being either one-to-one, one-to-many or many-to-many. In the example above, one account can have many users who can have many blog posts.
Next, enumerate and document all the access patterns to retrieve or manipulate your data. The access patterns should describe the query to implement and the entity attributes required to retrieve the item.
Ensure you consider access for user interfaces, APIs, CRUD for all entities and don’t forget required maintenance operations.
Your access pattern list should describe the entities and attributes queried and the required key fields.
For Example:
Access Pattern | Query | Entities Retrieved |
---|---|---|
Get account | Get account where “name” == NAME | Account |
Get user by email | Get user where “email” == EMAIL | User |
Find users for account | Find users where “accountName” == ACCOUNT_NAME | Users, Account |
Find posts for a user | Find posts where “email” == EMAIL | Posts, User |
… | … |
It can sometimes be helpful to split your access patterns into real-time and batch access groupings. The batch group may need to utilize the DynamoDB scan() method.
Once you have defined all your access patterns, you can design your primary and secondary keys. This is inherently an iterative process as there are often several viable options when selecting an entity’s keys.
The goal is to identify the required indexes and create a key structure that will satisfy all the identified access patterns via efficient queries using as few indexes as possible.
This is achieved by overloading key contents and careful selection of key prefix labels.
The physical database primary and secondary keys should have generic names like pk
and sk
for partition and sort key. For secondary indexes, they should have equally generic names like gs1pk
and gs1sk
. These physical keys are “overloaded” by multiple entities that use the same physical key attributes for multiple access patterns. This is achieved by using unique prefix labels for key values to differentiate the items. In this way, a single index can be used to query multiple entities in different ways.
All database items will have a logical primary partition (hash) key with an optional sort key. Some item entities may also have one or more secondary keys to support additional access patterns.
It can be tempting to set your primary key to be a simple unique entity field. For example: you may initially select your User partition key to be the user’s email address. While this may be a solid choice in terms of distribution of key values, there may be better choices that are both unique and facilitate retrieval of item collections where you can fetch multiple related items with a single request.
For example, consider the “Find users for account” access pattern. It would be useful to retrieve the users and the account at the same time. If the User primary key is simply the user’s email address, then you will have to perform a separate request to get the owning account and then join them in your application code.
However, if the User’s primary partition key is set to the account name and the sort key is set to user’s email address, then a query using a partition key set to the account name and an empty sort key will retrieve an item collection of all the users and the account in one request.
The general strategy for key design is:
Select a primary partition key with a high cardinality that will distribute load over all partitions and avoid hot keys.
Use the same partition key for retrieving related items required for a single access pattern and create a set of sort keys to differentiate between the items.
Select your sort key values to support multiple access patterns by using concatenated sub-fields. This strategy is similar to nested “Russian Dolls” in that you can query different levels via queries using the begins_with
operator. For example: in a shopping cart app, you could specify the leading prefix to get orders by account, user or product.
Orders by account: order#${accountId}
Orders by user: order#${accountId}#${userId}
Orders by product: order#${accountId}#${userId}#${productId}
Use the sort key with a query limit limit to determine how many items in the collection to read (use the sorted or reversed sort order)
Don’t store unique attribute values in the keys themselves. Rather project the values of other attributes via OneTable template strings. This uncouples your keys from the entity attributes and will give you more flexibility to evolve your design in the future.
Handle post-processing and data aggregation needs separately via DynamoDB streams. This may simplify your key structure by handling these use cases separately and may potentially avoid the need for real-time use of the costly DynamoDB scan
operation.
Modeling relationships including one-to-many and many-to-many relationships is the heart of most data models. As a NoSQL database, DynamoDB does not join tables via foreign keys. Instead, you must model your data in such a way so that data is “pre-joined” by design to enable your access patterns.
There are several strategies to implement item relationships.
Adjacency List. The adjacency list pattern is ideal for modeling one-to-many relationships. With this pattern, target items use the same partition key but use different sort keys or partial sort-keys. Retrieving items using only the partition key with an empty or partial sort key will retrieve the entire collection of items or a suitable subset.
Reverse Secondary Index. You can model many-to-many relationships by using the adjacency list strategy (1) and add a secondary index that has the partition key and sort key reversed. In this manner, you can follow the many-to-many relationship in either direction by using either the primary or secondary index.
Denormalization. You can denormalize related items by including them as a complex attribute. The attribute can be either a list or map with the target items. This strategy works well for smaller item sets that are not updated often. OneTable makes this particularly easy by marshaling data to and from JavaScript arrays and objects automatically. Remember DynamoDB enforces a 400KB limit on items.
Duplication. Similar to (1) you can simply duplicate the referenced item. This can work well if the data is not extensively duplicated and if the data is not updated regularly. i.e. works best for read-only constant data.
Simple reference. A last resort is to include the primary key of the target item as an attribute. This then requires a second query to retrieve the target item. With OneTable, you only need to store the logical ID attributes of the item and not the physical key values.
Using these strategies, consider each access pattern and design your keys and then add to your access patterns table.
Access Pattern | Query | Entities | Index | Hash Key | Sort Key |
---|---|---|---|---|---|
Get account | account where “name” = NAME | Account | Primary | account#NAME | |
Get user by email | user where “email” = EMAIL | User | GS1 | user#EMAIL | |
Find users for account | users where “accountName” = ACCOUNT_NAME | Users, Account | Primary | account#NAME | |
Find posts for a user | posts where “email” = EMAIL | Posts, User | GS1 | user#email | begins(post) |
Using the updated access pattern table, we can extract the key structure for each of the entities.
Entity | Hash Key | Sort Key | GSI-1 hash | GSI-1 sort |
---|---|---|---|---|
Account | account#NAME | account# | ||
User | account#NAME | user#EMAIL | user#EMAIL | account#NAME |
Post | post#EMAIL | post#ID | user#EMAIL | post#ID |
Your design can then be coded as a OneTable schema by creating a “model” for each entity. List each of the entity attributes and identify the primary key for each entity.
const MySchema = {
version: '0.1.0',
format: 'onetable:1.0.0',
indexes: {
primary: {
hash: 'pk',
sort: 'sk',
},
gs1: {
hash: 'gs1pk',
sort: 'gs1sk',
}
},
models: {
Account: {
pk: { value: 'account#${name}' },
sk: { value: 'account#' },
name: { type: String },
address: { type: String },
},
User: {
pk: { value: 'account#${accountName}' },
sk: { value: 'user#${email}' },
gs1pk: { value: 'user#${email}' },
gs1sk: { value: 'account#${accountName}' },
accountName: { type: String },
email: { type: String },
},
Post: {
pk: { value: 'post#${email}' },
sk: { value: 'post#${id}' },
gs1pk: { value: 'user#${email}' },
gs1sk: { value: 'post#${id}' },
id: { type: String, lsid: true },
date: { type: Date },
message: { type: String },
email: { type: String },
}
},
}
You are now finally ready to actually create your DynamoDB database. Use Cloud Formation, the Serverless Framework, CDK or equivalent to specify and create your database. Don’t use the console to create production resources.
You should create a single table with a generic primary key and any additional secondary indexes.
The example below depicts a Serverless Framework resource file that creates a database with one GSI with the key names: pk
, sk
, gs1pk
and gs1sk
.
resources:
Resources:
MyDatabase:
Type: AWS::DynamoDB::Table
DeletionPolicy: Retain
Properties:
TableName: BlogDatabase
AttributeDefinitions:
- AttributeName: pk
AttributeType: S
- AttributeName: sk
AttributeType: S
- AttributeName: gs1pk
AttributeType: S
- AttributeName: gs1sk
AttributeType: S
KeySchema:
- AttributeName: pk
KeyType: HASH
- AttributeName: sk
KeyType: RANGE
GlobalSecondaryIndexes:
- IndexName: gs1
KeySchema:
- AttributeName: gs1pk
KeyType: HASH
- AttributeName: gs1sk
KeyType: RANGE
Projection:
ProjectionType: 'ALL'
BillingMode: PAY_PER_REQUEST
Once the physical database is created, the next step is to create some test data so that queries can be prototyped to test the access patterns.
You can use the OneTable CLI to apply your schema and populate your database with test data. The CLI applies discrete changes to your database via “migrations”. These are reversible scripts that can quickly and easily make changes to the structure and data of your database.
Conventional wisdom for DynamoDB has been to be that changing a DynamoDB design is “extremely difficult” and you want to avoid it at all costs. However, with single-table designs that uncouple your logical and physical keys, and with reversible migrations, you can make small and large changes to your live production database without downtime.
The ability to evolve your DynamoDB database may be the most important benefit of single-table designs.
Install the OneTable CLI via:
npm i onetable-cli -g
Make a directory for your migrations in your project and create a migrate.json
with your DynamoDB OneTable configuration.
{
name: 'your-dynamo-table-name',
endpoint: 'http://localhost:8000',
schema: './schema.js',
}
The endpoint
property specifies the local DynamoDB endpoint. To connect to DynamoDb in a real AWS account, read the OneTable CLI article for details.
Generate your first migration:
onetable generate
Migrations are Javascript files that contains up
and down
methods that are invoked to upgrade or downgrade the database. Edit the up
and down
methods to create and remove the test data.
Here is an example migration to create an Account, User and two posts.
export default {
version: '0.0.1',
description: 'Initial migration',
schema: Schema,
async up(db, migrate) {
let account = await db.create('Account', {
name: 'Acme Rockets',
})
let user = await db.create('User', {
email: 'user1@example.com',
accountName: account.name,
})
await db.create('Post', {
email: user.email,
message: 'Post 1',
user: user.email,
})
await db.create('Post', {
email: user.email,
message: 'Post 2',
user: user.email,
})
},
async down(db, migrate) {
let items
do {
// A rare case where scan is justified!
items = await db.scanItems({}, {limit: 100})
for (let item of items) {
await db.deleteItem(item)
}
} while (items.length)
}
}
Apply the migration via the command:
onetable up
This will create the test data according to the defined schema.
After testing, you can at anytime reset the database with new test data via:
onetable reset
Read more about the CLI at OneTable CLI.
When coding your queries to implement and test the access patterns, you can filter the items and attributes returned by using DynamoDB filter and projection expressions.
Filter expressions are applied by DynamoDB after reading the data. They are thus not a substitute for a well designed key structure and query. But filter expressions are useful to select items based on matching non-key attributes.
Projection expressions select the attributes to return after filtering the items. A projection expression can reduce I/O transfer time especially if the item is large.
OneTable makes both filter expressions and projection expression easy to use via the where
and fields
options. For example:
let accounts = await Account.find({}, {
where: '${balance} > {100.00}'
fields: ['id', 'name', 'balance', 'invoices']
})
To fetch an item collection, use the queryItems
API and parse the results. Then use the groupByType
if you want the returned items to be organized into groups. For example:
let items = await table.queryItems({pk: 'account:AcmeCorp'}, {parse: true})
items = db.groupByType(items)
let users = items.Users
let products = items.Products
We have several pre-built working samples that demonstrate OneTable.
This completes the journey to design your single-table DynamoDB database.
As we learn more about single-table design patterns and develop better data modeling tools and libraries, the performance and operational benefits of single-table designs outweigh their initial, apparent complexity. When considering the greatly improved ability to evolve and change your DynamoDB data design, single-table patterns emerge as the preferred option over multi-table designs for most DynamoDB implementations.
At SenseDeep, we’ve used DynamoDB, OneTable and the OneTable CLI extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use single-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent and we’ve been able to extensively evolve our design in production without downtime.
Please try our Serverless developer studio SenseDeep.
Read more about OneTable and the OneTable CLI and the DynamoDB Checklist.
The SenseDeep Migration manager is part of the SenseDeep DynamoDB Suite and provides a controlled way to manage DynamoDB data migrations. With it, you can easily and reliably evolve your DynamoDB data design to upgrade or downgrade your table data without downtime.
Now that DynamoDB single-table design practices are commonplace, the importance of being able to evolve data designs and schemas is becoming increasingly important. Without migrations, it can be difficult to upgrade code and data when your data design and schemas change. Data migrations are a way of separating data upgrades from code deploys so they can be appropriately coordinated without downtime.
Data migrations are sequenced, reversible pieces of code that mutate your data by applying a changed DynamoDB design (schema) to your data. Using migrations expressed as code and committed to your code repository, you can confidently evolve your DynamoDB design knowing that you can easily and quickly reverse a change.
Migrations are named and versioned using Semantic Versioning to indicate and control data compatibility. Migrations thus form a sequence of version numbers through which you can step to apply or reverse migrations.
Each migration contains upgrade
and downgrade
functions that will apply or reverse a migration to your table. After a migration is applied, the data table is said to be at the version of that migration.
For example, here is a simple migration with up and down functions.:
import Schema from './schema-0.1.6'
const migrate = new Migrate(OneTableParams, {
migrations: [
{
version: '0.1.6',
description: 'Create a status record',
schema: Schema,
async up(db, migrate) {
await db.create('Status', {active: true})
},
async down(db, migrate) {
await db.remove('Status', {})
}
}
]
})
Migrations are optionally (and advisably) paired with a specific single-table schema version that describes the data model at that migration version. Schemas are created via the SenseDeep Single Table Designer or via One Table.
The SenseDeep database migration manager displays what migrations have been applied to your data and what migrations are outstanding. It shows what is the migration version for your current data and schema.
You can run outstanding migrations by clicking “up” for each migration. You can downgrade one step by clicking “down”.
The “Repeat” button will run the last migration again. The “Reset” button will run the special “latest” migration which is useful for dev environments where you want to reset the database to a good known state.
After applying a migration, the migration manager will create an item in the table to contain the current schema version. This means your table is self-describing and always contains the appropriate schema to interpret the data.
SenseDeep can orchestrate DynamoDB migrations controlled by the OneTable Migration library. You can use this library even if you are not using OneTable for your apps. Check out the OneTable Controller sample OneTable Controller in GitHub and deploy to host and manage your data migrations. Read OneTable Migrate for more information and setup.
Once you have deployed the OneTable Controller, enter the Lambda Arn for the controller in the Table Edit slide out panel under the monitoring tab.
There are several important design techniques that will make creating discrete migrations easier.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>The SenseDeep provisioning planner guides your selection of billing and provisioning options for DynamoDB.
While there are various DynamoDB calculators, they are “theoretical” and not based on actual data usage. The SenseDeep provisioning planner displays displays the actual costs of your current billing plan based on real, historical and live data. It then compares this usage with the the cost of an alternate plan if you were to switch your billing plan from OnDemand to Provisioned or vice-versa.
The DynamoDB Provisioning page displays your current and projected provisioning, utilization and costs.
The top two graphs display the actual read and write capacity usage by your table. A red-line overlay will show your provisioned or predicted provisioned usage if you are using OnDemand billing.
Below the graphs, the left card will display the current billing configuration, be it OnDemand or Provisioned. The right card will display the alternate billing mode estimate.
The billing costs include read and write capacity I/O units and the current storage requirements.
If there is a significant benefit to changing billing modes, SenseDeep will highlight that in the lower green card.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>SenseDeep DynamoDB metrics is a suite of in-depth performance metrics for DynamoDB. It includes metrics at the account, table and application entity levels.
Developers are adopting DynamoDB single-table design patterns as the preferred design model where all application data is packed into a single-table. However the standard DynamoDB metrics are not able to gain insight into these new levels and cannot offer per-entity metrics.
SenseDeep metrics instrument your application to expose performance metrics at a granular level, down to each application entity or API invocation.
SenseDeep provides standard AWS DynamoDB metrics, enhanced single-table metrics and account level metrics.
The DynamoDB standard metrics display overview metrics and operation details for a whole DynamoDB table. These metrics come from the standard AWS CloudWatch DynamoDB metrics.
In a single page, all the important standard metrics are aggregated and presented. These include:
Below the graphs are metric cards that include the average and maximum values. Also displayed is the DynamoDB cost of service over the period. This includes the raw DynamoDB I/O charge and does not include transfer, storage charges or consider reserved pricing discounts.
Below the info cards is the Operations table which includes per-operation metrics.
If you need access to a specific metric that is not provided by the dashboard. You can easily create a custom widget and dashboard for that metric from the SenseDeep dashboard.
DynamoDB Single-table designs are more complex and require per entity/model performance monitoring and metrics. Traditional monitoring covers table level and operation level metrics only. What is missing is the ability to see single-table entities and their performance and load on the database.
If you’ve wondered:
SenseDeep single-table metrics can answer these and other questions.
SenseDeep supports single-table metrics for apps using the DynamoDB OneTable library or for any JavaScript app that utilizes DynamoDB Metrics. These libraries generate a configurable set of enhanced CloudWatch metrics that track DynamoDB performance.
For users of OneTable, the metrics can be enabled by setting dbmetrics
to true in the Table constructor. For any other JavaScript application, install the DynamoDB Metrics package and follow the package’s installation instructions.
Both OneTable and DynamoDB Metrics are configurable at run-time which means you can tailor the tracked application dimensions and enable or disable metrics as you wish. With both packages, there is negligible computational overhead. Both incur CloudWatch metric charges depending on the number of dimensions tracked. See the package README’s for details.
SenseDeep tracks a set of performance metrics across a configurable set of dimensions. The supported dimensions are: Table, Tenant, Source, Index, Model and Operation.
The Tenant dimension is used for multi-tenant applications to track usage by tenant. The Source dimension tracks performance by code module, function or source. The Index dimension corresponds to the primary or global secondary index used for queries. The Model dimension is the single-table entity name and the Operation is the DynamoDB API operation.
You can customize which dimensions you wish to track via the “Edit” button for the table and select the Monitoring tab.
For each dimension combination, the following metrics are tracked:
On the Single-Table page, you can drill-down by clicking on any row in the table at the bottom of the page. Initially, the table displays the source (code/function) of the metrics. This enables you to see the load caused individually by your applications and functions.
By clicking on a row, you can display metrics for the table primary and secondary indexes. Click again to display the schema entities and so on. As you drill-down, the graphs and metrics at the top of the page will display information for your selected choice.
Go back up the chain by clicking on the cookies to the right of the table filter box.
You can also generate metrics for specially profiled DynamoDB APIs (queries and scans). For both OneTable and DynamoDB metrics, you can emit per-API metrics to profile a specific operation in your app.
Once you have added the profile parameter your code, you need to add the profiled dimension to the dimensions list via the “Edit” button for the table and select the Monitoring tab.
SenseDeep also tracks a account level limits and metrics. These include the total provisioned read and write utilization and the account level read and write maximums.
For any metric page, you can select the desired time range for your metrics. You can select last hour, day, week or month or any custom period.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
Previously, single-table design with DynamoDB was a black box and it was difficult to peer inside and see how the components of your apps are operating and interacting. Now, SenseDeep can understand your data schema and can transform raw DynamoDB data to highlight your application entities and relationships and transform your effectiveness with DynamoDB.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>SenseDeep manages the DynamoDB tables for your enabled clouds. These tables can be in multiple AWS accounts or regions. You can create and delete tables and indexes for your tables.
SenseDeep will automatically discover your tables and will dynamically update this list as new tables are created or destroyed. The table list includes the table size, number of items, billing scheme and provisioned capacity.
The SenseDeep table management is not meant to replace appropriate “infrastructure-as-code” deployment of tables to production. Rather, it intends to provide a quick and easy way to create and manage tables and indexes while developing your DynamoDB applications.
Gaining insight into single-table design patterns is the new frontier for DynamoDB and the SenseDeep DynamoDB Studio is the start of a new wave of tools to elevate and transform DynamoDB development.
SenseDeep includes a table manager, data item browser, single-table designer, provisioning planner, database migration manager and in-depth table metrics — all of which are single-table aware.
Try the SenseDeep DynamoDB studio with a free developer license at SenseDeep App.
You may also like to read:
]]>DynamoDB OneTable (OneTable) is an access library for DynamoDB applications that use single-table design patterns with NodeJS. OneTable makes dealing with DynamoDB and single-table design patterns dramatically easier while still providing easy access to the full DynamoDB API.
OneTable is provided open source (MIT license) from GitHub OneTable or NPM OneTable.
OneTable is supported by the SenseDeep DynamoDB Studio which provides single-table aware data browser, table designer, provisioning planner and detailed metrics.
After watching the famous Rick Houlihan DynamoDB ReInvent Video, we changed how we used DynamoDB for our SenseDeep serverless developer studio to use single-table design patterns. However, we found the going tough and thus this library was created to make our single-table patterns less tedious, more natural and a joy with DynamoDB.
A big thank you to Alex DeBrie and his excellent DynamoDB Book. Highly recommended.
OneTable provides a convenience API over the DynamoDB APIs. It offers a flexible high-level API that supports single-table design patterns and eases the tedium of working with the standard, unadorned DynamoDB API. OneTable can invoke DynamoDB APIs or it can be used as a generator to create DynamoDB API parameters that you can save or execute yourself.
OneTable is not opinionated (as much as possible) and provides hooks for you to customize requests and responses to suit your exact needs.
Here are some of the key features of OneTable
npm i dynamodb-onetable
Import the OneTable library. If you are not using ES modules or Typescript, use require
to import the libraries.
import {Table} from 'dynamodb-onetable'
If you are using the AWS SDK V2, import the AWS DynamoDB
class and create a DocumentClient
instance.
import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)
This version includes prototype support for the AWS SDK v3.
If you are using the AWS SDK v3, import the AWS v3 DynamoDBClient
class and the OneTable Dynamo
helper. Then create a DynamoDBClient
instance and Dynamo wrapper instance.
import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import Dynamo from 'dynamodb-onetable/Dynamo'
const client = new Dynamo({client: new DynamoDBClient(params)})
Initialize your your OneTable Table
instance and define your models via a schema. The schema defines your single-table entities, attributes and indexes.
const table = new Table({
client: client,
name: 'MyTable',
schema: MySchema,
})
This will initialize your your OneTable Table instance and define your models via a schema.
Schemas define your models
(entities), keys, indexes and attributes. Schemas look like this:
const MySchema = {
version: '0.1.0',
format: 'onetable:1.0.0',
indexes: {
primary: { hash: 'pk', sort: 'sk' }
gs1: { hash: 'gs1pk', sort: 'gs1sk' }
},
models: {
Account: {
pk: { value: 'account:${name}' },
sk: { value: 'account:' },
id: { type: String, uuid: true, validate: /^[0-9A-F]{32}$/i, },
name: { type: String, required: true, }
status: { type: String, default: 'active' },
zip: { type: String },
},
User: {
pk: { value: 'account:${accountName}' },
sk: { value: 'user:${email}', validate: EmailRegExp },
id: { type: String },
accountName: { type: String },
email: { type: String, required: true },
firstName: { type: String, required: true },
lastName: { type: String, required: true },
username: { type: String, required: true },
role: { type: String, enum: ['user', 'admin'],
required: true, default: 'user' }
balance: { type: Number, default: 0 },
gs1pk: { value: 'user-email:${email}' },
gs1sk: { value: 'user:' },
}
},
params: {
isoDates: true,
timestamps: true,
}
}
Schemas define your models
and their attributes. Keys (pk, gs1pk) can derive their values from other attributes via templating.
Alternatively, you can define models one by one:
const Card = new Model(table, {
name: 'Card',
fields: {
pk: { value: 'card:${number}'}
number: { type: String },
...
}
})
To create an item:
let account = await Account.create({
id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
name: 'Acme Airplanes'
})
This will write the following to DynamoDB:
{
pk: 'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
sk: 'account:98034',
id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
name: 'Acme Airplanes',
status: 'active',
zip: 98034,
created: 1610347305510,
updated: 1610347305510,
}
Get an item:
let account = await Account.get({
id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
zip: 98034,
})
which will return:
{
id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
name: 'Acme Airplanes',
status: 'active',
zip: 98034,
}
To use a secondary index:
let user = await User.get({email: 'user@example.com'}, {index: 'gs1'})
To find a set of items:
let users = await User.find({accountId: account.id})
let adminUsers = await User.find({accountId: account.id, role: 'admin'})
let adminUsers = await User.find({accountId: account.id}, {
where: '${balance} > {100.00}'
})
To update an item:
await User.update({id: userId, balance: 50})
await User.update({id: userId}, {add: {balance: 10.00}})
To do a transactional update:
let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)
OneTable fully supports TypeScript apps and OneTable APIs are fully type checked.
However, OneTable goes further creates type declarations for your table entities and attributes. TypeScript will catch any invalid schema, entity or entity attribute references.
Using TypeScript dynamic typing, OneTable automatically converts your OneTable schema into fully typed generic Model APIs.
For example:
const schema = {
version: '0.1.0',
format: 'onetable:1.0.0',
models: {
Account: {
pk: { type: String, value: 'account:${name}' },
name: { type: String },
}
}
}
// Fully typed Account object based on the schema
type AccountType = Entity<typeof schema.models.Account>
let account: AccountType = {
name: 'Coyote', // OK
unknown: 42, // Error
}
// Create a model to get/find/update...
let Account = new Model<AccountType>(table, 'Account')
let account = await Account.update({
name: 'Acme', // OK
unknown: 42, // Error
})
account.name = 'Coyote' // OK
account.unknown = 42 // Error
DynamoDB is a great NoSQL database that comes with a steep learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.
The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type expressions, key, filter, condition and update expressions are anyone’s idea of a good time.
Net/Net: it is not easy to write terse, clear, robust Dynamo code for single-table patterns.
Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, “JavaScripty / TypeScripty” way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.
You can read more in the detailed documentation at:
We also have several pre-built working samples that demonstrate OneTable.
At SenseDeep, we’ve used the OneTable module extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use single-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.
SenseDeep includes a OneTable GUI with single-table designer, data item browser, migration manager, provisioning planner and table manager.