New Logging Engine for SenseDeep

As promised, the latest SenseDeep update brings an all new logging engine that offers fast, predictable, low-latency access to any log data regardless of the date.

SenseDeep has always prioritized fast access to log data, but utilizing CloudWatch logs as a data store made this difficult at times, especially, when trying to access data at specific date ranges.

The new logging engine changes the logging strategy used by SenseDeep to now capture log data and store it in an open DynamoDB table in your account.

The result is fast, predictable, secure access to your log and Lambda data in an open architecture solution.

SenseDeep Watcher and Log Capture

SenseDeep captures log data by automatically subscribing to required CloudWatch log groups. To do this, SenseDeep runs a small Lambda function called the Watcher in your account. The Watcher receives log data and stores it in a DynamoDB database together with your Alarm and Alert configuration.

The Watcher automatically subscribes (and unsubscribes) to logs as required and efficiently captures log events and saves them in a DynamoDB table.

The Watcher and the DynamoDB table run inside your AWS account in your designated regions. This offers the highest performance and eliminates latencies when accessing your recent log data. It is also more secure as your log data never leaves your account.

Why DynamoDB for Logging

Traditionally DynamoDB has not been used for storing log data. However, for the SenseDeep use case, DynamoDB is ideal. There are several characteristics of DynamoDB that particularly suit the storage of shorter-term log data.

• Secure storage of log data in your account as log events never leave your AWS account.

• Efficient and fast indexing and retrieval of log items by any date range.

• Automatic removal of older log items when they expire.

• Ability to provide open access to log data for custom solutions.

We use a custom self-scaling to provision that DynamoDB table so that the cost is tailored to the required log ingestion rate.

Secure Log Storage

Most 3rd party logging solutions ship your log data offsite to their proprietary log storage mechanisms. However, log data often contains sensitive information and identifiers. Protecting your log data is critical for the security of your account and you are relying on these vendors to never expose or have accidents with your log data.

SenseDeep is different. Your log data never leaves your account and is stored securely in a database in your account over which you have full control. SenseDeep captures your log data and stores it in a DynamoDB table in your account. When you use the SenseDeep app, your browser directly accesses the log data in your account.

Automatic Pruning

Log data never stops and so older log entries must eventually be pruned from the database. This can be a costly maintenance task and DynamoDB has a great solution.

DynamoDB supports automatic removal of table items via a Time-to-live (TTL) attribute. When SenseDeep adds log items to the table, it defines an expire TTL date attribute for when the log item can be removed. This is configured according to the SenseDeep log lifespan setting. When the expire date is reached, DynamoDB will automatically remove the log item. No table maintenance required!

Open Access

While SenseDeep provides log capture, a fast log viewer and alarms based on your log data, there are many possible needs that cannot be foreseen. So SenseDeep publishes the log data schema so you can extend upon SenseDeep and create your own log analysis capabilities.

The schema for the DynamoDB SenseDeep table is defined at:

• https://github.com/sensedeep/sensedeep/blob/master/schema/index.js

This table schema is defined using OneTable which makes it easy to use single-table DynamoDB designs. You can use OneTable in your code with this schema to freely access and utilize your log data in your account.

Scalable

The SenseDeep logging architecture is 100% serverless. As your log volume increases, AWS Lambda will scale the Watcher as required to capture your log data. As your logging load decreases, the Lambda service adjusts and scales down the Watcher concurrency.

This enables SenseDeep to offer pricing plans that have no log ingestion limits. You are not capped on the volume of logs or log data that is captured. As your log load increases, the Watcher Lambda and SenseDeep DynamoDB table will scale steadily and predictably with a very low cost.

Lowest Latency

When developing serverless apps, developers need immediate results when testing, debugging and maintaining their serverless services.

SenseDeep is offers the lowest latency in detecting and presenting Lambda invocations and changes.

Whereas most other monitoring services delay minutes to display Lambda results, the new SenseDeep logging engine is unique in its ability to display results immediately. You will see a typical < 5 second delay in displaying Lambda results. This directly translates into increased developer productivity.

SenseDeep is unique in how it accesses Lambda metrics and invocation logs. SenseDeep is the only service to access your AWS Lambda and CloudWatch logs directly in AWS without any intermediate processing or double storage or buffering. This provides the lowest possible latency when presenting critical Lambda results for your viewing.

Long Term Log Storage?

The SenseDeep developer studio aims to provide serverless developers with fast access to critical information to debug and maintain serverless apps. So it does not attempt to provide a long-term log archival solution that can store log data forever for audit purposes. For that, you can use CloudWatch which uses low cost S3 to save logs.

Instead, SenseDeep captures log data and retains recent log data that is relevant to troubleshooting serverless apps. You can configure the time period for which you wish to retain log data in the SenseDeep database and by default, this is set to 7 days but you can set on a per-log basis to any time period.

Upgrading

When you next run SenseDeep you may need to re-connect your clouds top upgrade the SenseDeep IAM roles for use by the Watcher and DynamoDB log database.

Summary

SenseDeep is still the fastest and easiest serverless troubleshooting service. SenseDeep displays Lambda results in real-time with full metrics, analysis and application log data in context. It offers alarms on application events and log data with alerts and notifications for 24x7 monitoring.

Links

• SenseDeep App