Observability in Production without the Cost

If you want improved Observability, but the cost of logging in production is prohibitive, then SenseDeep dynamic logging may be your solution.

SenseDeep supports on-demand Dynamic Log Control to scale up or down the volume and focus of your serverless logging.

You can use this to include a base level of log telemetry in your application and then increase or focus logging when the need arises. This improves the Observability of your serverless functions without having a heavy constant price due to excessive logging.

The entire concept of Dynamic Log Control is best explained in this post Dynamic Log Control and you may want to read that first.

Log Control

To control log output, SenseDeep manages three environment variables:

• LOG_FILTER
• LOG_OVERRIDE
• LOG_SAMPLE

These variables control which log levels or channels are enabled for output. Together, they control the base logging level, a time-limited override and background sampling of logs for a percentage of requests.

Log library support

SenseDeep log control supports a wide selection of platforms and log libraries including: Node, Python and Java, and the Node log libraries Bunyan, Debug, Pino, SenseLogs or Winston.

See Dynamic Log Control for Log Libraries for code samples for each platform and library.

LOG_FILTER

The LOG_FILTER defines the set of enabled log levels or channels. Enabled levels will emit output. Disabled levels will be silent. For example:

LOG_FILTER=error,warn

This will enable logging output in your Lambdas for log.error() and log.warn() but disable output for calls to trace or debug.

The LOG_OVERRIDE will replace the LOG_FILTER for a given period of time before reverting to the LOG_FILTER settings. LOG_OVERRIDE is prefixed with an expiry time expressed as a Unix epoch in milliseconds since Jan 1 1970.

For example:

LOG_OVERRIDE=1629806612164:debug,trace

The LOG_SAMPLE is a sampling filter that applies to a given percentage of requests. The list of levels is is prefixed with a percentage.

For example, this will trace 5% of requests.

LOG_SAMPLE=5%:trace

SenseDeep Lambda Configuration

SenseDeep provides a convenient one-stop interface to control your logging levels via the LOG* parameters.

When you view a Lambda function, click on Edit to display the Lambda Logging slide-out panel. From here, you can define the LOG_FILTER which specifies the log level or channels to enable. If your log library uses numeric levels, enter the level here. If your log library uses alphabetical levels such as info or error enter those. Some libraries such as SenseLogs support a comma separate set of channels.

Log Overrides

If you need to diagnose an issue or debug a Lambda function, enter increased log levels in the LOG_OVERRIDE. Enter a time period after which the override will be lifted.

Log Sampling

It is often desirable to fully log a small percentage of requests so that you always have a complete request trace. To do this, enter a LOG_SAMPLE and the percentage rate that you wish to use increased sample logging.

Scaling Metrics

If you use the SenseLogs logging library, you can emit CloudWatch custom metrics via log commands.

SenseLogs emits AWS CloudWatch custom metrics via the Embedded Metric Format (EMF). For example:

log.metrics('metrics', 'Acme/Rockets', {Launches: 1})

Using the SenseDeep log control, you can enable or disable these metrics by enabling or disabling the metrics log level.

Once emitted, you can view and track your metrics using the SenseLogs dashboard and create Alarms and Alerts.

Summary

SenseDeep makes it easy to instantly scale up or down your logging as your needs vary. You can keep a good base level of logging and metrics without breaking the bank, and then increase logging on-demand.

More?

The logging library code samples are available at Log Control Samples and SenseLogs is available from NPM SenseLogs.

Links

• Dynamic Log Control
• SenseDeep Quick Tour
• SenseDeep App
• SenseLogs