Open Architecture Observability

All enterprise logging solutions ship your log data into their proprietary log storage mechanisms. And thereafter, your log data is accessible only via their interfaces. That is great if the logging solution does 100% of what you need today and tomorrow. But if you need to get insights that their interface does not permit – you are stuck.

SenseDeep is different. Your log data never leaves your account and is stored in an open architecture database in your account, over which you have full control.

You can read, transform and analyze your log data in any way you want. You are not limited by SenseDeep in what you can do with your serverless and log data.

SenseDeep Watcher and Log Capture

SenseDeep captures serverless and log data via a small Lambda function called the SenseDeepWatcher which receives log data and stores it in a DynamoDB database.

The Watcher and the DynamoDB table run inside your AWS account. This offers the highest level of security as your log data never leaves your account. Log performance is enhanced and latencies are greatly reduced because log data does not have far to travel.

Open Access

While the SenseDeep UI provides a fast, powerful and flexible log viewer, there are many possible needs that cannot be foreseen. So SenseDeep provides open access to your log database and publishes the log data schema so you create your own log analysis capabilities. This is an open, transparent architecture for your logging needs.

SenseDeep uses best-practices via a DynamoDB single-table design and uses the OneTable access library for easy access and manipulation of log data.

The schema for the DynamoDB SenseDeep table is defined at:

• https://github.com/sensedeep/sensedeep/blob/master/schema/index.js

Example

Here is a quick sample that demonstrates access to your log data.

import DynamoDB from "aws-sdk/clients/dynamodb.js"
import { Table } from "dynamodb-onetable"
import Schema from "./schema.js"

const client = new DynamoDB.DocumentClient({params})

const table = new Table({
    client,
    delimiter: ":",
    hidden: false,
    name: "SenseDeep",
    schema: Schema
})

const Log = table.getModel("Log")
const Event = table.getModel("Event")

async function main() {

    let logs = await Log.find()
    console.log("Logs", JSON.stringify(logs, null, 4))

    // Retrieve log events for /aws/lambda/HelloWorld between Jun 10 and July 10 2021

    let events = await Event.find({
        pk: "/aws/lambda/HelloWorld",
        sk: {between: [
            "2021-06-10T00:00:00.000Z",
            "2021-07-10T00:00:00.000Z",
        ]},
    }, {limit: 100})
    console.log("Events", JSON.stringify(events, null, 4))

    // Get the most recent 10 events, most recent first

    events = await Event.find({
        pk: "/aws/lambda/HelloWorld",
        sk: {between: [
            "2021",
            new Date().toISOString(),
        ]},
    }, {limit: 10, reverse: true})
    console.log("Most recent Events", JSON.stringify(events, null, 4))

    //  Get log events that match a pattern
    //  Get recent events that match the pattern "ERROR"

    events = await Event.find({
        pk: "/aws/lambda/HelloWorld",
        sk: {between: [ "2021", new Date().toISOString() ]},
    }, {
        maxPages: 100,
        limit: 100,
        where: `contains(\${message}, {"ERROR"})`,
        reverse: true,
    })
}
main()

Scalable

The SenseDeep logging architecture is 100% serverless. As your log volume increases, AWS will scale the Watcher and DynamoDB table as required to cache recent log data. As your logging load decreases, the Lambda/DynamoDB services automatically adjust and scale down as required.

This enables SenseDeep to offer pricing plans that do not have log ingestion limits. You are not capped on the volume of logs or on the amount of log data captured. As your log load increases, the Watcher Lambda and SenseDeep DynamoDB table will scale predictably with a very low cost basis.