September 30 Release Notes
This release adds centralized server log capture and storage.
System log files record critical security information. When securely captured in real-time, they can record the footprints of attackers and be a vital aid in determining what happend during an attack.
This major release brings to SenseDeep a comprensive and integrated log capture and storage.
Existing Logging Solutions
Existing log capture solutions are often big (bloated) and consume a lot of system resources on the server instance. Many leading solutions require more than 50MB RAM to run the log capture agent, use more than 100MB on disk and consume more than 20% of the system CPU when ingesting log data. A common reason these logging solutions are so big, is that they are often written in interpretive languages like Python. Using Python may make it easier and quicker to develop the logging agent, but it is unreasonable to devote such a high percentage of system resources just for log capture. Some core parts of the system need to be written in a native language for efficiency.
SenseDeep Log Capture
Our goal was to implement real-time log capture into the existing SenseDeep security agent and keep total memory usage below 5MB and CPU usage less than 1%.
Modern Linux versions support a unified logging mechanism based on the SystemD Journal. This provides a high performance, zero-copy API to ingest log data. SenseDeep utilizes this API and an event based architecture to import log data without polling. The result is an extremely efficient log capture agent that uses minimal system resources. It also performs real-time intrustion detection with automated defenses.
- High performance capture of SystemD Journal log data
- Capture instance log files
- Capture Docker container logs
- Capture JSON log files
- Extensible to capture custom logs
- Automatically save log data to AWS Cloud Watch Logs
- Manage and control log data expiry
- View log data in the SenseDeep app
- Search log data
- Live-tail viewer of log data
- Again, many small fixes