Description:

Friday / Systems Group Talk
October 3rd / 3:30 PM
Hybrid / Mudd 3514

Speaker
Franco Solleza, Brown University

Talk Title
Loom: Efficient Capture and Querying of High-Frequency Telemetry

Abstract
To debug performance issues, engineers often rely on high-frequency telemetry (HFT) from sources like perf, DTrace, or eBPF, which can generate millions of records per second. Current database systems are too slow to capture such high-rate data in its entirety, and the de facto standard approach of writing HFT to raw files makes queries slow and cumbersome. Engineers must therefore either work with incomplete data, which risks missing critical events, or accept slow queries.

Loom is a new system specialized for capturing and analyzing HFT with timely, interactive queries. Key to Loom’s design is that it combines the high ingest capability of log-based storage with lightweight, sparse, and domain-specific indexes that accelerate queries. This design strikes a balance: it prioritizes capturing complete data at high rate while indexing just enough to support interactive queries on HFT.

Experiments show that Loom supports both higher ingest throughput and lower query latency than best-in-class systems for ingest-optimized storage (FishStore) and time series databases (InfluxDB), all while consuming substantially fewer host resources and ensuring data completeness.

Biography
Franco Solleza is a final year PhD student at Brown University working with Malte Schwarzkopf. His current research focuses on making it easier for users to understand their complex system deployments. He is also investigating how to make it easier for non-experts to safely extend an OS kernel using domain-specific languages.