Introduction

Welcome to the Iceburst.io Documentation! Iceburst is an open-source real-time data lake for monitoring and security.

Why Iceburst for monitoring and security?

Our approach is simple: ingest all the telemetry and security data in an S3 bucket as Parquet files in Iceberg table format and query them using DuckDB with milliseconds retrieval and zero egress cost. Parquet is converted to Arrow format in-memory for efficient query processing.

S3 as a storage layer architecture enables us to offer so many advantages out-of-the-box like separation of storage and compute, time travel, fault tolerance, infinite concurrency reads, fast recovery, and a better developer experience.

Ease of use

• Simple to adopt

  There is no need to install and maintain a Kubernetes or Kafka cluster to start using Iceburst. We have a simple S3 Exporter for OpenTelemetry to kickstart your setup.

• Simple to scale

  When it comes time to scale up to terabytes per day, simply switch to Iceburst Exporter for OpenTelemetry and we will do all the heavy lifting of setting up Kafka for data streaming and Flink jobs for stream processing.

• Improved efficiency

  Better query performance and new query functionality drastically improve the overall efficiency. In addition, the cost savings that is realized because of S3 as the data store.

Cost efficiency

• Simple to adopt

  There is no need to install and maintain a Kubernetes or Kafka cluster to start using Iceburst. We have a simple S3 Exporter for OpenTelemetry to kickstart your setup.

• Simple to scale

  When it comes time to scale up to terabytes per day, simply switch to Iceburst Exporter for OpenTelemetry and we will do all the heavy lifting of setting up Kafka for data streaming and Flink jobs for stream processing.

• Improved efficiency

  Better query performance and new query functionality drastically improve the overall efficiency. In addition, the cost savings that is realized because of S3 as the data store.