Encrypting Data at Rest for Kafka Workloads with Kroxylicious

calendar Nov 8, 2026 · 7 min read · Kafka Kroxylicious Encryption

In enterprise event streaming, securing data at rest is a non-negotiable compliance requirement. Whether adhering to PCI-DSS, HIPAA, or strict financial industry mandates, organizations must guarantee that sensitive data stored within Apache Kafka cannot be accessed by unauthorized parties.

However, standard storage-level encryption leaves significant blind spots, and client-side encryption is notoriously hard to scale across polyglot microservice architectures.

This is where Kroxylicious enters the picture. In this post, we will explore how Kroxylicious leverages protocol-aware proxying to provide transparent, envelope-based encryption at rest for Kafka workloads.

The Illusion of Storage-Level Encryption

Most organizations today check the “encryption-at-rest” box by enabling disk or volume encryption—such as AWS EBS KMS encryption, Google Cloud Persistent Disk encryption, or Linux LUKS / dm-crypt.

While volume encryption protects against someone physically stealing a hard drive from a datacenter, it does not protect data from active operating system or application-level threats:

  1. Privileged Access Leakage: Any operator, engineer, or compromised credential with SSH, root access, or kubectl exec permissions into a Kafka broker pod can read the partition log segment files (.log) in plaintext directly from the filesystem.
  2. Untrusted Infrastructure: If you are running Kafka in a shared multi-tenant cluster or on managed cloud infrastructure, your cloud provider and infrastructure administrators have implicit access to your raw streaming payloads.
  3. No Multi-Tenant Key Isolation: Volume-level encryption uses a single disk key. If that key is compromised, every topic and tenant residing on that broker is exposed.
  4. Tiered Storage Risks: When Kafka offloads older log segments to object storage (such as AWS S3 or Google Cloud Storage) via Tiered Storage, the security guarantees depend entirely on bucket-level permissions rather than intrinsic payload protection.

Why Client-Side Encryption Is Hard to Scale

The traditional alternative is application-level client-side encryption: producers encrypt messages before sending them to Kafka, and consumers decrypt them upon receipt.

While architecturally sound, client-side encryption introduces massive operational friction:

  • Polyglot Hell: You must implement, audit, and maintain cryptographic libraries across every language your teams use (Java, Python, Go, Node.js, C#).
  • Key Management Sprawl: Every producing and consuming microservice must be configured with direct KMS credentials, access policies, and caching logic to prevent hitting KMS rate limits.
  • Complex Key Rotation: Rotating keys or upgrading cryptographic algorithms requires coordinated deployments across dozens or hundreds of independent microservice repositories.

What is Kroxylicious?

Kroxylicious is an open-source, API-aware proxy purpose-built for Apache Kafka.

Unlike generic L4/L7 proxies (such as Envoy, HAProxy, or NGINX) that treat Kafka traffic as opaque TCP streams, Kroxylicious natively parses and understands the Kafka wire protocol—including requests like ProduceRequest, FetchResponse, and MetadataRequest.

Architecture Flow: Transparent Envelope Encryption with Kroxylicious

Architecture Flow: Transparent Envelope Encryption for Apache Kafka with Kroxylicious and KMS

Operating as a transparent intermediary between your Kafka clients and the Kafka brokers, Kroxylicious uses a pluggable filter pipeline to inspect, manipulate, or enrich Kafka protocol frames on the fly.

Its flagship capability is the Record Encryption Filter.

How Kroxylicious Encrypts Data at Rest

Kroxylicious implements Envelope Encryption directly at the Kafka protocol layer:

1. Key Hierarchy: KEKs and DEKs

  • Key Encryption Key (KEK): A long-lived master key managed inside an external Key Management Service (such as HashiCorp Vault, AWS KMS, Azure Key Vault, or GCP KMS).
  • Data Encryption Key (DEK): A short-lived, cryptographically random key generated to encrypt a specific batch of records using symmetric encryption (e.g., AES-GCM-256).

2. The Produce Path (Transparent Encryption)

  1. A producer sends a standard Kafka ProduceRequest containing plaintext records to Kroxylicious.
  2. Kroxylicious intercepts the request and evaluates its configured encryption policies (e.g., “encrypt topic customer-transactions using Vault KEK finance-key”).
  3. The Record Encryption Filter requests a new DEK from the KMS (or reuses a cached, wrapped DEK) and encrypts the record batch payload.
  4. The encrypted payload, along with the encrypted (wrapped) DEK metadata in the record headers, is assembled into a standard Kafka protocol frame.
  5. The request is forwarded to the real Kafka broker, which persists the ciphertext directly to disk.

3. The Storage State

On the Kafka brokers, the data stored on disk is pure ciphertext. Even if an intruder dumps the underlying .log segment files or gains root access to the broker filesystem, they obtain nothing more than unintelligible encrypted bytes.

4. The Consume Path (Transparent Decryption)

  1. An authorized consumer issues a standard FetchRequest through Kroxylicious.
  2. The broker returns the encrypted records to Kroxylicious.
  3. The Record Encryption Filter extracts the wrapped DEK from the record headers, sends it to the KMS to unwrap using the authorized KEK, and decrypts the records.
  4. Kroxylicious streams the original plaintext records to the consumer.

Major Advantages for Enterprise Workloads

1. Zero Client Changes

Because Kroxylicious implements standard Kafka protocol proxying, your application developers do not change a single line of code. They continue using standard kafka-clients, Spring Boot, librdkafka, or Python Kafka drivers. The bootstrap.servers simply points to the Kroxylicious proxy instead of the brokers directly.

2. Per-Topic and Per-Tenant Key Isolation

Kroxylicious allows you to map different topics to different KEKs in your KMS. In multi-tenant environments, Tenant A’s topics can be encrypted with Tenant A’s AWS KMS key, while Tenant B’s topics use Tenant B’s key.

3. Cryptographic Erasure (“Crypto-Shredding”) for GDPR

Under regulations like GDPR’s “Right to be Forgotten”, deleting specific data from immutable, append-only Kafka logs can be complex and expensive. With Kroxylicious, if you assign a specific KEK to a tenant or dataset, deleting the KEK from the KMS renders all historical records instantly and irreversibly unreadable—across live brokers, backups, and tiered storage alike.

4. Secure Tiered Storage and MirrorMaker 2 Replication

  • Tiered Storage: When Kafka offloads historical log segments to AWS S3, Google Cloud Storage, or Azure Blob Storage via Tiered Storage, records remain safely encrypted with their respective DEKs. Cloud storage misconfigurations or bucket leaks will not expose raw event payloads.
  • Replication Without Decryption: Cross-cluster tools like MirrorMaker 2 or cluster replication engines can replicate raw ciphertext between regions or data centers without having access to the KMS keys, preventing data exposure over transit paths.

5. Compatibility with Kafka Compaction and Schema Registries

  • Log Compaction: Because the Record Encryption filter can encrypt record values while preserving record keys (or using deterministic key encryption), Kafka’s background log-cleaner threads can still perform topic compaction seamlessly based on the key.
  • Schema Registry Neutrality: Producers and consumers continue using standard Avro, Protobuf, or JSON Schema serializers. Serialization occurs on the client before reaching the proxy, and deserialization happens after decryption on the consumer side. Kroxylicious treats the serialized payload transparently.

Example Kroxylicious Configuration

Here is an example snippet illustrating how Kroxylicious is configured with HashiCorp Vault to enforce encryption on designated topics:

 1adminHttp:
 2  endpoints:
 3    prometheus: {}
 4virtualClusters:
 5  - name: secure-kafka
 6    clusterNetworkAddressConfigProvider:
 7      type: PortPerBrokerClusterNetworkAddressConfigProvider
 8      config:
 9        bootstrapAddress: "kroxylicious.kafka.svc:9092"
10        brokerAddressPattern: "kroxylicious.kafka.svc:$(nodeId)"
11    targetCluster:
12      bootstrap_servers: "my-cluster-kafka-bootstrap.kafka.svc:9092"
13    filters:
14      - type: RecordEncryption
15        config:
16          kms: HashiCorpVaultKmsService
17          kmsConfig:
18            vaultUrl: "https://vault.internal.net:8200"
19            vaultToken: "${VAULT_TOKEN}"
20          selector: TemplateTopicName
21          selectorConfig:
22            template: "kafka-kek-%(topicName)"

In this configuration:

  • Producers and consumers connect to kroxylicious.kafka.svc:9092.
  • Any message destined for a topic (e.g. orders) dynamically resolves to a corresponding KEK in Vault (kafka-kek-orders).
  • Unencrypted data never touches the broker disks.

Architectural Considerations

While Kroxylicious brings immense security value, consider these production design factors:

  • Proxy Latency & Throughput: Symmetric AES-GCM encryption is hardware-accelerated on modern CPUs (AES-NI). However, ensure Kroxylicious pods have adequate CPU resources allocated in Kubernetes.
  • KMS Caching: To avoid saturating your KMS API rate limits, Kroxylicious incorporates intelligent DEK caching so that it does not call the KMS for every single individual record.
  • High Availability: In Kubernetes, deploy Kroxylicious as a scaled Deployment behind a load balancer with anti-affinity rules to match the fault tolerance of your Kafka cluster.

Conclusion

Securing streaming data at rest should not force an enterprise to choose between weak disk-level protections and the immense engineering burden of client-side encryption.

By operating as an API-aware proxy, Kroxylicious provides the best of both worlds: true envelope encryption with fine-grained key isolation, completely transparent to your applications.

comments powered by Disqus