EKS and AWS OpenSearch Integration

Your application logs are valuable not just for debugging, but also for business analytics and security. One of the most powerful tools for analyzing logs on Amazon EKS is Amazon OpenSearch Service.

The most efficient way to set up this integration is to use Fluent Bit, a lightweight log agent.

Architecture

1. Fluent Bit (DaemonSet): Runs one on every worker node. Reads logs of all pods on that node.
2. Buffer & Filter: Enriches logs (adds Kubernetes metadata: Pod name, namespace, etc.) and filters unnecessary logs.
3. Output: Sends logs securely (HTTPS) to Amazon OpenSearch.

Installation

AWS offers a ready configuration (Helm Chart) for Fluent Bit.

1. Create a Domain on OpenSearch.
2. Create an IAM role (IRSA) granting write permission to OpenSearch for Fluent Bit.
3. Provide this role and the OpenSearch endpoint as parameters when installing the Helm chart.

Visualization with Kibana

After logs flow into OpenSearch, enter the Kibana interface and create an "Index Pattern". Now you can get answers to questions like "Which microservice gave the most 500 errors in the last hour?" with charts in seconds.