Improving Kubernetes Infrastructure with AWS Bottlerocket

Bottlerocket is a Linux-based operating system specifically designed and optimized for running containers, particularly in AWS and Kubernetes environments.

AWS created Bottlerocket to provide a better foundation for running containers on their services, particularly:

• Amazon EKS (Elastic Kubernetes Service)
• Amazon ECS (Elastic Container Service)

However, while AWS developed it, they made it open source and cloud-agnostic. Meaning:

• Anyone can use it, contribute to it, or modify it.
• It can run on bare metal, VMware, and other cloud providers.
• The community can participate in its development.

This approach is similar to AWS's open-sourcing of other projects, such as Firecracker (the microVM technology behind AWS Lambda and Fargate).

Will Bottlerocket Replace Amazon Linux?

No, Bottlerocket is not replacing Amazon Linux. They serve different purposes and will coexist. Amazon Linux is a general-purpose operating system.

Bottlerocket and Talos Linux are probably the most similar in philosophy — both disable SSH access by default and use API-driven configuration, making them well-suited to immutable infrastructure patterns.

Do We Have All Bash Commands in Bottlerocket?

No, Bottlerocket does not have all the traditional bash commands you'd find in a standard Linux distribution. This is by design.

• Bottlerocket is intentionally stripped down to only what's needed to run containers.
• It uses a control container and an admin container model for management.
• Most traditional Linux utilities and shells are not included in the base OS.

Bottlerocket itself does not have a shell. It doesn't need one. You can still interact with the system through privileged "host" containers (that do have shells).

Let's See Some Action!

Bottlerocket uses TOML. Unlike YAML, indentation doesn't affect parsing, and unlike JSON, it supports comments, making it more human-friendly.

To set up a Bottlerocket cluster on EKS, we assume you already have an AWS account, the AWS CLI, kubectl, and eksctl installed.

Project Structure

The project is structured to make it easy to identify the purpose of each .tf file.

We can automate the infrastructure setup with Terraform. The bottlerocket-userdata.toml.tpl file is rendered by Terraform with cluster-specific values (Cluster Name, API Server, Certificate), and the Bottlerocket API reads this on first boot to configure the node.

You can use a bash script (setup-terraform.sh) to automate the process. This script checks prerequisites, initializes Terraform, and performs the deployment.

Once setup is complete, you can verify that the nodes are running the Bottlerocket OS using kubectl get nodes -o wide.

Note: You cannot upgrade the Kubernetes cluster version (e.g., from 1.30 to 1.31) using the Bottlerocket API. The API only manages settings and updates for the Bottlerocket OS on individual nodes.

eBPF and Observability

Bottlerocket comes with a recent Linux kernel and the necessary configuration to run eBPF programs. You don't need to manually install or configure kernel headers or modules.

With an eBPF demo, you can monitor file opens or TCP connections on the system in real-time. These tools are critical for debugging and security monitoring.

Why Bottlerocket?

Advantages compared to general-purpose OS like Amazon Linux or Ubuntu:

• Smaller Attack Surface: Only includes what's needed to run containers.
• Automatic Updates: Supports transactional OS updates.
• Optimized for Kubernetes: Designed to work seamlessly with EKS.
• eBPF Support Out-of-the-Box: No manual configuration required.
• Consistency: All nodes run the same minimal OS.

What Does Immutability Mean?

Bottlerocket is immutable, meaning:

1. Read-Only Root Filesystem: / is mounted as read-only. You cannot modify system files.
2. Image-Based Updates: Updates replace the entire OS image atomically.
3. Dual Partition: Updates are written to the passive partition and activated upon reboot. Can rollback on failure.

Migration Considerations

Migrating from Amazon Linux or Ubuntu to Bottlerocket is not transparent:

• No Package Manager: apt install or yum install won't work.
• No SSH: You must use SSM or admin containers.
• No Docker: Bottlerocket only uses containerd. Since Kubernetes 1.24+ already standardized on containerd, your workloads won't notice the difference.

Cleanup

Don't forget to clean up resources when the demo is done to avoid costs. You can use the cleanup.sh script or terraform destroy.

Bottlerocket makes it easier, safer, and more consistent to run container workloads (and eBPF tools) at scale in your DevOps processes.