Upload Files to S3 Using API Gateway: A Step-by-Step Guide

When it comes to uploading files on AWS, the combination of API Gateway and Lambda is a common pattern. However, for simple file uploads, using Lambda solely as a proxy can introduce unnecessary costs and latency. In this guide, we will explore how to upload files directly to Amazon S3 via API Gateway, bypassing Lambda entirely.

Why Direct Integration Instead of Lambda?

In the traditional approach (API Gateway -> Lambda -> S3), the Lambda function acts as a middleman. While flexible, this adds overhead:

• Cost: You pay for Lambda execution time while waiting for S3 to respond.
• Latency: The extra hop adds milliseconds to your request.
• Limits: API Gateway has a 10MB payload limit, and Lambda has 6MB, making large uploads tricky.

Direct integration (API Gateway -> S3) offers:

• Lower Costs: No Lambda execution fees.
• Reduced Latency: Faster processing by removing the middle layer.
• Simplified Architecture: Fewer moving parts to manage.

Step 1: Setting Up IAM Roles and Policies

First, we need an IAM role that allows API Gateway to write objects to your S3 bucket.

1. Go to the IAM console in AWS.
2. Click Create role.
3. Select AWS Service as the trusted entity and choose API Gateway.
4. Click Next and Create Policy with the following JSON (replace with your bucket name):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

5. Attach this policy to the role and save it. Note down the Role ARN.

Step 2: Create API Gateway REST API

1. Navigate to API Gateway in the AWS Console.
2. Click Create API and select REST API.
3. Name your API (e.g., S3UploadAPI) and click Create.

Step 3: Define Resources and Methods

We'll define a path structure like /{bucket}/{filename}.

1. Under Resources, select Create Resource and name it bucket.
2. Inside /bucket, create a child resource named filename.
3. With /filename selected, click Create Method and choose PUT.

Step 4: Integration Configuration

Configure the PUT method as follows:

• Integration type: AWS Service
• AWS Region: Your S3 bucket region (e.g., us-east-1).
• AWS Service: S3
• HTTP method: PUT
• Action Type: Path override
• Path override: {bucket}/{key} (We will map these from URL parameters).
• Execution role: The ARN of the IAM role created in Step 1.

After saving, go to the Integration Request section and edit URL Path Parameters:

• Name: bucket -> Mapped from: method.request.path.bucket
• Name: key -> Mapped from: method.request.path.filename

Step 5: Binary Media Types

To handle binary files like images or PDFs correctly:

1. Go to Settings in the API menu.
2. Under Binary Media Types, add */* (for all types) or specific MIME types like image/jpeg.
3. Save changes.

Step 6: Deploy and Test

1. Click Actions -> Deploy API.
2. Create a new Stage (e.g., dev).
3. Copy the Invoke URL.

Test using curl:

curl -X PUT -H "Content-Type: image/jpeg" \
  --data-binary "@test-image.jpg" \
  https://your-api-id.execute-api.us-east-1.amazonaws.com/dev/your-bucket-name/uploaded-image.jpg

If successful, you will see the file in your S3 bucket.

Conclusion

You have now set up a streamlined, serverless file upload mechanism on AWS. This architecture is efficient, cost-effective, and removes the complexity of managing Lambda functions for simple proxy tasks. It's a great pattern for modern cloud-native applications.