KAHIBARO
Discord Login Register ☰
☰

5.9.6. S3-Compatible Storage

Table of Contents

Why S3-Compatible Storage Matters

Most beginner projects store files on the local disk of the server. This is fine for experiments, but it quickly breaks in real applications:

Object storage solves these problems by giving you a separate, durable place to store files. The most famous example is Amazon S3. Many other services implement the same API, which is why we talk about S3-compatible storage.

In this chapter you will learn what S3-compatible storage is, how it is different from normal file storage, and how to use it in a backend application.

What “S3-Compatible” Means

The S3 API

S3 exposes a simple HTTP-based API with operations such as:

When a provider is S3-compatible, it means:

Examples of S3-compatible providers:

ProviderTypical Use
Amazon S3Original S3, part of AWS
MinIOSelf-hosted S3-compatible server
DigitalOcean SpacesManaged S3-compatible object storage
Backblaze B2Cheap long-term storage
WasabiLow-cost S3-compatible storage

You can usually switch providers with no application code changes, only environment variables like endpoint URLs or credentials.

Important:
S3-compatible means “same API style”, not “identical behavior”.
Always check:

  • Region names and endpoints,
  • How public URLs are formed,
  • Limits, pricing, and performance.

Buckets and Objects

Core Concepts

Object storage does not use folders in the same way as a normal filesystem. It uses two main concepts:

You can think of a bucket as a drive, and the object key as the path on that drive.

Example:

There are no real nested directories. The key is just a string. Providers may simulate folders by treating / characters in keys as path separators.

Example keys in one bucket:

KeyWhat it looks like
avatars/user_1.pngAvatar of user 1
avatars/user_2.pngAvatar of user 2
documents/invoices/2024-01.pdfInvoice of January 2024
logs/2024/08/28/log.txtLog file for specific date

To your application, each of these is just an object identified by its key string.

How S3-Compatible Storage Differs from Local Files

File System vs Object Storage

Typical differences:

FeatureLocal File SystemS3-Compatible Storage
StructureDirectories and filesBuckets and object keys
Operationsopen, read, write, os.*HTTP API (PUT, GET, DELETE, etc.)
LatencyVery lowHigher (network request)
Shared across serversHard without network filesystemsEasy, all servers talk to same endpoint
Durability & backupsUp to youHandled by the provider
ScalingYou manage disksProvider scales capacity & throughput

In object storage you cannot:

Instead, you upload the entire object again. For logs and similar use cases you usually create new objects instead of appending.

Basic Operations with S3-Compatible Storage

You will usually use a client library instead of crafting HTTP requests by hand. In Python, the standard choice for S3 APIs is boto3.

Below we show general patterns. The exact code examples use AWS-style syntax, but you can connect to any S3-compatible endpoint by changing configuration.

Connecting to an S3-Compatible Endpoint

You usually need:

Example configuration using environment variables:

bash 
export S3_ENDPOINT_URL="https://nyc3.digitaloceanspaces.com"
export S3_ACCESS_KEY_ID="your-access-key"
export S3_SECRET_ACCESS_KEY="your-secret-key"
export S3_REGION="nyc3"
export S3_BUCKET="my-app-uploads"

Python client setup:

python 
import os
import boto3
session = boto3.session.Session()
s3 = session.client(
    service_name="s3",
    endpoint_url=os.environ["S3_ENDPOINT_URL"],
    aws_access_key_id=os.environ["S3_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["S3_SECRET_ACCESS_KEY"],
    region_name=os.environ.get("S3_REGION"),
)
BUCKET_NAME = os.environ["S3_BUCKET"]

Uploading an Object

Example: upload a user avatar.

python 
def upload_avatar(user_id: int, file_bytes: bytes, content_type: str) -> str:
    key = f"avatars/user_{user_id}.png"
    s3.put_object(
        Bucket=BUCKET_NAME,
        Key=key,
        Body=file_bytes,
        ContentType=content_type,
    )
    return key

Here key is what you store in your database to remember where the avatar lives.

Example DB record:

user_idavatar_key
1avatars/user_1.png
2avatars/user_2.png

Downloading an Object

To read the bytes back:

python 
def download_avatar(key: str) -> bytes:
    response = s3.get_object(Bucket=BUCKET_NAME, Key=key)
    data = response["Body"].read()
    return data

Deleting an Object

python 
def delete_avatar(key: str) -> None:
    s3.delete_object(Bucket=BUCKET_NAME, Key=key)

Listing Objects

You can list all objects with a given prefix, which behaves like listing a folder.

python 
def list_user_avatars() -> list[str]:
    response = s3.list_objects_v2(
        Bucket=BUCKET_NAME,
        Prefix="avatars/",
    )
    contents = response.get("Contents", [])
    return [obj["Key"] for obj in contents]

If your bucket has:

The function will return only the two avatar keys.

Public URLs and Permissions

Controlling Who Can Access Objects

There are two main access patterns:

  1. Public objects
    Anyone with the URL can download the file. You usually use this for public assets, for example product images.
  2. Private objects
    Only authenticated requests or presigned URLs can access the file. You use this for user-specific or sensitive data.

Security for S3-compatible storage has two layers:

Public Objects

Some providers allow you to make an entire bucket or specific prefixes public. Then, each object can be accessed by a predictable URL.

Common patterns:

Example:

Public URL might look like:

Presigned URLs

Presigned URLs allow temporary access to a private object without sharing credentials. This is very useful in backend development.

Two common use cases:

  1. Your backend gives the client a presigned download URL for a file.
  2. Your backend gives the client a presigned upload URL to upload directly to storage.

Presigned Download URL Example

python 
def create_presigned_download_url(key: str, expires_in: int = 3600) -> str:
    url = s3.generate_presigned_url(
        "get_object",
        Params={"Bucket": BUCKET_NAME, "Key": key},
        ExpiresIn=expires_in,
    )
    return url

The client can use the returned url with a normal HTTP GET to download the file for the next 1 hour.

Presigned Upload URL Example

python 
def create_presigned_upload_url(
    key: str, content_type: str, expires_in: int = 3600
) -> dict:
    url = s3.generate_presigned_url(
        "put_object",
        Params={
            "Bucket": BUCKET_NAME,
            "Key": key,
            "ContentType": content_type,
        },
        ExpiresIn=expires_in,
    )
    return {"url": url, "key": key}

The client can then perform an HTTP PUT to that URL with the file body, and the object will be stored under key in your bucket.

Security rule:
Never trust the path or file names chosen by the client.
Your backend should generate the object key and control which prefixes the client can use.

Integrating S3-Compatible Storage with a Backend API

Typical Flow for File Uploads

A common pattern for modern backends:

  1. Client asks your API for a presigned upload URL.
  2. API:
    • Generates a safe key,
    • Creates a presigned upload URL,
    • Returns URL and key to client.
  3. Client uploads the file directly to S3-compatible storage using the URL.
  4. Client notifies your API that upload is finished and sends the key.
  5. API stores the key in the database.

This pattern avoids sending large files through your application server, which:

Example JSON API Contract

StepRequest from clientResponse from backend
1POST /files/avatar/upload-url (no file body){ "upload_url": "...", "key": "avatars/user_123/abc123.png" }
2PUT upload_url with file bytesS3-compatible service returns 200 OK
3POST /users/me/avatar with { "key": "avatars/user_123/abc123.png" }Backend stores key for the user and returns updated user

Serving Files

To serve files to users you have options:

Naming and Organizing Object Keys

Good Key Design

Key design affects:

Useful patterns:

  1. Prefix by resource type
    • avatars/user_123.png
    • products/53/images/main.jpg
    • orders/2024/08/28/order_987.pdf
  2. Include IDs and random parts

To avoid collisions and make guessing harder:

  1. Avoid user-controlled paths

Do not let clients send ../../secret or entire folder names.
Generate keys on the server.

Example Key Strategy

For a user uploads feature:

Example resulting keys:

PurposeKey example
Avataravatars/42/3e7ad2ed-09cc-4e3e-8c38-d36f40c8cbd9.png
Documentuser-files/42/2024-08-28/1a2b3c4d_invoice_march.pdf

Local Development with MinIO

Why MinIO

Using real cloud storage during development can be:

MinIO is an S3-compatible storage server you can run locally, for example with Docker. Your application connects in the same way as with real S3, but to localhost.

Mini example docker-compose.yml:

yaml 
version: "3.8"
services:
  minio:
    image: minio/minio
    command: server /data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin123
    ports:
      - "9000:9000"   # API endpoint
      - "9001:9001"   # Web console
    volumes:
      - ./minio-data:/data

Then set environment variables for your backend:

bash 
export S3_ENDPOINT_URL="http://localhost:9000"
export S3_ACCESS_KEY_ID="minioadmin"
export S3_SECRET_ACCESS_KEY="minioadmin123"
export S3_REGION="us-east-1"
export S3_BUCKET="my-app-uploads"

You create the bucket once using MinIO’s web console or a script, and your code stays mostly the same as with real S3.

Common Pitfalls and Best Practices

Common Pitfalls

PitfallProblemBetter approach
Hardcoding bucket and endpoint in codeHard to change per environmentUse environment variables and config files
Using user-provided filenames as keysName collisions, path tricks, information disclosureGenerate safe keys, store original name separately if needed
Returning internal object keys directlyLeaks bucket structure and provider infoUse public / CDN URLs or presigned URLs
Uploading via backend server for all filesHigh load, slow responsesUse presigned upload URLs for large files
Mixing public and private data in same pathHard to write security policiesUse separate prefixes or even separate buckets

Best Practices

Key best practices:

  • Use environment variables for all S3-compatible configuration.
  • Treat object keys as sensitive references, not as public identifiers.
  • Prefer presigned URLs for protected downloads and direct client uploads.
  • Design predictable key structures that are easy to clean up per user or resource.

Additional recommendations:

Summary

You have learned:

With these concepts, you can now replace simple local file storage with scalable and robust S3-compatible storage in your backend applications.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 2994 / 24h
Impressum / Imprint
Datenschutz / Data Policy