KAHIBARO
Discord Login Register ☰
☰

5.9.5. Object Storage

Table of Contents

Why Object Storage Matters for Backends

When your backend needs to store user files, images, videos, logs, or backups, you usually have two broad choices:

Object storage is the standard choice for modern production backends. It scales easily, is relatively cheap, and can be shared by many servers.

You will build on this chapter in the later “S3-Compatible Storage” and “Presigned URLs” chapters, so the focus here is on core concepts and how they differ from local file storage.

Key idea: In object storage, you store objects (file data + metadata) inside buckets, and you access them using keys (unique paths), usually through an HTTP API.

What Is Object Storage?

Object storage is a way of storing data where each file is treated as an independent object:

Unlike a traditional filesystem:

Common examples of object storage services:

ProviderExample service
Amazon Web ServicesS3 (Simple Storage Service)
Google CloudCloud Storage
Microsoft AzureBlob Storage
Self-hostedMinIO, Ceph, etc.

Buckets, Objects, and Keys

Object storage has a few core terms you will see everywhere.

Buckets

A bucket is a top-level container for your objects:

You usually:

Objects

An object is what you store inside a bucket:

Examples of objects:

From your backend’s point of view, you typically:

Keys (or Object Names)

Every object is identified by a key (also called object name). It looks like a path, but it is just a string.

Examples of keys:

There is no real directory structure. The part before / is not a real folder, it is just part of the key string.

This has important consequences:

Rule: Treat keys as opaque strings you control. Do not let users control keys directly, or they can overwrite other users’ files.

Object Storage vs Local File Storage

To understand why object storage is used so often, compare it with storing files on the server’s disk.

Local File Storage

Problems in real backend systems:

Object Storage

Benefits:

AspectLocal filesystemObject storage
ScalingHard, manualEasy, service scales for you
DurabilityYour responsibilityReplicated, resilient by default
Multi-serverExtra work (NFS, shared disk)All servers can access same bucket
AccessDirect file I/OHTTP API, SDKs
Cost modelDisk / volume costPer GB + bandwidth

Trade-offs:

For backend development, object storage is usually the default for user-uploaded content and large assets.

Typical Backend Use Cases

Here are common situations where your backend will use object storage.

User-Uploaded Files

Examples:

Typical flow:

  1. User uploads a file (e.g., via an HTTP form).
  2. Backend receives the file stream.
  3. Backend uploads the file to object storage (for example to a bucket).
  4. Backend stores the resulting key or URL in the database.

You never keep the whole file in your database. You keep only a reference.

Generated Files

Your backend might generate:

Instead of storing them on disk:

  1. Generate the file in memory or in a temp location.
  2. Upload it to object storage.
  3. Return a URL or key to the client.

Logs and Backups

Object storage is also used for:

These are usually written by background jobs or scheduled tasks, not by user-facing API endpoints.

Naming and Folder-like Structures

Even though there are no real folders, you will often want a predictable “structure” in your keys.

Using Prefixes as “Folders”

You can simulate folders using prefixes:

If you list objects with prefix avatars/, you get all profile images.

If you list objects with prefix invoices/2025/, you get all invoices for that year.

Designing Key Patterns

You should design keys so they are:

Common patterns:

Use caseKey example
User avataravatars/{user_id}/{uuid}.jpg
Public blog imagesblog/{post_id}/{slug}-{uuid}.webp
Private documentsusers/{user_id}/docs/{uuid}.pdf
Backupsbackups/{date}/{db_name}-{timestamp}.sql.gz

You will combine keys with presigned URLs in a later chapter to control access.

Rule: Never trust user file names directly as keys. Generate your own key that includes a random or unique ID, then store the original filename in metadata or in your database if you need it.

Metadata and Content Types

Each object can have metadata. Two important types are:

Content Type

When your backend uploads a file, it should set the correct content type so clients receive the correct header when they download it.

Examples:

File extensionContent type
.jpg, .jpegimage/jpeg
.pngimage/png
.pdfapplication/pdf
.jsonapplication/json
.csvtext/csv

If you do not set it, clients might download files with an incorrect or generic type, such as application/octet-stream.

Custom Metadata

You can attach lightweight custom metadata to objects, such as:

However, do not overuse object metadata for business data. Complex business information should live in your database, not in object metadata.

Uploading and Downloading in a Backend

Even though the details depend on the specific object storage service and SDK, most backends follow similar patterns.

Upload Flow Example (High Level)

  1. Receive a file in an API endpoint.
  2. Generate a new key for the object.
  3. Use the storage SDK or HTTP API to upload the file stream.
  4. On success, store the key in your database.
  5. Return some representation to the client, such as:
    • An internal ID that maps to the key.
    • A URL built from the bucket and key.
    • Or a short-lived presigned URL (covered later).

Download Flow Example (High Level)

  1. Client hits an endpoint, for example GET /users/{id}/avatar.
  2. Backend looks up the avatar key for that user in the database.
  3. Backend fetches the object from storage, or redirects client to object storage using a secure URL.
  4. Backend returns the data or redirect to the client.

You will often choose between:

Direct access is usually more efficient for large files, since your backend does not have to relay every byte.

Durability, Availability, and Cost

Object storage services usually promise high durability and good availability.

Durability

Durability is the probability that your data will remain safe over time.

For example, some services advertise something like:

This means the chance of losing data is extremely small, because the service:

Availability

Availability is the percentage of time the service is accessible.

For example:

Useful formula: Downtime per year (hours) $\approx (1 - \text{availability}) \times 8760$.

For example, with $99.9\%$ availability:

$$
\text{Downtime} = (1 - 0.999) \times 8760 = 8.76 \text{ hours}
$$

You normally rely on the storage provider’s durability and focus your backend on handling:

Cost Considerations

Object storage is usually billed by:

This affects backend design:

Permissions and Access Control

Object storage is separate from your application, so you must manage its permissions carefully.

Basic Model

You will use access keys or an identity system from the provider to authenticate.

Public vs Private Objects

For sensitive backend data, you should:

Common Patterns in Backend Design

Object storage changes how you think about files in your system.

Store Only References in the Database

You almost never store the binary file data in your relational database. Instead, you:

| Column | Example value |
|--------------|--------------------------------------------------|
| id | 42 |
| user_id | 123 |
| storage_key | users/123/docs/7f9a-uuid.pdf |
| mime_type | application/pdf |
| created_at | 2025-01-01 10:00:00 |

Idempotent Operations

Uploads or deletes can fail partially due to network issues.

You will often design operations that are idempotent, which means:

Examples:

You will see “Idempotency” in more detail in a later chapter.

Summary

Object storage is a central building block of modern backend systems:

Later chapters will build on this foundation to show how to connect to S3-compatible services, generate presigned URLs, and integrate object storage cleanly into your backend APIs.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 2992 / 24h
Impressum / Imprint
Datenschutz / Data Policy