KAHIBARO
Discord Login Register ☰
☰

5.9.4. Local File Storage

Table of Contents

Why Local File Storage Matters

When you build backend applications, sooner or later you must handle files: user avatars, documents, reports, exports, logs, and more. Local file storage means you save files directly on the server’s filesystem instead of in an external service like Amazon S3.

Local storage is simple, fast to start with, and great for development or small projects. This chapter focuses on how to do it correctly and safely.

Key idea: Local file storage = files stored on the server’s filesystem, referenced in your app by paths or URLs, not stored in the database.

You should already know how to upload and download files in general. Here we focus on how to organize and manage those files on the local filesystem.

Basic Concepts of Local File Storage

Files vs Database Records

Usually, you do not store entire files in your database. Instead, you:

  1. Store the file content on disk (local storage).
  2. Store a reference to the file in the database.

Typical database fields:

ColumnExample valuePurpose
id42Record identifier
user_id10Who owns the file
file_namereport.pdfOriginal name
stored_namef2b7aafc-9ab1-4b6c-b65b-1c43f5f3.pdfUnique name on disk
pathuploads/reports/2026/08/f2b7...3.pdfRelative path on disk
size_bytes204800File size
mime_typeapplication/pdfFile type
created_at2026-08-28 12:34:56When it was uploaded

The database knows about the file, but the actual bytes live on disk.

Choosing a Storage Location

Project-relative vs Absolute Paths

You usually configure a base directory for your files, for example:

text 
/my-app/
  app/
  venv/
  uploads/        <-- base storage directory
  static/

In code you might have:

python 
BASE_DIR = Path(__file__).resolve().parent.parent
UPLOADS_DIR = BASE_DIR / "uploads"

Whenever you save a file, you join UPLOADS_DIR with a relative path such as "avatars/user_10.png".

Avoid hard-coded absolute paths like /home/username/app/uploads inside your code. Instead, use a configuration setting or environment variable such as:

bash 
FILE_STORAGE_DIR=/var/myapp/uploads

And load it in Python:

python 
import os
from pathlib import Path
UPLOADS_DIR = Path(os.environ.get("FILE_STORAGE_DIR", "uploads"))

This makes it easy to change the storage location in different environments (development, staging, production).

Directory Structure and Organization

Local storage can grow messy if you dump everything into a single folder. A good directory structure helps performance and maintainability.

Example Directory Structures

Some common patterns:

PatternExample pathUse case
By file typeuploads/avatars/123.pngSimple separation by purpose
By useruploads/users/10/avatar.pngUser-specific files
By dateuploads/2026/08/28/file-uuid.pdfSpreads files over folders
By user and dateuploads/users/10/2026/08/file-uuid.pdfLarge apps with many users
By hash prefixuploads/ab/cd/ef/uuid.jpgFor millions of files

Why you should avoid huge flat directories

Most filesystems perform badly if you put hundreds of thousands of files in a single directory. To avoid this, you can:

In Python, constructing such paths:

python 
from pathlib import Path
from datetime import datetime
import uuid
def build_user_file_path(user_id: int, original_name: str) -> Path:
    today = datetime.utcnow()
    extension = Path(original_name).suffix  # e.g. ".png"
    unique_name = f"{uuid.uuid4()}{extension}"
    return Path("uploads") / "users" / str(user_id) / str(today.year) / f"{today.month:02d}" / unique_name

This returns a relative path like:
uploads/users/10/2026/08/550e8400-e29b-41d4-a716-446655440000.png

Generating Safe and Unique Filenames

You should never trust user-provided filenames directly.

Problems with user filenames

Rule: Always sanitize or ignore the original filename and generate your own safe and unique filename.

Strategies for unique filenames

  1. UUID-based names
python 
   import uuid
   from pathlib import Path
   def generate_unique_name(original_filename: str) -> str:
       ext = Path(original_filename).suffix.lower()  # keep .png, .jpg, etc.
       return f"{uuid.uuid4()}{ext}"
  1. Timestamp + random
python 
   import time, secrets
   from pathlib import Path
   def generate_unique_name_ts(original_filename: str) -> str:
       ext = Path(original_filename).suffix.lower()
       ts = int(time.time() * 1000)
       rand = secrets.token_hex(4)
       return f"{ts}-{rand}{ext}"
  1. Hash of content (for deduplication)

You can hash the file content with SHA-256 and use part of that:

python 
   import hashlib
   from pathlib import Path
   def hash_file_content(file_bytes: bytes, original_filename: str) -> str:
       ext = Path(original_filename).suffix.lower()
       file_hash = hashlib.sha256(file_bytes).hexdigest()
       # Use first 16 chars to keep names shorter
       return f"{file_hash[:16]}{ext}"

Usually you also store the original name separately in the database to show it back to the user.

Writing Files Safely

Creating directories if missing

Before writing a file, ensure its parent directories exist:

python 
from pathlib import Path
def save_bytes_to_path(base_dir: Path, relative_path: Path, content: bytes) -> Path:
    full_path = base_dir / relative_path
    full_path.parent.mkdir(parents=True, exist_ok=True)
    full_path.write_bytes(content)
    return full_path

This method:

  1. Combines base_dir and relative_path.
  2. Creates all missing parent directories.
  3. Writes the file content.

Streaming uploads instead of loading everything in memory

When files are large, you do not want to load all bytes into memory at once. Most web frameworks give you a file-like object you can stream to disk in chunks:

python 
def save_uploaded_file(base_dir: Path, relative_path: Path, upload_file) -> Path:
    full_path = base_dir / relative_path
    full_path.parent.mkdir(parents=True, exist_ok=True)
    with full_path.open("wb") as f:
        for chunk in iter(lambda: upload_file.file.read(1024 * 1024), b""):
            f.write(chunk)
    return full_path

This reads at most 1 MB at a time.

Avoiding directory traversal attacks

User input might try to escape your upload directory, for example with ../../etc/passwd. To guard against this, you:

  1. Generate your own file names and relative paths.
  2. If you ever use a user-provided path segment, validate it strictly.

Example of validation with Path.resolve:

python 
def is_path_inside_base(base: Path, child: Path) -> bool:
    base = base.resolve()
    child = child.resolve()
    try:
        child.relative_to(base)
        return True
    except ValueError:
        return False

After computing a file path, you can assert:

python 
if not is_path_inside_base(UPLOADS_DIR, full_path):
    raise ValueError("Invalid file path")

Reading and Serving Files

Your application needs to:

  1. Find the file based on the database record.
  2. Check permissions: only the owner or allowed users can access it.
  3. Serve the file over HTTP.

Finding the file

Typical record in your database:

python 
class FileRecord(BaseModel):
    id: int
    user_id: int
    path: str  # e.g. "uploads/users/10/2026/08/file-uuid.png"
    mime_type: str
    original_name: str

To get the full path:

python 
from pathlib import Path
full_path = UPLOADS_DIR.parent / record.path  # if path is relative to project root

Or if you store path relative to uploads:

python 
full_path = UPLOADS_DIR / record.path  # e.g. "users/10/2026/08/file-uuid.png"

Checking permissions

Before serving, check access:

If not, return an HTTP 403 Forbidden or 404 Not Found.

Serving the file with correct headers

For example, in FastAPI (only the idea here, details belong to other chapters):

python 
from fastapi.responses import FileResponse
return FileResponse(
    path=str(full_path),
    media_type=record.mime_type,
    filename=record.original_name
)

This sends the file with proper Content-Type and optional Content-Disposition so browsers can display or download it.

Updating and Deleting Files

Replacing a file

When a user uploads a new avatar:

  1. Save new file to disk.
  2. Update database record to point to the new path.
  3. Delete the old file from disk.

Example:

python 
import os
def replace_user_file(base_dir: Path, record, new_content: bytes, new_original_name: str):
    old_full_path = base_dir / record.path
    # 1. Save new file
    new_name = generate_unique_name(new_original_name)
    new_rel_path = Path("users") / str(record.user_id) / new_name
    new_full_path = save_bytes_to_path(base_dir, new_rel_path, new_content)
    # 2. Update database fields
    record.path = str(new_rel_path)
    record.file_name = new_original_name
    # Save record to DB here (omitted)
    # 3. Remove old file if it exists
    try:
        if old_full_path.exists():
            old_full_path.unlink()
    except OSError:
        # Log error, but do not fail user request just because cleanup failed
        pass

Deleting a file

When deleting a record:

  1. Delete the database record.
  2. Delete the associated file from disk.
python 
def delete_user_file(base_dir: Path, record):
    full_path = base_dir / record.path
    # 1. Delete DB record (omitted)
    # 2. Delete file
    try:
        if full_path.exists():
            full_path.unlink()
    except OSError:
        # Log and continue
        pass

You must decide what happens if the file is missing. Usually you ignore it and log a warning.

Configuration and Environment-specific Paths

Local file storage paths should not be hard-coded in code. Use configuration instead.

Example configuration sources

Example with environment variables and defaults

python 
import os
from pathlib import Path
BASE_DIR = Path(__file__).resolve().parent.parent
DEFAULT_UPLOADS_DIR = BASE_DIR / "uploads"
UPLOADS_DIR = Path(os.getenv("FILE_STORAGE_DIR", DEFAULT_UPLOADS_DIR))

In development, you use the default ./uploads. In production, you set FILE_STORAGE_DIR to some directory with enough space.

Security Considerations for Local Storage

Local file storage introduces several security concerns.

Executable files on the server

If a user uploads a file with .py, .php, .sh, etc, and you serve it from a web server that can execute those file types, you can get remote code execution.

To reduce risk:

Path validation

Never allow a user to control the full filesystem path.

Rule: Build all paths from safe base directories and safe, generated names. Never use raw user-provided path strings as filesystem paths.

Permissions on the filesystem

On Linux you can create a dedicated user, for example myapp, and give it write access only to /var/myapp/uploads. If an attacker breaks into the app, they can only modify that directory, not the whole server.

Backups and Local Storage

Files in local storage must be backed up, just like your database.

Common backup approaches:

You also need a plan for restore:

Local Storage vs Object Storage (Brief Comparison)

You will learn more about object storage and S3-compatible storage in another chapter. Here, understand quickly how local storage compares.

AspectLocal file storageObject storage (e.g. S3)
SetupVery simple, built-in filesystemNeeds external service
ScalabilityLimited to a single disk or serverDesigned to scale
Multiple serversHard: need shared disk or syncingEasy: all servers use same bucket
AccessFast local disk accessNetwork-based, slightly more latency
BackupsYou manage them manuallyOften built-in or easier to integrate
Best forDevelopment, small projects, prototypesProduction, large apps, distributed apps

Local storage is usually fine for:

As your app grows, you may migrate to an object storage solution.

Example: Simple Local File Storage Helper

Below is a small utility module that summarizes many ideas from this chapter. In a real project you might create something similar.

python 
# file_storage.py
import os
import uuid
from pathlib import Path
from datetime import datetime
from typing import BinaryIO
BASE_DIR = Path(__file__).resolve().parent.parent
DEFAULT_UPLOADS_DIR = BASE_DIR / "uploads"
UPLOADS_DIR = Path(os.getenv("FILE_STORAGE_DIR", DEFAULT_UPLOADS_DIR))
def generate_unique_name(original_filename: str) -> str:
    ext = Path(original_filename).suffix.lower()
    return f"{uuid.uuid4()}{ext}"
def build_relative_path(user_id: int, original_filename: str) -> Path:
    today = datetime.utcnow()
    unique_name = generate_unique_name(original_filename)
    return Path("users") / str(user_id) / str(today.year) / f"{today.month:02d}" / unique_name
def save_file_stream(user_id: int, original_filename: str, file_obj: BinaryIO) -> str:
    rel_path = build_relative_path(user_id, original_filename)
    full_path = UPLOADS_DIR / rel_path
    full_path.parent.mkdir(parents=True, exist_ok=True)
    with full_path.open("wb") as f:
        for chunk in iter(lambda: file_obj.read(1024 * 1024), b""):
            f.write(chunk)
    # Return relative path as string to store in DB
    return str(rel_path)
def delete_file(relative_path: str) -> None:
    full_path = UPLOADS_DIR / relative_path
    try:
        if full_path.exists():
            full_path.unlink()
    except OSError:
        # Log in real app
        pass

Your API endpoint would:

  1. Receive the file from the client.
  2. Call save_file_stream with the user id and original filename.
  3. Store the returned relative path and metadata in the database.

Summary

Views: 5

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3040 / 24h
Impressum / Imprint
Datenschutz / Data Policy