KAHIBARO
Discord Login Register ☰
☰

5.9. Working with Files

Table of Contents

Why File Handling Matters for Backend Developers

Working with files is a core backend task. You will often need to:

In this chapter you will learn how to work with files in Python in a safe and backend friendly way.

Always treat file handling as an I/O operation that can fail. Never assume a file exists, is readable, or has valid content. Always validate and handle errors.

We will assume you already know basic Python syntax, functions, and exceptions from earlier chapters, and focus here on what is unique to file handling.

Paths and Directories

Before opening a file, you need to know where it is located. Backend projects usually handle two kinds of files:

Absolute vs Relative Paths

A path is a string that tells the operating system where a file is.

bash 
  # Suppose current directory is /home/app
  logs/app.log      # refers to /home/app/logs/app.log
  ../config/app.yml # refers to /home/config/app.yml

In backend applications, you often want to avoid plain string paths and instead use pathlib.

Using pathlib for Safer Paths

pathlib gives you an object oriented way to work with paths and is cross platform.

python 
from pathlib import Path
# Current working directory
cwd = Path.cwd()
print(cwd)
# Absolute path to a file in a "data" subdirectory
data_file = cwd / "data" / "users.csv"
print(data_file)
# Check if a file or directory exists
print(data_file.exists())      # True or False
print(data_file.is_file())     # True if it is a file
print(data_file.is_dir())      # True if it is a directory

Using the / operator with Path joins segments in a platform independent way.

In many backend apps you will want a path relative to the project root or the file containing your code, not the current working directory.

python 
from pathlib import Path
# Path to this Python file
this_file = Path(__file__)
# Project root might be the parent of this file, or higher
project_root = this_file.parent.parent
config_file = project_root / "config" / "app.yml"

This pattern is common when reading internal config or template files.

Opening and Closing Files

To read or write files, you use the built in open function. The most important part is to always close files properly.

The with Statement

You should almost always open files using a with block. This guarantees the file is closed even if an error occurs.

python 
from pathlib import Path
log_path = Path("logs/app.log")
# Writing to a file
with open(log_path, mode="a", encoding="utf-8") as log_file:
    log_file.write("Application started\n")
# Reading from a file
with open(log_path, mode="r", encoding="utf-8") as log_file:
    content = log_file.read()
    print(content)

As soon as Python leaves the with block, it automatically calls log_file.close() for you.

Rule: Always use with open(...) as f: for file operations to avoid resource leaks.

File Modes

The mode argument controls how you open the file.

ModeDescriptionCreates file?Truncates existing?
"r"Read textNoNo
"w"Write textYesYes, clears file
"a"Append text at endYesNo
"x"Create new file, fail if existsYesN/A
"rb"Read binaryNoNo
"wb"Write binaryYesYes
"ab"Append binaryYesNo

You can also combine with + for reading and writing, for example "r+", "w+", "a+", but for backend code this is less common and often less clear.

Use encoding="utf-8" for text to avoid encoding problems.

Reading Files

You often need to read configuration files, templates, or data imports.

Reading Entire File

Use .read() when the file is reasonably small, for example under a few megabytes.

python 
from pathlib import Path
config_path = Path("config/settings.json")
with open(config_path, mode="r", encoding="utf-8") as config_file:
    content = config_file.read()
print(content)

Reading Line by Line

For larger files, you should not load the whole file into memory at once. Instead, iterate line by line.

python 
from pathlib import Path
log_path = Path("logs/app.log")
with open(log_path, mode="r", encoding="utf-8") as log_file:
    for line in log_file:
        # Remove trailing newline
        line = line.rstrip("\n")
        print(line)

This pattern is very useful when you process large CSV exports or other line based logs.

You can also use .readline() or .readlines(), but the simple for line in file is usually best.

Handling Missing Files

When a file does not exist, open raises FileNotFoundError. In backend code, you should handle this gracefully.

python 
from pathlib import Path
def read_optional_config(path: Path) -> str | None:
    try:
        with open(path, mode="r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        # Config is optional, just return None
        return None

If a file is required, you might want to log a clear error and re raise the exception instead.

Writing Files

Backends often write logs, temporary files, exports, and cached data.

Overwriting vs Appending

python 
from pathlib import Path
output_path = Path("output/report.txt")
# Overwrite any existing file
with open(output_path, mode="w", encoding="utf-8") as report:
    report.write("User Report\n")
    report.write("===========\n")
# Append a line later
with open(output_path, mode="a", encoding="utf-8") as report:
    report.write("User: alice\n")

Be careful with "w" because it clears the file before writing.

Never open important data files in "w" mode unless you really want to erase existing content. Use "a" or explicit backups instead.

Ensuring Directories Exist

If you write to logs/app.log and the logs directory does not exist, open will fail. You should ensure directories exist.

python 
from pathlib import Path
log_dir = Path("logs")
log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / "app.log"
with open(log_path, mode="a", encoding="utf-8") as log_file:
    log_file.write("Application started\n")

Text vs Binary Files

Backends deal with both plain text files and binary files such as images, PDFs, or archives.

Text Files

Use text mode (for example "r", "w", "a") when the content is textual, such as JSON, CSV, logs, or source code.

python 
from pathlib import Path
env_file = Path(".env")
with open(env_file, "r", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

Binary Files

Use binary mode ("rb", "wb", "ab") for non text content, for example:

python 
from pathlib import Path
image_path = Path("images/logo.png")
copy_path = Path("images/logo_copy.png")
with open(image_path, "rb") as src, open(copy_path, "wb") as dst:
    # Copy bytes in chunks
    while chunk := src.read(8192):
        dst.write(chunk)

Note how we use a chunk size (here 8192 bytes) to avoid loading the entire file into memory. This is important for large files in backend systems.

Common Patterns for Backend File Handling

Backend applications often reuse a few typical patterns for file work.

Reading Configuration from Files

You might have configuration in JSON, YAML, or a simple key value format.

Example: simple KEY=VALUE configuration file.

python 
from pathlib import Path
def load_simple_env(path: Path) -> dict[str, str]:
    config: dict[str, str] = {}
    try:
        with open(path, "r", encoding="utf-8") as f:
            for line in f:
                line = line.strip()
                # Skip empty lines and comments
                if not line or line.startswith("#"):
                    continue
                key, sep, value = line.partition("=")
                if sep != "=":
                    # Ignore invalid lines
                    continue
                config[key.strip()] = value.strip()
    except FileNotFoundError:
        # Optional config
        return config
    return config
env_config = load_simple_env(Path(".env"))
print(env_config)

This is similar to what many web frameworks do internally when loading .env files.

Processing Uploaded Files

In a web backend, an uploaded file often arrives as bytes. Even though the framework may already manage the streaming, you sometimes need to save the uploaded data to disk.

A typical pattern:

python 
from pathlib import Path
def save_uploaded_file(base_dir: Path, filename: str, content: bytes) -> Path:
    base_dir.mkdir(parents=True, exist_ok=True)
    file_path = base_dir / filename
    with open(file_path, "wb") as f:
        f.write(content)
    return file_path

In a real FastAPI endpoint, you would receive the content differently, but the core binary write pattern is the same.

Generating Export Files

Backends often create CSV or text exports.

python 
from pathlib import Path
from typing import Iterable
def export_users_to_csv(path: Path, users: Iterable[dict]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        # Write header
        f.write("id,username,email\n")
        for user in users:
            line = f"{user['id']},{user['username']},{user['email']}\n"
            f.write(line)
users_data = [
    {"id": 1, "username": "alice", "email": "alice@example.com"},
    {"id": 2, "username": "bob", "email": "bob@example.com"},
]
export_users_to_csv(Path("exports/users.csv"), users_data)

Later, another part of your system might serve this file as a download.

Safe and Secure File Handling

File handling is also a security sensitive area for backends.

Avoid User Controlled Paths

Never let users control full paths directly. For example, this is dangerous:

python 
# Dangerous: user can access arbitrary files
def read_log_file(file_name: str) -> str:
    with open(file_name, "r", encoding="utf-8") as f:
        return f.read()

A malicious user could pass "../../etc/passwd" and read system files.

Instead, restrict file operations to a specific directory and clean the file name.

python 
from pathlib import Path
LOGS_DIR = Path("logs").resolve()
def safe_read_log_file(file_name: str) -> str:
    # Only allow simple names without path separators
    if "/" in file_name or "\\" in file_name:
        raise ValueError("Invalid file name")
    file_path = (LOGS_DIR / file_name).resolve()
    # Ensure the resolved path is still inside LOGS_DIR
    if not str(file_path).startswith(str(LOGS_DIR)):
        raise ValueError("Invalid file path")
    with open(file_path, "r", encoding="utf-8") as f:
        return f.read()

This pattern helps prevent directory traversal attacks.

Validate File Sizes

Users can upload very large files that might exhaust memory or disk space.

A common pattern is to check the file size before fully processing it, or process in chunks and stop if the limit is exceeded.

Pseudo pattern:

python 
MAX_FILE_SIZE = 10 * 1024 * 1024  # 10 MB
def validate_file_size(file_path: Path) -> None:
    size = file_path.stat().st_size
    if size > MAX_FILE_SIZE:
        raise ValueError("File too large")

Or, if you stream the content, you can count bytes as you read them and stop early.

Handle Errors Gracefully

Many things can go wrong with files:

Typical backend pattern:

python 
from pathlib import Path
def read_template(path: Path) -> str:
    try:
        with open(path, "r", encoding="utf-8") as f:
            return f.read()
    except FileNotFoundError:
        # Log and re raise, or fallback to a default template
        raise RuntimeError(f"Template not found: {path}")
    except PermissionError:
        raise RuntimeError(f"No permission to read template: {path}")

Central error handling higher in your stack can then convert RuntimeError into a proper HTTP response.

Temporary Files

Temporary files are useful when you need to hold data briefly, for example for an external tool, before deleting it again.

The tempfile module in the standard library helps with this.

python 
import tempfile
from pathlib import Path
def create_temp_report(content: str) -> Path:
    # Creates a named temporary file that we can reopen
    with tempfile.NamedTemporaryFile(
        mode="w",
        encoding="utf-8",
        delete=False,  # Do not delete automatically
        suffix=".txt",
        prefix="report_",
    ) as tmp:
        tmp.write(content)
        temp_path = Path(tmp.name)
    # Later we can serve or move this file
    return temp_path
report_path = create_temp_report("Temporary report content")
print("Temp report at:", report_path)

Backends often combine this with cron jobs or cleanup tasks that remove old temporary files.

Binary Streaming and Large Files

For very large files, you must avoid reading everything into memory at once.

The chunk pattern applies for both reading and writing:

python 
from pathlib import Path
def copy_large_file(src: Path, dst: Path, chunk_size: int = 1024 * 1024) -> None:
    with open(src, "rb") as f_src, open(dst, "wb") as f_dst:
        while True:
            chunk = f_src.read(chunk_size)
            if not chunk:
                break
            f_dst.write(chunk)

This is similar to what a web framework does internally when it streams a file to a client.

Summary

In backend development, file handling is about:

These patterns will appear again when you handle uploads, downloads, logs, and integrations in later chapters of this course.

5.9.1. File Downloads
5.9.2. File Validation
5.9.3. Image Uploads
5.9.4. Local File Storage
5.9.5. Object Storage
5.9.6. S3-Compatible Storage
5.9.7. Presigned URLs
5.9.8. Secure File Handling

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 2999 / 24h
Impressum / Imprint
Datenschutz / Data Policy