KAHIBARO
Discord Login Register ☰
☰

16.3.5. Distributed Locks

Table of Contents

Why Distributed Locks Are Needed

When your backend runs on a single server, you can often rely on in-process synchronization tools like mutexes or locks provided by the programming language.

In a distributed system, you usually have:

If two instances try to modify the same resource at the same time, you can get:

A distributed lock is a mechanism that lets multiple nodes coordinate access to a shared resource by ensuring only one node holds the lock at a time.

Common use cases:

Redis is a common choice for distributed locks because:

Key idea: A distributed lock must be mutual exclusive, have a timeout, and be safely released only by its owner.

Basic Locking with Redis

The simplest way to think about a lock in Redis is:

Naive Implementation

A naive lock:

  1. Try to create a key if it does not exist.
  2. If successful, you hold the lock.
  3. When you are done, delete the key.

In Redis, you might try:

bash 
SETNX lock:report:daily "locked"

In Python (pseudocode):

python 
acquired = redis.setnx("lock:report:daily", "locked")
if acquired:
    # Do the job
    ...
    redis.delete("lock:report:daily")
else:
    # Someone else is already running it
    pass

SETNX means SET if Not eXists. It returns true if the key was created.

This shows the basic idea, but it has several important problems.

Pitfalls of Naive Locking

Problem 1: Locks Without Expiration

If you use SETNX directly and never set an expiry, then:

The key will stay in Redis forever.

Result: The lock is never released and no one else can acquire it. This is called a deadlock in this context.

Example

  1. Worker A acquires lock with SETNX lock:report:daily
  2. Worker A crashes before DEL lock:report:daily
  3. Worker B will always fail to acquire the lock
  4. The daily report will never run again

Problem 2: Releasing Someone Else’s Lock

Consider this flow:

  1. Process A acquires the lock.
  2. Process A takes a long time, longer than expected.
  3. Another process B might try a workaround and delete the lock key (or use some bug or retry logic).
  4. Process B now thinks it has the lock and starts working, while process A is still working, which breaks mutual exclusion.

Even simpler, if multiple processes use DEL blindly, any process can delete any lock held by someone else.

We need a way to ensure only the owner of a lock can release it.

Problem 3: Race Conditions Around Timeouts

You might think: "I will set an expiration on the lock":

bash 
SETNX lock:report:daily "locked"
EXPIRE lock:report:daily 60

But there is a race here:

  1. You run SETNX and get the lock.
  2. Before you call EXPIRE, the process crashes.
  3. No expiration is set, so the lock becomes permanent again.

You need a single atomic command that both acquires the lock and sets the TTL.

Implementing Safe Locks with SET, NX, and PX

Redis provides a more powerful SET command with options:

bash 
SET key value [NX|XX] [EX seconds|PX milliseconds]

This allows an atomic operation:

bash 
SET lock:report:daily my-unique-id NX EX 60

This means:

If Redis returns OK, you acquired the lock. If it returns nil (or your client returns False), the lock is held by someone else.

Rule: Always use SET key value NX EX ttl or SET key value NX PX ttl for acquiring locks. This guarantees the key creation and expiration are atomic.

Using a Unique Value

Why not just write "locked" as the value?

Because we want to:

The unique value:

Example in Python:

python 
import uuid
lock_key = "lock:report:daily"
lock_value = str(uuid.uuid4())  # random unique ID
acquired = redis_client.set(lock_key, lock_value, nx=True, ex=60)
if acquired:
    # We hold the lock
    ...
else:
    # Someone else holds it
    ...

Releasing Locks Safely with Lua Scripts

We have a lock value that identifies the owner. Now we must only delete the lock if we are still the owner.

Naive unlock:

python 
redis_client.delete(lock_key)

This is unsafe because:

Correct unlock pattern:

  1. Read the current value of the lock key.
  2. Compare it with your unique value.
  3. If it matches, delete the key.
  4. If not, do nothing.

If you do this as separate commands:

python 
if redis_client.get(lock_key) == lock_value:
    redis_client.delete(lock_key)

There is a race condition:

To avoid this, use a Lua script that runs atomically in Redis.

Typical Unlock Lua Script

lua 
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
else
    return 0
end

Use it from Python:

python 
import uuid
lock_key = "lock:report:daily"
lock_value = str(uuid.uuid4())
# Try to acquire
acquired = redis_client.set(lock_key, lock_value, nx=True, ex=60)
if not acquired:
    print("Could not acquire lock.")
else:
    try:
        # Do protected work
        print("Lock acquired, running job")
        # ... your critical section ...
    finally:
        # Release safely
        unlock_script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("DEL", KEYS[1])
        else
            return 0
        end
        """
        redis_client.eval(unlock_script, 1, lock_key, lock_value)

Important points:

Rule: Always release locks with a compare-and-delete operation, not with a blind DEL.

Example: Single-Instance Job with Redis Lock

Scenario: You have 5 application servers, and each runs a scheduled task every minute to process a daily report. You want only one of them to actually run the job.

Pseudocode:

python 
import uuid
import time
LOCK_KEY = "lock:report:daily"
LOCK_TTL = 120  # seconds
def acquire_lock(redis_client, key, ttl):
    value = str(uuid.uuid4())
    acquired = redis_client.set(key, value, nx=True, ex=ttl)
    if acquired:
        return value
    return None
def release_lock(redis_client, key, value):
    script = """
    if redis.call("GET", KEYS[1]) == ARGV[1] then
        return redis.call("DEL", KEYS[1])
    else
        return 0
    end
    """
    redis_client.eval(script, 1, key, value)
def run_daily_job(redis_client):
    lock_value = acquire_lock(redis_client, LOCK_KEY, LOCK_TTL)
    if not lock_value:
        print("Another instance is already running the job.")
        return
    try:
        print("Running daily job...")
        time.sleep(10)  # simulate work
        print("Job done.")
    finally:
        release_lock(redis_client, LOCK_KEY, lock_value)
        print("Lock released.")

If you run run_daily_job on multiple instances at the same time, only one will print "Running daily job...". Others will print "Another instance is already running the job."

Lock TTL and Auto Expiration

A lock must have a timeout to avoid permanent deadlocks.

Choosing TTL

You should pick a TTL that:

For example:

Table of example TTL choices:

Job TypeTypical DurationSuggested TTL
Small cache refresh< 1 second5 seconds
Short background task~5 seconds30 seconds
Long report generation~60 seconds300 seconds
Nightly batch processing~5 minutes1200 seconds

Rule: Always give locks a TTL. Never create locks without expiration in a distributed system.

What if the Job Exceeds TTL?

If your job takes longer than the lock's TTL:

This can be a problem in some use cases, for example payment processing.

Solutions:

A simple extension strategy:

Leases and Lock Renewal

Instead of a plain lock, you can think of a lease: a lock that you hold for a limited time, which you can renew.

Basic lease logic:

  1. You acquire the lock with a TTL, for example 30 seconds.
  2. While you are working, every 10 or 20 seconds you:
    • Check if the lock is still owned by you.
    • If yes, extend the TTL by another 30 seconds.
  3. If the process crashes, no one will renew the lock and it will expire.

This reduces the risk that a very long running task loses the lock prematurely, while still giving a bound on how long the system can be stuck if the node fails.

Pseudo implementation idea:

python 
import threading
import time
def renew_lock_periodically(redis_client, key, value, ttl, interval, stop_event):
    while not stop_event.is_set():
        time.sleep(interval)
        # Renew only if we still own the lock
        renew_script = """
        if redis.call("GET", KEYS[1]) == ARGV[1] then
            return redis.call("PEXPIRE", KEYS[1], ARGV[2])
        else
            return 0
        end
        """
        redis_client.eval(renew_script, 1, key, value, int(ttl * 1000))
def do_work_with_lease(redis_client):
    lock_key = "lock:long:task"
    ttl = 30
    lock_value = str(uuid.uuid4())
    acquired = redis_client.set(lock_key, lock_value, nx=True, ex=ttl)
    if not acquired:
        print("Could not acquire lock")
        return
    stop_event = threading.Event()
    renewer = threading.Thread(
        target=renew_lock_periodically,
        args=(redis_client, lock_key, lock_value, ttl, 10, stop_event),
    )
    renewer.start()
    try:
        # Long running work
        time.sleep(120)
    finally:
        stop_event.set()
        renewer.join()
        release_lock(redis_client, lock_key, lock_value)

This strategy is more advanced and must be used carefully. It also assumes a relatively stable connection to Redis.

Using Redlock (Advanced Concept)

You might hear about Redlock, an algorithm proposed for distributed locks with Redis instances.

Conceptually:

The goal is to keep locks safe even if some Redis instances fail or are partitioned.

However:

For this course, focus on:

Common Patterns and Practical Tips

Pattern: Per-Resource Locks

You can create lock keys that are tied to specific resources:

When you process a request that modifies such a resource:

  1. Build the lock key from the resource ID.
  2. Acquire the lock.
  3. Perform the operation.
  4. Release the lock.

Example: prevent two workers from updating the same order status at the same time.

python 
def process_order(order_id):
    lock_key = f"lock:order:{order_id}"
    lock_value = str(uuid.uuid4())
    acquired = redis_client.set(lock_key, lock_value, nx=True, ex=30)
    if not acquired:
        raise Exception("Order is being processed by another worker")
    try:
        # Load order from database
        # Validate and update status
        # Save changes
        pass
    finally:
        release_lock(redis_client, lock_key, lock_value)

Pattern: Try Once vs Wait and Retry

When trying to get a lock, you can:

Example of wait and retry:

python 
import time
def acquire_lock_with_retry(redis_client, key, ttl, wait_timeout=5, retry_interval=0.1):
    value = str(uuid.uuid4())
    end_time = time.time() + wait_timeout
    while time.time() < end_time:
        if redis_client.set(key, value, nx=True, ex=ttl):
            return value
        time.sleep(retry_interval)
    return None

Use cases:

Pattern: Idempotency with Locks

Locks reduce concurrency, but they do not remove all possible double work. For example:

To reduce harm:

Locks and idempotency are often used together for robustness.

Limitations and Failure Scenarios

Even with a correct lock implementation, some limitations remain:

  1. Clock assumptions
    • Many lock algorithms assume some bound on delays and that TTL is larger than worst case processing plus drift.
  2. Redis availability
    • If Redis is down, you may not be able to acquire or release locks.
    • Decide if your system should fail closed (block operations) or fail open (proceed without locks) in that case.
  3. Network partitions
    • If the network splits, some nodes may not reach Redis and see outdated lock state.
    • In high consistency scenarios, this can be critical.

For a basic backend:

Example:

Summary

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3442 / 24h
Impressum / Imprint
Datenschutz / Data Policy