Bugs related to race conditions

Race conditions are among the most frustrating bugs to fix for a developer. The problem is simple to understand: several operations can execute in parallel or be interleaved, and the result depends on the order in which they occur.

The code can work perfectly for months or even years, then one day, with a higher load or a particular timing situation, a bug appears for no apparent reason.

This is what makes these problems particularly difficult: they are often random and almost impossible to reproduce in a development environment. Even with logs, they are often difficult to understand.

Two queries modifying the same data

To understand this problem, let's take a simple example.

An application manages the stock of a product:

Product:
Name: Keyboard
Stock: 1

Two users buy this product almost at the same time.

The logic seems correct:

1. Read the available stock;
2. Check that enough products remain;
3. Decrease the stock;
4. Save the new value.

The first request reads:

Stock = 1

The second request also reads:

Stock = 1

Both therefore think they can complete the purchase.

Then:

User A:
Stock = 1
↓
Buys the product
↓
Stock = 0


User B:
Stock = 1
↓
Buys the product
↓
Stock = 0

Result: two orders have been validated even though only one product was available.

The problem is not an error in the business logic. Each operation works correctly when considered individually. It is the simultaneous execution that creates the problem, which appears when the application performs a separate read and write operation.

Non-atomic read and write operations

This type of problem often appears when an operation is composed of several steps.

For example:

stock = get_stock()

if stock > 0:
    update_stock(stock - 1)

Between the read and the write, another request can modify the value. There is therefore a window during which the state of the data can change.

An atomic operation, on the other hand, is executed as a single indivisible unit.

For example in SQL:

UPDATE products
SET stock = stock - 1
WHERE id = 42
AND stock > 0;

The database directly performs the check and the modification.

We can then check the number of modified rows:

1 row modified → purchase accepted
0 rows modified → out of stock

This approach avoids some race conditions.

SQL locks

Databases provide locking mechanisms to prevent multiple transactions from modifying the same data simultaneously.

For example with PostgreSQL:

BEGIN;

SELECT stock
FROM products
WHERE id = 42
FOR UPDATE;

UPDATE products
SET stock = stock - 1
WHERE id = 42;

COMMIT;

FOR UPDATE asks the database to lock the row. If another transaction tries to modify the same row, it will have to wait until the lock is released. This ensures that other transactions must wait before modifying this row.

However, locks must be used carefully. A lock that is too broad can cause:

• slowdowns;
• blocking;
• deadlocks.

I have already had deadlocks in production, and some of them were difficult to solve without impacting performance.

Transactions

Transactions allow multiple operations that must be processed together to be grouped.

Let's take a payment example:

1. Debit the user's account
2. Create the order
3. Reduce the stock

If step 3 fails, the first two operations should not be kept. A transaction guarantees this behavior:

BEGIN;

UPDATE accounts
SET balance = balance - 100
WHERE id = 1;

INSERT INTO orders (...);

UPDATE products
SET stock = stock - 1
WHERE id = 42;

COMMIT;

If there is a problem:

ROLLBACK;

The database returns to the previous state.

However, a transaction alone does not solve all concurrency problems. Two transactions can sometimes read the same data before modifying it. You must also think about isolation levels and the required locks.

Mutexes

Race conditions are not only related to databases.

In an application using multiple threads or processes, several parts of the program can access the same resource simultaneously.

A mutex (mutual exclusion) ensures that only one part of the program can enter a critical section at a time.

Example:

Thread A:
    Acquire the mutex
    Modify the data
    Release the mutex


Thread B:
    Wait until the mutex is available
    Modify the data

This is widely used in concurrent applications.

However, mutexes can also create problems:

• blocking if a lock is never released;
• performance degradation;
• deadlocks when multiple mutexes are used.

Random bugs that are difficult to reproduce

The main difficulty with race conditions is their unpredictable nature.

A typical bug:

• works in development;
• works in testing;
• appears only in production;
• disappears when logs are added.

Why?

Because modifying the code slightly changes the execution timing. A simple print, a breakpoint, or a slower network connection can change the order of operations and make the problem disappear.

This is what makes these bugs particularly frustrating after several hours or days (true story) of investigation.

How to limit race conditions?

Some best practices help reduce risks:

• use atomic operations when possible and necessary;
• let the database guarantee data integrity;
• use transactions correctly;
• avoid keeping locks for too long;
• think about concurrent access from the design phase;
• test with multiple simultaneous requests;
• monitor concurrency issues in production.

Conclusion

Race conditions are subtle bugs, but with good practices, we can limit them without being able to completely eliminate them before they appear in production.

I did not cover everything in this article. I could also have talked about issues related to distributed applications.

Good luck with your next bugs of this type!