Chapter 21: Atomics & Memory Consistency

Level 3 -- Experience

Lock-free programming with atomic operations and memory ordering -- the deepest level of C concurrency, where even the compiler and CPU are your adversaries.

Atomics are the foundation upon which all higher-level concurrency primitives are built. Mutexes use atomics. Condition variables use atomics. Even the simplest "is this flag set?" check in a multithreaded program requires atomic operations for correctness. This chapter covers the most technically demanding material in the entire book.

---

21.1 The Happened-before Relation

When two threads access the same memory, the order in which those accesses "happen" is not always clear. The compiler may reorder instructions. The CPU may reorder memory operations. Cache hierarchies may delay visibility.

The happened-before relation defines whether one operation is guaranteed to be visible to another:

// C Deep Dive: happens-before
// Thread 1:           Thread 2:
// x = 1;              if (flag == 1)
// flag = 1;               assert(x == 1);  // MAY FAIL

Without proper synchronization, Thread 2 may see flag == 1 but x == 0. The compiler or CPU may reorder the stores, or the cache may not have propagated the write to x by the time Thread 2 reads flag.

C Deep Dive: The happened-before relation is defined by the C standard (and by the Java Memory Model before it). It is a partial ordering: not all operations are ordered relative to each other. Only operations connected by synchronization (mutex lock/unlock, atomic operations with appropriate memory ordering) establish happened-before relationships.

The key insight: program order is not memory order. The code you write is not the order in which the CPU executes or the order in which other threads observe the effects.

Auto's approach. Auto's actor model avoids this entirely. Each actor owns its data. Messages are delivered in order. No shared mutable state means no memory ordering concerns.

---

21.2 C Library Calls That Provide Synchronization

C11 provides atomic operations through <stdatomic.h>:

// C Deep Dive: atomic operations
#include <stdatomic.h>
#include <stdio.h>
#include <threads.h>

static _Atomic int counter = 0;

int worker(void *arg) {
    for (int i = 0; i < 100000; i++) {
        atomic_fetch_add(&counter, 1);
    }
    return 0;
}

int main(void) {
    thrd_t t1, t2;
    thrd_create(&t1, worker, NULL);
    thrd_create(&t2, worker, NULL);
    thrd_join(t1, NULL);
    thrd_join(t2, NULL);

    printf("Counter: %d\n", atomic_load(&counter));  // 200000
    return 0;
}

The _Atomic type qualifier makes operations on that object atomic. The atomic_fetch_add function atomically increments the counter. No mutex needed.

C Deep Dive: Atomic operations are implemented using CPU-level primitives:

• x86: LOCK prefix instructions (e.g., LOCK XADD)
• ARM: LDXR/STXR exclusive load/store pairs
• Some architectures use a global compare-and-swap loop

_Atomic types may have different sizes or alignments than their non-atomic counterparts. _Atomic(int) is always lock-free on mainstream platforms, but _Atomic(struct large) may use internal locks.

Common atomic operations:

Operation	Function	Description
Load	atomic_load(&x)	Read atomically
Store	atomic_store(&x, val)	Write atomically
Fetch-add	atomic_fetch_add(&x, 1)	Add and return old value
Fetch-sub	atomic_fetch_sub(&x, 1)	Subtract and return old
Compare-exchange	atomic_compare_exchange_strong	CAS operation
Exchange	atomic_exchange(&x, val)	Swap atomically
Flag test-and-set	atomic_flag_test_and_set	Simple spinlock primitive