A Fresh Process Is Not a Cold System: Measuring import torch with Rust and wait4

TL;DR

This post continues Part 1, following coldtrace as it moves from recording wall time and exit status to collecting Linux child-resource information through the wait4 API.

The richer data reveals that Torch's first-run penalty changes shape depending on the surrounding system state. One first run was dominated by kernel CPU time, while a post-reboot run produced major page faults and extensive blocking. In short, a fresh process is not necessarily running in a cold system.

Starting Point

At the end of Part 1, coldtrace was a basic Rust process runner. It could spawn a command, redirect its stdout and stderr, measure wall-clock duration, and record whether the child exited normally or was terminated by a signal.

I collected this information using Rust's std::process module. It provided a solid starting point, but wall time and exit status alone revealed little about what the child process did while it was running.

Linux exposes richer process accounting through a structure called rusage. It includes measurements such as user and system CPU time, page faults, context switches, and peak resident memory.

To retrieve that information for a specific child, I needed to understand the child-process lifecycle. When the Python process exits, the kernel retains a small termination record containing its exit status and resource usage. The parent process, coldtrace, must eventually wait for the child and consume that record. This operation is called reaping, and the record can only be consumed once.

Unsafe Rust

To collect child resource usage, I needed to move below Rust's standard Child::wait() API and call libc::wait4. The libc crate exposes low-level C-compatible operating-system interfaces. Its wait4 function accepts pointers where it can write the child's termination status and resource usage.

Calling wait4 requires an unsafe block. That does not mean the operation is necessarily dangerous or incorrect. It means Rust cannot verify all of its safety requirements. In this case, the compiler cannot prove that the raw pointers refer to valid, writable memory, remain alive during the call, or are only read after wait4 initializes them. By writing unsafe, I was taking responsibility for proving those conditions myself.

The parent calls wait4 after spawning the Python process. If Python is still running, the parent blocks. When the child exits, the kernel retains its termination information. A successful wait4 call writes that status and the child's rusage into memory supplied by the parent, then reaps the child.

The status output was straightforward because I could initialize a valid integer to zero and pass a mutable reference to it. The rusage output was different: I needed storage for a libc::rusage, but no valid rusage existed until wait4 successfully filled that storage.

Rust normally assumes that every value of type T is already valid according to the rules of T, even inside unsafe code. Unsafe code does not disable those invariants. This is why Rust provides MaybeUninit<T>, a wrapper representing correctly sized and aligned storage that may not yet contain a valid T.

I created a MaybeUninit<libc::rusage>, passed its raw mutable pointer to wait4, and checked the return value. Only after wait4 succeeded did I call assume_init(). Despite its name, assume_init() does not initialize the memory. It tells Rust that an earlier operation already initialized it and that the bytes may now be treated as a real libc::rusage.

The lifecycle would then look something like:

reserve MaybeUninit<rusage> storage
    -> spawn Python
    -> call wait4 with the storage pointer
    -> check whether wait4 succeeded
    -> call assume_init
    -> read the rusage fields

Safety Proof

But MaybeUninit only gave me the way to represent the output storage. It did not make the FFI call safe by itself. So the next question was: what exactly do I need to prove before Rust could trust the result?

It ended up being local and surprisingly concrete. wait_status was an initialized integer, while raw_rusage owned correctly sized and aligned storage for a libc::rusage. Both variables remained alive for the entire synchronous wait4 call, and nothing else accessed them while the kernel was writing through their pointers.

After that call, I checked its return value before reading either output. With options set to 0, a successful call returns the waited-for child PID; and -1 indicates failure. If it failed, I returned the OS error and never treated the contents of the MaybeUninit storage as a real rusage by exiting right there.

Only on the success paths did I call assume_init(). This is per the wait4 contract: when the call succeeds and receives a non-null rusage pointer, it fills the pointed-to structure with the child's resource accounting. And then from there I removed the old Child::wait() call since not only was it redundant, but also that wait() and wait4 both consume the child's retained exit record.

The unsafe block was not an escape from Rust's rules. It was a boundary where I had to uphold the rules that Rust could not verify for me.

Measurements Added

Before diving into the results, it is worth quickly reviewing the measurements I added: user and system CPU time, peak resident set size (RSS), minor and major page faults, and voluntary and involuntary context switches. Linux also maintains counters for block input and output operations, but this version does not record them. That omission was an oversight, and I plan to include them in a future iteration. The remaining historical rusage fields are not maintained by Linux, so I omitted them rather than recording misleading zeroes.

Experiment Harness

I had AI write a quick Python script that became my temporary test harness. It runs the release build of coldtrace five times against the same import torch workload, preserves each run's individual artifacts, and writes a batch summary containing the minimum, maximum, mean, and median for every collected metric. It also records the Python interpreter, kernel, boot ID, and VM uptime, and rejects failed workloads rather than including them as benchmark data. Comically, this results in Python launching Rust, which then launches Python again.

Of course, this led to quite a comical nested structure:

Python experiment harness
`-- release coldtrace (Rust)
    `-- virtual-environment Python importing Torch

Side Note: Failure Discovery

After rebooting my Lima VM, I ran the harness and obtained results that were suspiciously fast. On closer inspection, Python had started successfully, but import torch had failed with ModuleNotFoundError because I had forgotten to reactivate my virtual environment.

coldtrace behaved correctly: it preserved Python's exit code and stderr while treating the failed child process as a valid trace result. The harness, however, initially treated coldtrace's successful execution as proof that the workload had also succeeded, so it aggregated the failed runs as benchmark data. The preserved failed run record and five-run failure summary show the distinction.

I updated the harness to use the same Python interpreter that launched it, verify that Torch is available without importing it, and reject any workload that exits nonzero or is terminated by a signal. I would like to think that we've all been there.

Cold-Guest Result

After fixing the harness issue, I rebooted the Lima VM again, activated the virtual environment, and immediately ran another five-import batch. The harness started roughly 95 seconds after the guest booted.

The table is backed by the complete pre-reboot batch, Missing-Torch batch, and post-reboot batch summaries. The successful first-run rows also link to their original pre-reboot and post-reboot records.

The reboot also changed the guest kernel from 6.8.0-124-generic to 6.8.0-136-generic, so comparisons between batches include that additional variable. Comparisons between the first and warm runs within each batch still used the same kernel.

The two warm averages were remarkably consistent at approximately 560 milliseconds. The first runs were much slower, but their resource profiles were different. The pre-reboot first run spent far more CPU time in the kernel without recording any major page faults. The post-reboot first run instead showed 668 major faults and more than 2,000 voluntary context switches.

It is worth pausing to define those measurements. According to the Linux getrusage(2) manual, a minor page fault is resolved without I/O. The required page may already be available in memory, or the kernel may resolve the fault through mechanisms such as zero-filled pages or copy-on-write. A major page fault requires I/O before execution can continue. Neither type necessarily represents an application error.

The post-reboot result therefore provides evidence of guest-visible I/O and blocking. It does not prove that the Mac's physical storage was cold, because macOS may still have cached the VM's backing data. From the Linux guest's perspective, however, those page faults required I/O.

This is where wait4 provided meaningful attribution beyond wall time. Wall time alone showed that the first import took 938 milliseconds compared with a warm average of 562 milliseconds, making it roughly 67 percent slower. The resource data showed that this was accompanied by major faults and a dramatic increase in voluntary context switches.

Every successful measurement used a fresh Python process. What changed was the surrounding system state. A new process is therefore not necessarily a cold process in every meaningful sense: it may still benefit from warm filesystem pages, metadata, and other kernel state left behind by earlier executions.

Peak RSS remained close to 230 MiB, and every valid workload exited successfully. This gave me confidence that the cold and warm measurements represented the same completed Torch import rather than different execution paths.

Conclusion

This exercise made several systems concepts concrete. Setting aside the Rust learning value, the most important lesson was that a fresh process is not equivalent to cold storage, a freshly booted guest, or a stable warm system. Each represents a different initial state and can produce a different performance profile for the same command.

That is why meaningful systems benchmarking requires clearly defining what "cold" means, preserving the surrounding environment, separating first-run behavior from the warm regime, and repeating experiments across multiple boots. A cold-start time is not a single universal property of import torch; it is a measurement produced under a particular set of system conditions.

AI Assistance

I used AI coding tools while building Coldtrace, including for implementation suggestions, debugging, code review, and explaining unfamiliar APIs. I also used AI assistance to organize and edit this post.

I reviewed and modified the code I kept, ran the experiments myself, inspected the resulting artifacts, and am responsible for the measurements, interpretations, and any mistakes in the project or this post.