Keep guest rlimits out of the host process

XA
14h ago
+949-267

Intent: a guest setrlimit or prlimit64 changes a guest table and never reaches host setrlimit. This PR stores the limits; it does not enforce them.

Failure fixed: on main, every limit but NOFILE goes to host setrlimit, so the limit lands on elfuse itself:

  • A guest that sets RLIMIT_FSIZE to 4 KiB and writes 8 KiB kills elfuse with SIGXFSZ (exit 153), even when the guest ignores SIGXFSZ. On Linux the write returns 4096 and the next write fails with EFBIG.
  • A guest that sets RLIMIT_CPU to 1 s and spins stalls the vCPU until the watchdog reports "vCPU execution timed out after 10s", and elfuse then hangs. On Linux the guest gets SIGXCPU after 1 s.
  • setrlimit(RLIMIT_AS, 64 MiB) and setrlimit(RLIMIT_DATA, 1 MiB) fail with EINVAL from the host. Linux accepts both.
  • RLIMIT_RSS overwrites RLIMIT_AS, RLIMIT_LOCKS through RLIMIT_RTTIME are EINVAL, and only NOFILE crosses fork.

So the enforcement main has for FSIZE and CPU is already wrong: it acts on the host process. This PR removes it and does not yet put the Linux enforcement in its place.

This PR: all sixteen limits live in a guest table, seeded from host getrlimit at boot, carried in the fork IPC header and kept across execve. RSS starts unlimited as on Linux, because macOS defines RLIMIT_RSS as RLIMIT_AS. /proc/self/limits prints the table. NOFILE stays enforced as before. prlimit64 reaches only the caller's own process (pid 0, its pid, or one of its thread ids). A clone(CLONE_VM) child without CLONE_THREAD is a process of its own, so its pid is ESRCH unless the child names itself. Any other pid is ESRCH. For a pid that doesn't exist, that matches Linux. For a live fork-family process or VM-clone child, Linux would act on that process; that is follow-up 4 and #454. On main, the pid is ignored, so prlimit64(child, ...) changes the caller's own limits.

Follow-ups, each building on this table:

  1. RLIMIT_FSIZE on write, truncate, fallocate, sendfile, copy_file_range and splice, with SIGXFSZ to the thread that wrote.
  2. RLIMIT_CPU from sampled process CPU time, with SIGXCPU.
  3. RLIMIT_AS and RLIMIT_DATA on brk, mmap, mremap and mprotect.
  4. prlimit64 on another fork-family pid, with the kernel's credential check. A VM-clone child's own table (#454) overlaps with it.
  5. RLIMIT_STACK, by growing the stack on demand, which also sizes what execve accepts as arguments.

Depends on: none.

Review focus:

  • The fork IPC header now carries the table in place of the two NOFILE fields, so the protocol magic moves from ELFR to ELFS (src/runtime/fork-state.h).
  • The first commit makes thread_alloc set is_vm_clone, parent_tid and exit_signal before it publishes the entry. sys_clone_vm used to set them after thread_lock was dropped, so a reader holding the lock could take the child for a thread. The prlimit64 pid check reads them under thread_lock.
  • sys_rlimit_cur reads a slot with no lock; writers hold rlimit_lock (src/syscall/sys.c).
  • rlimit_exchange follows do_prlimit: raising a hard limit takes guest euid 0, and the NOFILE hard limit cannot rise above the fd table size.
  • The host stack hard limit at macOS's MAXSSIZ cap seeds the guest hard limit as unlimited, since that cap bounds host threads, not guest memory. Only the macOS default soft limit (8 MiB less 16 KiB) is rounded up to 8 MiB; a soft or hard limit the user lowered reaches the guest as set. test-rlimit-seed sets host limits with ulimit and reads the guest's /proc/self/limits.
  • ELFUSE_HOST_RLIMIT_CHECK is a test hook: with it set, a process exits 86 when a host limit moved during the run. test-rlimit-unchanged-host runs a guest under it.

Verified on ae9252b, rebased onto 173b23b: make check-format and make check pass; make test-matrix-elfuse-aarch64 gives 312 passed, 0 failed, 10 skipped; make test-matrix-qemu-aarch64 gives 286 passed, 1 failed, 35 skipped. The failure is test-pidfd-fork on the qemu reference kernel, which this PR does not touch; run alone there it failed 4 of 6 times ("child checks failed").

aceb8f9 changes only tests/test-rlimit-seed.sh: its host AS limit goes from 1.5 TiB to 6 TiB, and a failing case reports its exit status. On the macOS 26.4.1 CI runner the lane exited 139 at 1.5 TiB on both runs of ae9252b. Draft PR #457 traced that to two crashes in the elfuse host process during boot under a host AS limit, #458 at 1 TiB and #459 at 1.5 TiB, which this PR leaves to those issues. At 6 TiB the runner boots normally and the lane passes.

Summary by cubic

Guest setrlimit and prlimit64 now write a guest rlimit table and never call host setrlimit, so guest limits no longer reach elfuse itself. The table is seeded from host getrlimit at boot, rides the fork IPC header (protocol magic moves from ELFR to ELFS) so it crosses fork, and survives execve. Limits are stored only; enforcement beyond NOFILE is follow-up.

Bug fixes

  • A write past RLIMIT_FSIZE used to kill elfuse with SIGXFSZ even when the guest ignored it; a finite RLIMIT_CPU used to stall the vCPU until the watchdog fired.
  • RLIMIT_AS and RLIMIT_DATA used to fail with EINVAL from the host; all sixteen resources are now accepted, and RLIMIT_RSS no longer overwrites RLIMIT_AS.
  • prlimit64 on a pid other than 0, the caller, or one of its thread ids used to change the caller's limits; it now returns ESRCH.
  • A clone(CLONE_VM) child used to be taken for a thread of the caller's group because its own-process fields were set after the slot was published; thread_alloc now sets them under thread_lock.
  • The guest stack default now reads as Linux's 8 MiB, a hard limit at macOS's MAXSSIZ cap reads as unlimited (the cap bounds host threads, not guest memory), and a stack limit the user lowered passes through.
  • ELFUSE_HOST_RLIMIT_CHECK makes an elfuse process exit 86 when a host limit moved during its run, and test-rlimit-unchanged-host runs the limits test under it.
  • test-rlimit-seed now starts elfuse under a 6 TiB host AS limit (1.5 TiB crashed at boot on the macOS 26 runner) and reports the exit status of a failed elfuse run.