Enforce guest RLIMIT_AS and RLIMIT_DATA
XA
2m ago+1499+1499+0123456789012345678901234567890123456789-102-102-012345678901234567890123456789
Intent: brk, mmap, mremap and mprotect hold a guest to its RLIMIT_AS and RLIMIT_DATA, and fail with ENOMEM past them, as Linux does. Follow-up 3 of #448.
Failure fixed: on main, both limits are stored and never checked. With RLIMIT_AS 24 MiB above VmSize, two 16 MiB mmaps both succeed. With RLIMIT_DATA at 24 MiB, sbrk(8 MiB) then sbrk(32 MiB) both succeed. On Linux 6.18 the second call fails with ENOMEM.
This PR:
- Counting:
mem_vm_totalssums two totals from the region table undermmap_lock: all mapped pages (RLIMIT_AS) and private writable pages (RLIMIT_DATA). Each page counts once, and the shim's infra reserve is left out.[stack]counts at the 8188 KiB elfuse maps up front, where Linux counts 132 KiB right after exec, so a limit sized from a Linux run leaves about 8 MiB less room (#492). - mmap and mremap: growth past either limit fails with ENOMEM, as may_expand_vm does. MAP_FIXED is charged only its net growth, as mmap_region does. A MREMAP_FIXED move is checked with its destination still counted, and a refused move leaves the destination mapped, as Linux does since 6.17.
- Error order: when a request exceeds a limit, EBADF, EEXIST and EACCES still come first, in do_mmap's order.
- brk: the page-aligned growth is checked, and the break extent is held to the soft RLIMIT_DATA whatever the heap's protection, as check_data_rlimit does. The data segment that Linux adds to that extent is not tracked (#487).
- mprotect: adding PROT_WRITE to a private mapping fails with ENOMEM at the first mapping that RLIMIT_DATA refuses, as mprotect_fixup does. The part below it is changed in one call, which does not stop at a hole or a refused mapping as Linux does (#489).
- RLIMIT_DATA of 0: a soft limit of 0 falls back to the hard limit, except for brk.
- SysV shm: an attach is mapped shared, as on Linux, so it counts toward RLIMIT_AS only. Its region is named
/SYSV<key> (deleted)as on Linux and carries a serial that no other region has, so it never merges with another mapping. A fork child shares it with the parent when the attach starts on a host page (16 KiB on Apple Silicon) and no other mapping shares its last host page; otherwise the child gets a private copy, as before. shmdt finds the attach from the mappings, as Linux does, so an attach moved with mremap is detached at its new address; it writes back and unmaps only the parts still mapped, and fails with EINVAL when none is there. A shmdt that finds only part of an attach leaves the rest attached until it too is gone, as Linux does. An attach the guest unmapped is detached at the next shmat or shmdt, and fork children hand out serials past the inherited ones. A fork child's shmdt still fails with EINVAL, as it did before (#490). - Left for later: running counters instead of the table scan (#488).
- Known gap: an attach the guest unmaps without
shmdt(bymunmapor aMAP_FIXEDover it) loses the writes made through it, because onlyshmdtcopies back to the segment, and two live attaches of one segment do not see each other's writes. upstream/main copies the same way. The fix, mapping the segment through the shared overlay path, is tracked in #497.
Verified on 7da629d, based on 170b5df, against the Linux 6.18 oracle:
make check-formatpasses.make check: 81 passed, 0 failed, 3 skipped (nslookupskipped, no reachable nameserver).make test-matrix-elfuse-aarch64: 318 passed, 0 failed, 10 skipped.make test-matrix-qemu-aarch64: 292 passed, 0 failed, 35 skipped.- New
test-rlimit-mem: 3 of 3 pass under elfuse and on Linux 6.18.60. Steps 8 and 12 expect the MREMAP_FIXED order of 6.17 and later, so a kernel before 6.17 fails them (seen on 6.12.115); the CI qemu lane still boots such a kernel, see #491. On170b5df, all 3 fail. Each step added in review fails with its fix reverted or on the commit before it, except step 14 ofdata_child(a fork child shares a SysV attach), which170b5dfnever reaches; a probe of it on170b5dfshows the child getting a private copy. - Not tested: the EEXIST and EACCES order on the high-VA MAP_FIXED path, which runs only under Rosetta.