PR #421 - sysprog21/elfuse

A flag elfuse drops or mistranslates on the way to the host fails nothing today unless some test happens to depend on it. This adds a table of syscall flags with the answer Linux gives when each flag is set and when it is clear, a test that asserts both, and fixes for what the table found.

Part of #354 (see the section on it below). No issue exists for the test itself.

The test

tests/flag-matrix.tbl has one row per syscall and flag: the arguments to call it with, and the answer with the flag set and with it clear. scripts/gen-flag-matrix.py joins the rows against dispatch.tbl and writes build/flag-matrix-vectors.h, the same arrangement usbdev-ioctl-departed uses. tests/test-flag-matrix.c runs every row both ways through a raw syscall against a fixture directory it builds fresh for each run. It has no code for any one syscall. Both answers are asserted. Where they differ, a row passes only when the flag is what changes the result; 31 rows end in =, which records that Linux answers the same either way.

The qemu lane runs the binary with strict and is the judge of every recorded answer. 139 rows over 26 syscalls, 278 answers.

A row can carry one of two marks for elfuse:

pending: a difference still to be fixed. The mismatch is printed as KNOWN and does not fail the lane. Once the row passes, the lane fails until the mark is removed.
unsupported:ANSWER: a flag elfuse declines on purpose. ANSWER has to be one Linux documents for a kernel or filesystem without the feature.

One row is pending and one is unsupported at the head; both are described under "Left as they are".

Removing one translation at a time from src/syscall/translate.c fails the rows of the syscalls that use it and no others. Each of these was measured at least once on this branch: O_APPEND, O_NONBLOCK, O_EXCL, O_TRUNC, O_NOFOLLOW, O_CREAT, O_DIRECTORY, the O_PATH branch, O_SYNC/O_DSYNC, AT_REMOVEDIR, AT_SYMLINK_FOLLOW, and AT_SYMLINK_NOFOLLOW in both translators. Removing O_CLOEXEC fails nothing, because a guest cannot see whether the host open carried it: a guest execve stays in the host process and the fork helper is spawned with POSIX_SPAWN_CLOEXEC_DEFAULT. The table says so beside that row.

check-syscall-coverage.py fails when one of the 26 syscalls loses all of its rows, and when the table gains a syscall its list does not name.

The commits

30 commits. Five add the test or rows that already pass (3db8916, 36c20bc, ec2de4d, 800215c, 81abd03), one moves code with no change in behavior (cffa733), one is the audit (a8c562e), 21 change elfuse, and two correct a test and a row after review (476d4e4, 2c747be). Each fix brings the rows that hold it, so the test passes at every commit and no row is added as pending to be unmarked later.

	Before	After	Commit
`epoll_create1`, `inotify_init1`, `memfd_create`, `accept4`	an undefined flag bit was accepted	`EINVAL`	d77b3b5
empty path, no `AT_EMPTY_PATH`: `newfstatat`, `statx`, `fchownat`, `fchmodat`, `openat`, `renameat2`, and the calls that read a path through `read_translated_path`	`ENOTDIR` when dirfd is not a directory	`ENOENT`	b99ceb4
`statx` with both sync flags	0	`EINVAL`	382b954
`unlink` of a directory	`EPERM`	`EISDIR`, except the three cases where Linux also says `EPERM`	aad176d
`pipe2(O_DIRECT)`	`F_GETFL` on the write end did not report it	reports it, as `F_SETFL` already did	23dca1e
`openat` with `O_TMPFILE\|O_RDONLY`	opened the directory and returned it	`EINVAL`	c6919ce
`mmap` with neither `MAP_SHARED` nor `MAP_PRIVATE`, or anonymous `MAP_SHARED_VALIDATE`	mapped	`EINVAL`	fd3da6a
`mmap` under `MAP_SHARED_VALIDATE` with `MAP_SYNC`, `MAP_FIXED_NOREPLACE`, or any bit from 31 up	mapped	`EOPNOTSUPP`	fd3da6a
`mmap` with a bad descriptor and a bad type	`EINVAL`	`EBADF`	fd3da6a
`mprotect` with an undefined bit, `PROT_GROWSUP`, or `PROT_GROWSDOWN` off the stack	0	`EINVAL`	c3b78d9
`mprotect` with `PROT_BTI` or `PROT_MTE`	0	`EINVAL`, as on a CPU without the feature; the guest is shown neither	c3b78d9
`mprotect` with `PROT_GROWSDOWN` on the stack	changed only the range given	extends down to the first page above the guard	c3b78d9
a fresh non-fixed `PROT_READ` mapping	a store succeeded	the store faults	9f7e0ff
anonymous `mmap` with an unaligned offset	mapped	`EINVAL`	a879ab9
`mmap` with `MAP_HUGETLB` on a file other than an `MFD_HUGETLB` memfd	mapped	`EINVAL`	3c14a5f
`mmap` or `mprotect` with `PROT_SEM` alone	readable	`PROT_NONE`	d8daa8c
`memfd_create` with `MFD_NOEXEC_SEAL`, and the mode of any memfd	no seal; mode 0600	`F_SEAL_EXEC`, held against `fchmod`, `fchmodat2` with `AT_EMPTY_PATH` and a chmod of `/proc/self/fd/N`; mode 0777 or 0666	a9f75f9
`faccessat2` and `utimensat` with `AT_EMPTY_PATH`	`EINVAL`, or the host's error	act on the descriptor; a descriptor on a `/proc` file is judged by the mode its path reports	9b47ac6
`faccessat2` with `AT_EMPTY_PATH` on a `/proc` file whose name is gone, such as `/proc/self/fdinfo/N` after N is closed	`ENOENT`	answered for the open file	ceb4104
`linkat` with `AT_EMPTY_PATH`	`EINVAL`	links the file the descriptor names; `EXDEV` for a descriptor on a `/proc` file	41b1d15
`linkat` with an empty old or new path and no `AT_EMPTY_PATH`	`ENOTDIR` when dirfd is not a directory	`ENOENT`	41b1d15
`readlinkat(fd, "")` on an `O_PATH\|O_NOFOLLOW` descriptor of a symlink	`ENOTDIR`	the link's target; `ENOENT` for anything that is not a link	31ef112
`clone` with `CLONE_THREAD` and no `CLONE_SIGHAND`, or `CLONE_SIGHAND` and no `CLONE_VM`	started a thread on the given stack, or forked	`EINVAL`	2e728fc
a pidfd	not close-on-exec in the guest's view	close-on-exec	8004e78
fork-style `clone` with `CLONE_PARENT_SETTID` or `CLONE_PIDFD`	neither slot was written	both are	5fe4e1b
`CLONE_PIDFD` with a slot that cannot be written, or no descriptor left	returned the pid and the child ran	`EFAULT` or `EMFILE`, and no child	5fe4e1b
`clone3` with `CLONE_PIDFD\|CLONE_PARENT_SETTID` and `pidfd == parent_tid`	both stores went to one word	`EINVAL`	5fe4e1b
`CLONE_VM` child with `CLONE_PARENT_SETTID`	the child read the slot's old contents	the child reads its tid	b469621

Five commits that deserve a closer look

9f7e0ff, a fresh PROT_READ mapping. A non-fixed mmap extended its 2 MiB block read-write and then gave the requested range the same permission whatever prot said. The comment on mprotect_same_prot_fast_path_safe records that this was known. The range now gets the permission asked for. This touches every non-fixed mmap, so I measured a map, touch-every-page, unmap loop three times before and three times after, in ns per iteration:

	before	after
anonymous 64 KiB	8250 to 9429	7674 to 8431
anonymous 8 MiB	91930 to 102307	86773 to 98790
file 64 KiB	10995 to 12451	12054 to 13738
file 8 MiB	561840 to 623260	545910 to 646010

Only the 64 KiB file case moves, by about 1 microsecond, and the runs overlap. If the read-write block was kept for a reason I did not see, this commit reverts on its own. It may also conflict with the open mmap pull requests.

c6919ce, O_TMPFILE. It stays unsupported, now by decision. macOS cannot link a file that has no name, so a file opened this way could never be published with linkat. open(2) documents EISDIR for a kernel without O_TMPFILE, callers fall back to a named temporary file on it, and that is what elfuse already answered by accident. An open that succeeded would turn that fallback into a failure with no answer. The row is marked unsupported:err:EISDIR.

9b47ac6, faccessat2 on a descriptor. macOS has no faccessat that takes a descriptor. The call asks fgetattrlist for ATTR_CMN_USERACCESS, which is the host's own answer for an open file, mode bits and ACL together, and needs no name. The answer is therefore the same for a file that was renamed or unlinked. A descriptor the host returns no such attribute for, such as a pipe, is judged by its mode bits against the credentials the host process runs with.

41b1d15, linkat on a descriptor. macOS has no linkat that takes a descriptor either, so the link goes through the name F_GETPATH returns. Used as a path, that name loses a race with rename: with one thread exchanging the file's directory through RENAME_EXCHANGE, 249 to 404 of 20000 calls linked a file the descriptor does not name, and a directory exchanged for a symlink leads out of the tree the guest is held in. The directory is opened once with O_NOFOLLOW_ANY and held, the entry in it is compared with the descriptor's file, and the host call acts on that directory and entry name. The same program links the wrong file 0 times in 20000. A file with no name left is ENOENT, which is also Linux's answer for a descriptor whose link count is zero.

5fe4e1b, sys_clone. sys_clone becomes a thin front and its old body becomes clone_dispatch. The INVENTORY entry in check-eintr-contract.py and the NOLINT marker for the function size follow the body. The pidfd is made inside clone_dispatch, at the point each path can still call the child off: after the fd table has gone to a forked child and before it is admitted, or before a thread or CLONE_VM child exists. In the second case the entry is found again by its generation, because the guest can close the fd number and get it back for another pidfd in between, and when the task then fails to start the descriptor is closed only if its fd generation still matches.

#354

#354 is assigned to @alanhc. Commit b99ceb4 changes stat_at_path in the way that issue sketches, because the same answer came out of the table for newfstatat, statx, fchownat and faccessat2. With this branch, the issue's table reads ENOENT in every row for fstatat(fd, "", &st, 0) and statx(fd, "", 0, ...): pipe, socketpair, eventfd, regular file, directory, AT_FDCWD and a closed descriptor, with --sysroot <empty dir> and without.

The issue's open question was whether a closed descriptor answers ENOENT or EBADF. On Linux 6.12.103 under qemu it answers ENOENT, and EBADF only with AT_EMPTY_PATH. elfuse now gives the same two answers.

The third case in the issue, a NULL pathname to statx with AT_EMPTY_PATH, is not touched here. If @alanhc would rather land their own change for the first two, b99ceb4 can come out of this branch.

Left as they are

Packet mode on a pipe. pipe2 O_DIRECT@packets is the one pending row: two writes then one read return both writes, where Linux returns the first. The host pipe is a byte stream. An AF_UNIX SOCK_DGRAM socketpair keeps the boundaries (SOCK_SEQPACKET is not supported on macOS), but end-of-file, EPIPE, a full pipe, poll after end-of-file and fstat all need translating, and F_SETFL(O_DIRECT) on a pipe that already exists cannot be honored at all. That is planned as a separate change. docs/testing.md has the note.
F_SETFL(O_DIRECT) on a pipe still records the bit without packet semantics, which tests/test-fcntl-flags.c asserts.
MFD_HUGETLB is accepted and backed by ordinary pages, and such a memfd takes MAP_HUGETLB. AT_EACCESS and O_NOCTTY have acceptance-only rows.
mprotect PROT_BTI is EINVAL, as Linux answers on a CPU without BTI: elfuse shows its guest ID_AA64PFR1_EL1 as 0. It has no row, because Linux's answer follows the qemu CPU, the host's under HVF (which shows BTI where the host has it) and cortex-a72 under TCG (which has none). PROT_MTE has a row: neither CPU has MTE.
readlinkat(fd, "") on an O_PATH descriptor of a /proc link, such as /proc/self/exe, is ENOENT: those descriptors carry no /proc stamp, so the link has nothing to be read from. The same call on a symlink whose last name is gone is ENOENT too, where Linux still reads it: macOS reads a link only by name.
A pidfd closed by number can race a reuse of that number before its cleanup hook runs, because pidfd_cleanup finds its entry by fd number. Every pidfd close has that window, not only the one clone makes on a failed start; closing it means passing the slot generation to the cleanup hooks of every fd type.
Not in the table yet: readlinkat (its empty-path cases are in test-file-ops), pidfd_open, openat2, mremap, msync, madvise, the splice family, the send/recv flags, epoll_ctl, clone3.

Checked by hand, with no test in the tree

aad176d: unlink on five directories set up on the host, because a guest cannot make an immutable directory or one owned by another user. Plain directory EISDIR; directory with uchg EPERM; own directory in a sticky parent EISDIR; directory in a uchg parent EPERM; root's directory in /tmp EPERM.
a9f75f9: on a MFD_NOEXEC_SEAL memfd, fchmod to 0777 is EPERM and to 0600 succeeds, and adding F_SEAL_EXEC|F_SEAL_SEAL reads back 0x21. The same program gives the same output on Linux 6.12.103.
c3b78d9: only the read-only case of PROT_GROWSDOWN on the stack has a row. The executable-stack path glibc uses was not run.
9b47ac6: on a 0600 file with an ACL that denies write, faccessat2 with AT_EMPTY_PATH answers EACCES for W_OK before and after the unlink.
5fe4e1b: EFAULT for a pidfd slot at address 8 and at 0 through clone, clone3 and CLONE_VM|CLONE_VFORK, EMFILE with RLIMIT_NOFILE at the next free descriptor, and no child to reap after any of them. Same answers on Linux 6.12. Only the address 8 case through clone has a row.
41b1d15: the two descriptor cases in test-linkat-symlink-fallback need a Case-sensitive HFS+ image, where they return EOPNOTSUPP without the fallback.

Validation

Rebased on main at 8612033. At 2c747be every CI check passes. At a8c562e, before the last three commits:

make check, make check-format, make lint: exit 0.
tests/test-matrix.sh elfuse-aarch64: 300 passed, 0 failed, 10 skipped.
tests/test-matrix.sh qemu-aarch64: 278 passed, 0 failed, 32 skipped. This is the run that holds all 280 recorded answers against Linux.
scripts/check-commit-log.sh over the 30 commits (git rev-list 8612033..HEAD | scripts/check-commit-log.sh): exit 0.

At each of the 30 commits: make elfuse builds, test-flag-matrix passes under elfuse, and check-flag-matrix, check-syscall-coverage, check-eintr-contract, check-lock-order, check-atomics, check-ascii, .ci/check-format.sh and .ci/check-commentflow.sh pass.

Not run at this head: make verify, make test-rosetta-all, tests/test-matrix.sh elfuse-x86_64, make check-tsan, make check-asan, make check-contracts. The series was rewritten from 54 commits to 26 after the last runs of the first three, with the same final tree apart from the rebase, and has since gained the review fixes and one commit.

This is larger than most pull requests here. If it is easier to review in pieces, the test and its rows (3db8916, 36c20bc, ec2de4d, 800215c, 81abd03) can go first and the fixes follow by area, each with its own rows.

Summary by cubic

Adds a table of syscall flags with the answer Linux gives when each is set and when clear, a test that asserts both, and fixes for the divergences the table found. A flag elfuse dropped or mistranslated used to fail nothing unless a test happened to depend on it.

The test

tests/flag-matrix.tbl has one row per syscall and flag; scripts/gen-flag-matrix.py joins the rows against dispatch.tbl and generates vectors that tests/test-flag-matrix.c runs twice — flag set and clear — against a fresh fixture directory. Both answers are asserted, so a row passes only when the flag is what changes the result; 26 rows record that Linux answers the same either way.
pending marks a divergence still to fix (reported as KNOWN, fails once fixed); unsupported:ANSWER marks a declined flag carrying a documented Linux answer for a kernel without the feature. 136 rows over 26 syscalls, all answers hold on Linux 6.12 in the qemu lane.
check-syscall-coverage.py fails when one of the 26 syscalls loses all of its rows or a row names a syscall dispatch.tbl no longer serves.

The fixes

epoll_create1, inotify_init1, memfd_create, and accept4 now refuse undefined flag bits with EINVAL. unlink of a directory answers EISDIR except where Linux also answers EPERM; statx refuses both sync flags; pipe2(O_DIRECT) reports O_DIRECT on the write end; O_TMPFILE stays unsupported, now by decision.
An empty path without AT_EMPTY_PATH now answers ENOENT instead of the host's ENOTDIR (addressing the first cases of #354); linkat, faccessat2, and utimensat act on the descriptor when AT_EMPTY_PATH is set, and such a call now holds its directory open with O_NOFOLLOW_ANY so a rename or symlink exchange cannot redirect it outside the guest's tree. linkat on a symlink's O_PATH descriptor falls back to recreating the symlink on hosts that refuse the hard link, with the test asserting the target exactly. readlinkat with an empty path reads the symlink an O_PATH descriptor is open on.
faccessat2 on a descriptor uses the host's fgetattrlist access check, ACLs included, so the answer no longer depends on whether the file still has a name; for a synthetic file that has left its tree, the host fd behind the descriptor answers instead of ENOENT.
mmap and mprotect follow Linux on mapping types, descriptor lookup order, alignment, and protection bits; a fresh PROT_READ mapping is actually read-only. Under MAP_SHARED_VALIDATE, bits at or above 31 and MAP_FIXED_NOREPLACE are EOPNOTSUPP, MAP_HUGETLB on a file mapping is EINVAL, and PROT_SEM alone is PROT_NONE.
clone refuses flag pairs Linux refuses, writes the CLONE_PARENT_SETTID slot, publishes CLONE_PIDFD; pidfds are close-on-exec; CLONE_PIDFD with an unwritable slot or no descriptor left is EFAULT or EMFILE with no child, the slot is probed before a child is spawned, and a CLONE_VM child reads its tid in the slot. clone3 refuses CLONE_PIDFD|CLONE_PARENT_SETTID naming the same word. A pidfd is tracked by an internal generation rather than a guest fd number, so descriptor reuse cannot point an old entry at a new task.
memfd_create honors MFD_NOEXEC_SEAL with F_SEAL_EXEC, which now holds against every chmod spelling, and gives a memfd the mode Linux reports.
The only measurable cost is about 1 µs on a 64 KiB file mapping; the other measured map loops are unchanged or faster.
Packet mode on an O_DIRECT pipe stays pending by decision — the host pipe is a byte stream — with a pointer to docs/testing.md. MFD_HUGETLB, PROT_MTE, AT_EACCESS, and O_NOCTTY have acceptance-only rows, and O_CLOEXEC on the host open is invisible to a guest. The PROT_BTI acceptance row was dropped because the qemu lane's answer depends on whether the host CPU has BTI; elfuse still refuses the bit, as Linux does on a CPU without it. test-fd-lifecycle now requires a MAP_HUGETLB map of an MFD_HUGETLB memfd to succeed or answer ENOMEM, never EINVAL.