File Descriptor / I/O · Linux userspace / kernel ABI

fcntl locks and the ioctl ABI

Separates ownership and ABI for advisory record locks, open-file-description locks, and device-specific ioctls instead of treating all fd control commands alike.

Series
16 / 38
Build
cc -std=c17 -Wall -Wextra -O2 lock_range.c -o lock_range
Run
./lock_range locked.dat
Kernel
Linux 6.18.37 LTS

fcntl and ioctl both take an fd; what does each control?

fcntl supplies controls understood generically by the VFS, including fd-slot flags, open-file status flags, duplication, and byte-range locks. ioctl descends into per-subsystem command ABIs for terminals, block devices, network devices, and other file types.

Traditional POSIX record locks are process-associated, so closing any fd for the inode in the same process can affect the locks. Linux open-file-description locks are attached to struct file and can make ownership across threads and processes more explicit.

Structure diagram

Figure 1. Overlapping advisory locks on inode byte ranges
0163248648096
Process A · WRLCK
Process B · RDLCK
Process C · WRLCK waiting

The meaning of ioctl arg depends on the command. For a command taking a structure pointer, validate its size, direction, and version separately.

A lock covers [start, start+len), not necessarily the entire file. Nonoverlapping read locks can coexist, but a write lock conflicts with any overlap.

Call path

Figure 2. From userspace code to observable results
fd check file type
fcntl/ioctl command + argument
VFS dispatch generic or f_op
lock/device change subsystem state
return result/errno

This is an overview of two distinct fd-control requests, not one serial call path. Depending on the command, ioctl arg is ignored, a value, or a userspace pointer. Pointer-taking commands require separate checks of size, direction, lifetime, and copying.

Figure 3. Separate fcntl and ioctl entry paths

fcntl(fd, F_SETLK/F_SETLKW, arg)

fcntl syscall fd + security check
do_fcntl cmd switch
F_SETLK / F_SETLKW copy struct flock
fcntl_setlk record-lock handling

ioctl(fd, cmd, arg)

ioctl syscall fd + security check
do_vfs_ioctl common / file-type dispatch
command result return, or -ENOIOCTLCMD fallback
vfs_ioctl .unlocked_ioctl if supported

The upper row follows F_SETLK/F_SETLKW record locking. The lower ioctl row handles common commands or dispatches by file operation. On -ENOIOCTLCMD, the syscall falls back to vfs_ioctl; commands such as FIONREAD can also dispatch directly inside do_vfs_ioctl. The calls may use the same fd, but no shared path runs from record locking into ioctl. Check 32-bit fcntl64 and compat paths separately.

Linux 6.18.37 LTS source locations

Go beyond the glibc function to the files where the syscall implementation meets kernel objects. Each link points to the original file at the same tag.

FileFunction / structureWhat to inspect
fs/fcntl.c do_fcntl(), f_dupfd() Dispatch fcntl commands and distinguish fd/open-file flags
fs/locks.c fcntl_setlk(), locks_lock_inode_wait() Record-lock conflicts, waits, and owners
fs/ioctl.c sys_ioctl(), do_vfs_ioctl() Dispatch generic ioctls and file operations

Complete runnable example

The code below is not pseudocode with explanatory lines omitted. It is a minimal example that can be built and run as a file.

Buildcc -std=c17 -Wall -Wextra -O2 lock_range.c -o lock_range
01#define _GNU_SOURCE
02#include <fcntl.h>
03#include <stdio.h>
04#include <string.h>
05#include <unistd.h>
06
07int main(int argc, char **argv)
08{
09    if (argc != 2)
10        return 2;
11    int fd = open(argv[1], O_RDWR | O_CREAT | O_CLOEXEC, 0644);
12    if (fd < 0)
13        return 1;
14
15    struct flock lock;
16    memset(&lock, 0, sizeof(lock));
17    lock.l_type = F_WRLCK;
18    lock.l_whence = SEEK_SET;
19    lock.l_start = 0;
20    lock.l_len = 16;
21
22    if (fcntl(fd, F_OFD_SETLKW, &lock) < 0) {
23        perror("F_OFD_SETLKW");
24        return 1;
25    }
26    puts("locked bytes [0, 16); press Enter");
27    getchar();
28    lock.l_type = F_UNLCK;
29    int rc = fcntl(fd, F_OFD_SETLK, &lock);
30    close(fd);
31    return rc < 0;
32}

Code notes

Source line 16memset(&lock, 0

Fill the UAPI structure with 0, including padding and unused fields, so architecture-specific garbage values are not passed.

Source line 18lock.l_whence = SEEK_SET

Makes the byte-range origin the beginning of the file. With SEEK_CUR, changes to a shared file offset can affect calculation of the lock range.

Source line 20lock.l_len = 16

Places an advisory write lock only on [0,16). A length of 0 means from the starting point through EOF and any future growth.

Source line 22F_OFD_SETLKW

A Linux OFD lock that waits for a conflict to clear. Ownership belongs to the open file description, not a process PID.

Source line 28lock.l_type = F_UNLCK

Explicitly unlocks the same owner and range. An OFD lock is also released when the last struct file reference closes.

Detailed behavior

01

Advisory locks do not forcibly block I/O

They protect data only when cooperating processes obey the lock protocol. The VFS does not generally stop a process that reads or writes without taking the lock.

A database or log writer must include lock ranges and acquisition order in its application protocol.

02

An ioctl request number is also an ABI

The _IO, _IOR, _IOW, and _IOWR macros encode type, number, direction, and size, but the kernel does not automatically solve pointer validation or structure versioning. Compat tasks may require conversion between 32-bit and 64-bit layouts.

When designing a driver-private ioctl, specify fixed-width types, padding, reserved fields, and size/version rules.

03

First check whether the fd type supports the command

Regular files, pipes, sockets, and ttys do not understand the same commands. ENOTTY can mean not only that an fd is not a terminal, but more generally that the ioctl command is not implemented for it.

Use fstat and subsystem queries to identify the type, then distinguish a command failure caused by an absent device from an ABI mismatch.

Objects and lifetimes

ObjectCreation and releaseValues to inspect
file_lockCreated by an fcntl lock request and attached to the inode lock tree/listowner, range, type
OFD ownerRetained for the lifetime of struct file, with sharing behavior across dup/forkfl_owner, last file ref
ioctl argumentDepending on the command, ignored, a value, or a userspace pointer. Check copying and validation in the implementation of pointer-taking commandssize, alignment, reserved field

Failure conditions and common misconceptions

Observed symptomLikely causesHow to verify
Another writer writes despite the lockThe peer does not follow the advisory protocolInspect the peer's fcntl calls and I/O path
The lock is released earlier than expectedPOSIX-lock semantics were confused with close semanticsDistinguish F_SETLK from F_OFD_SETLK
ioctl ENOTTYThe fd type is wrong or the command is unsupportedfstat, driver version, UAPI header

Verify it yourself

  1. Run the example from two terminals and verify that the second process blocks until the first unlocks.
  2. Change F_OFD_SETLKW to F_SETLKW and compare lock lifetimes when closing a duplicated fd and a separately opened fd.
  3. Call the TIOCGWINSZ ioctl on a terminal fd and determine when redirected stdin produces ENOTTY.
Run./lock_range locked.dat
Tracestrace -e trace=openat,fcntl,ioctl,close ./lock_range locked.dat

Primary sources