DPAS Lab Guide

Hands-on Performance Comparison of NVMe I/O Completion Modes

2026-1 Systems Technology
한국어

Prerequisites

You must have already completed the platform-specific QEMU installation and Ubuntu VM setup.

Required Environment Ubuntu 22.04 VM + mainline kernel 5.18 (uname -r5.18.0-051800-generic)
Why Kernel 5.18? This lab uses synchronous Direct I/O polling via preadv2(RWF_HIPRI). This feature was removed starting from Linux kernel 5.19, so 5.18 is the last kernel that supports it. Since the QEMU Guest VM runs kernel 5.18, the polling lab works regardless of the host OS kernel version.

I/O Completion Background Reading → — History of Interrupt/Polling, FAST '26 latest research, the future of io_uring and polling

Part 1: DPAS Kernel Installation

Install the DPAS-patched kernel using the provided deb packages.

What is a deb package? A .deb file is a software installation package used in Ubuntu/Debian-based Linux distributions. It serves the same role as .msi installers on Windows or .pkg on macOS. Install with sudo dpkg -i file.deb, which automatically places system files such as kernel images and headers in the correct locations.
Architecture Selection — Based on CPU Architecture, Not Host OS Kernel packages are selected based on the CPU architecture the VM runs on, regardless of the host operating system (macOS/Windows/Linux). Run uname -m inside the Guest VM to check the architecture.
Host Environmentuname -m OutputRequired Package
Apple Silicon Mac (M1/M2/M3/M4)aarch64ARM64
Intel Mac / Linux PC / Windows (WSL2)x86_64x86_64

Download Kernel Packages

Download the package matching your VM's CPU architecture.

ARM64 Kernel Image (16MB) ARM64 Headers (8MB)
x86_64 Kernel Image (12MB) x86_64 Headers (8MB)

1-1. Transfer Packages

Transfer the downloaded deb files from the host to the VM.

# Run on the host terminal
scp -P 2222 linux-image-5.18.0-dpas_*.deb <username>@localhost:~/
scp -P 2222 linux-headers-5.18.0-dpas_*.deb <username>@localhost:~/

1-2. Install and Reboot

# Run inside the VM
sudo dpkg -i ~/linux-image-5.18.0-dpas_*.deb ~/linux-headers-5.18.0-dpas_*.deb
sudo reboot

1-3. Verify Installation

uname -r
# 5.18.0-dpas
If the boot kernel did not change GRUB may boot into the previous kernel. Select the DPAS kernel with the following command and reboot:
sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 5.18.0-dpas"
sudo reboot

Part 2: Lab — Performance Comparison by Mode

2-1. Benchmark Scripts

Create the following 4 scripts inside the VM. Usage: ./script.sh <device> <cpu> <numjobs>

fioint.sh — Interrupt Mode

#!/bin/bash
sudo modprobe -r nvme; sudo modprobe nvme poll_queues=0
fio --filename=/dev/$1 --size=100m --direct=1 --bs=4k --ioengine=pvsync2 \
    --iodepth=1 --rw=randread --runtime=10 --numjobs=$3 --time_based \
    --group_reporting --name=test --eta-newline=1 --cpus_allowed=$2 \
    --nice=-10 --prioclass=2 --prio=0

fiocp.sh — Continuous Polling Mode

#!/bin/bash
sudo modprobe -r nvme; sudo modprobe nvme poll_queues=2
fio --filename=/dev/$1 --ramp_time=3 --size=100m --direct=1 --bs=4k \
    --ioengine=pvsync2 --iodepth=1 --rw=randread --runtime=10 \
    --numjobs=$3 --time_based --group_reporting --name=test \
    --eta-newline=1 --cpus_allowed=$2 --nice=-10 --prioclass=2 \
    --prio=0 --hipri

fiopas.sh — PAS (Polling after Sleep) Mode

#!/bin/bash
sudo modprobe -r nvme; sudo modprobe nvme poll_queues=2
echo 0 > /sys/block/$1/queue/io_poll_delay
echo 1 > /sys/block/$1/queue/pas_enabled
echo 1 > /sys/block/$1/queue/pas_adaptive_enabled
fio --filename=/dev/$1 --direct=1 --bs=4k --ioengine=pvsync2 \
    --iodepth=1 --rw=randread --runtime=10 --numjobs=$3 --time_based \
    --group_reporting --name=test --eta-newline=1 --cpus_allowed=$2 \
    --nice=-10 --prioclass=2 --prio=0 --hipri

fiodpas.sh — DPAS (Dynamic PAS) Mode

#!/bin/bash
sudo modprobe -r nvme; sudo modprobe nvme poll_queues=2
echo 0 > /sys/block/$1/queue/io_poll_delay
echo 1 > /sys/block/$1/queue/pas_enabled
echo 1 > /sys/block/$1/queue/pas_adaptive_enabled
echo 1 > /sys/block/$1/queue/switch_enabled
echo 10 > /sys/block/$1/queue/switch_param1
echo 10 > /sys/block/$1/queue/switch_param2
echo 10 > /sys/block/$1/queue/switch_param3
echo 1 > /sys/block/$1/queue/switch_param4
fio --filename=/dev/$1 --ramp_time=3 --size=100m --direct=1 --bs=4k \
    --ioengine=pvsync2 --iodepth=1 --rw=randread --runtime=10 \
    --numjobs=$3 --time_based --group_reporting --name=test \
    --eta-newline=1 --cpus_allowed=$2 --nice=-10 --prioclass=2 \
    --prio=0 --hipri
cat /sys/block/$1/queue/switch_stat
dmesg | tail -10

Grant execute permissions:

chmod +x fioint.sh fiocp.sh fiopas.sh fiodpas.sh

2-2. Running Experiments

Pin to CPU 0 and compare the 4 modes while varying the number of jobs.

# Example: INT mode, 1 job
./fioint.sh nvme0n1 0 1

# Repeat for each mode with jobs = 1, 2, 4, 8
# INT:  ./fioint.sh nvme0n1 0 {1,2,4,8}
# CP:   ./fiocp.sh  nvme0n1 0 {1,2,4,8}
# PAS:  ./fiopas.sh nvme0n1 0 {1,2,4,8}
# DPAS: ./fiodpas.sh nvme0n1 0 {1,2,4,8}

2-3. Recording Results

Record the IOPS and avg latency from the fio output of each run.

ModeJobs=1Jobs=2Jobs=4Jobs=8
INT
CP
PAS
DPAS

2-4. Verifying DPAS Mode Transition

After running DPAS, check the switch_stat and dmesg output:

MODE ValueMeaning
0INT (Interrupt)
1CP (Continuous Polling)
2PAS (Initial State)
3OL (Overloaded)

Expected Behavior

JobsExpected Final ModeReason
1CP (MODE 1)QD=1 → PAS→CP transition, lowest latency via polling
2+INT (MODE 0)QD>1 → PAS→OL→INT transition, yields CPU via interrupts

2-5. Analysis Questions

  1. What causes the IOPS difference between INT and CP modes when jobs=1?
  2. Why does IOPS not increase significantly when adding more jobs in CP mode?
  3. Explain why DPAS approaches CP performance at jobs=1 and approaches INT performance at jobs=4+ from the perspective of mode transition.
  4. Why is the pas io count always less than the polled io count in switch_stat?

Part 3: DPAS sysfs Interface

The DPAS kernel provides the following parameters under /sys/block/nvme0n1/queue/.

ParameterDescriptionDefault
switch_enabledEnable DPAS mode switching0
switch_statPer-CPU mode statistics output (read-only)
pas_enabledEnable PAS mode0
pas_adaptive_enabledEnable adaptive sleep0
switch_param1PAS→OL threshold (tf > param1)0
switch_param2OL→PAS threshold (avg QD ≤ param2)10
switch_param3OL→INT threshold (avg QD > param3)10
switch_param4Enable PAS→CP transition (0/1)1
QD 10x Multiplier QD-related parameters use a 10x scale. param2=10 means average QD ≤ 1.0. This design provides decimal precision using integer arithmetic.

Part 4: Building the Kernel from Source Challenge

Instead of using deb packages, apply the DPAS patch directly to the kernel source and build it yourself.

4-1. Install Build Dependencies

sudo apt install -y build-essential libncurses-dev bison flex libssl-dev \
    libelf-dev bc pahole dwarves zstd

4-2. Download Kernel Source

cd ~
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.18.tar.xz
tar xf linux-5.18.tar.xz
cd linux-5.18

4-3. Apply the DPAS Patch

# After transferring the patch from host to VM
patch -p1 < ~/dpas.patch
Makefile Hunk Failure Since the patch is based on 5.18-rc6, one Makefile hunk will fail. This can be safely ignored.
Instead, set localversion manually: echo "-dpas" > localversion

4-4. Kernel Configuration (Minimized)

cp /boot/config-$(uname -r) .config
yes '' | make localmodconfig
Why localmodconfig? Building with the default config includes thousands of modules, leading to disk exhaustion (No space left on device). localmodconfig includes only currently loaded modules, drastically reducing build time and disk usage.

4-5. Build

make -j$(nproc) bindeb-pkg

Upon successful build, linux-image-*.deb and linux-headers-*.deb files will be generated in the home directory.

4-6. Install

sudo dpkg -i ~/linux-image-5.18.0-dpas_*.deb ~/linux-headers-5.18.0-dpas_*.deb
sudo reboot

Troubleshooting

SymptomCauseSolution
No space left on deviceBuild artifacts exceed disk spacemake clean, delete tarball, use localmodconfig
Makefile Hunk #1 FAILEDPatch version mismatchIgnore, use localversion instead
Kernel unchanged after rebootGRUB default boot kernelSelect DPAS kernel with grub-reboot
Multiple hunk failuresPatch re-applied to already-patched sourceRe-extract the source and try again
15GB+ required for buildInsufficient free disk spaceCheck df -h /, extend LVM or clean up files

Reference: Limitations of the QEMU Environment

ItemQEMUReal Hardware
CP vs INT comparisonValid (1.7-1.8x IOPS difference)Valid
PAS adaptive sleepInaccurate timers, tf noiseAccurate
DPAS mode transitionWorks, but absolute performance is for reference onlyWorks with default parameters
Multi-core scalingDistorted by emulation overheadNear-linear
Key Takeaway Focus on the relative differences between modes and the transition behavior rather than absolute performance numbers. The goal is to understand the principles behind I/O completion mechanisms.