A starting guide to Kernel Development ====================================== Preamble -------- This document has been viewed through many perspectives from many reviewers, each wanting a conflicting adaptation. Unlike other Kernel documentation, this has different aims. This document is either a concise starting point to a curious reader, or a dissected script to set up a quick development environment. A teaching tool. It is aimed to those individuals who have at least a middling skill level in C and Linux, and so should be suitable for upper level university courses. Parts which a reader does not understand are readily searchable, or at a level to build the reader's research ability to be more suitable to Kernel development. It will not explain everything, only a high level description to keep readers on track of the flow of what is happening, since that has been the most consistent deficiency against this document from reviewers. Introduction ------------ This guide intends to serve as an adaptable example of how to set up an environment for Kernel Development. It uses as an example a non-hardware focused development pattern for the ``fanotify(7)`` system. It is composed of the following high level steps: - Configure the shell environment, folders, and scripts - Gathering a suitable Virtual Machine image, and Linux Repository - Configure the Virtual Machine image - Obtain a Kernel configuration - Compile the Kernel - Using ``gdb``, show basic interaction and debugging steps with the Kernel. ``C`` development past that point is considered out of scope, as is developing for any part of the Kernel proper since there are better sources for each component. Requirements ------------ - git - wget - qemu - gdb - g++ supporting C++23 - make - bison - flex - guestmount - expect `NOTE: On some distributions, kernels (/boot/vmlinuz\*) lack global read permissions. Administrator permissions are required to make the kernel chosen by` ``guestmount`` `to be readable. There is debate about the effectiveness of this security decision. On some distributions like Ubuntu, this will cause a problem. In the context of changing a one-off system, having this file globally read-only is considered safe.` Setup ----- Configure Environment ~~~~~~~~~~~~~~~~~~~~~ This step will set up some environment variables which will be used in this guide, as well as variables used for ``guestmount`` and to allow ``gdb`` to use the debugging scripts provided by the Linux Kernel source repository. .. code:: bash mkdir -p "$HOME/Documents/linux-workspace/kernel-dev" cd "$HOME/Documents/linux-workspace/kernel-dev" export LINUX_REPO_PATH="$(pwd)/linux" echo 'qemu-system-x86_64 -s -cpu host -nographic -accel kvm -m 1G Arch-Linux-x86_64-basic.qcow2' > run_vm.sh chmod +x run_vm.sh echo 'qemu-system-x86_64 -s -cpu host -nographic -accel kvm -m 1G -kernel linux/vmlinux -append "root=/dev/sda3 rw console=ttyS0,115200n8 nokaslr" Arch-Linux-x86_64-basic.qcow2 -S' > debug_vm.sh chmod +x debug_vm.sh echo 'guestmount -w -a Arch-Linux-x86_64-basic.qcow2 -m /dev/sda3 mnt/' > mount_vm.sh chmod +x mount_vm.sh echo "export LINUX_REPO_PATH=\"$LINUX_REPO_PATH\"" >> ~/.bashrc echo 'add-auto-load-safe-path "$LINUX_REPO_PATH/scripts/gdb/vmlinux-gdb.py"' >> "$HOME/.config/gdb/gdbinit" echo 'export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1' >> $HOME/.bashrc echo 'export USERCFLAGS=" -ggdb "' >> $HOME/.bashrc source "$HOME/.bashrc" Obtain Resources ~~~~~~~~~~~~~~~~ Here we need to obtain the basic materials to work with. These are a Virtual Machine image and a copy of the Linux Kernel source code itself. `NOTE: While this example uses the main Linux trunk, you should look up the most specific repo and branch for your development case.` .. code:: bash git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git wget https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-basic.qcow2 Modify the Virtual Machine to be scriptable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This step performs several actions and is slow. First, it copies the Linux Kernel source into the VM. Next, it changes the default GRUB settings in the VM such that on each GRUB update, the VM will be set up to use a serial console. After that, since we require a serial console to script VM interaction, an unfortunately complex pipeline is used to augment the ``grub.cfg`` file so that both GRUB and Linux use a configured serial console to interact. .. code:: bash mkdir -p mnt ./mount_vm.sh cp -r linux mnt/home/arch/ pushd mnt/ echo 'GRUB_TERMINAL_INPUT="console serial"' >> etc/default/grub echo 'GRUB_TERMINAL_OUTPUT="gfxterm serial"' >> etc/default/grub echo 'GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"' >> etc/default/grub echo 'serial --unit=0 --speed=115200' > tmp cat grub.cfg >> tmp mv -f tmp grub.cfg sed 's/terminal_input console/terminal_input serial/' < grub.cfg | sed 's/terminal_output gfxterm/terminal_output serial/' > tmp && mv -f tmp grub.cfg BOOT_LINES="$(grep '/boot/vmlinuz-linux' < grub.cfg | sort -u)" echo "$BOOT_LINES" | while IFS= read -r l ;\ do l_escaped="$(printf '%s' "$l" | sed -e 's/[\/&]/\\&/g')" ;\ l_escaped="$(printf '%s' "$l_escaped" | sed -e 's/\t/\\t/g')" ;\ sed "s/$l_escaped/$l_escaped console=ttyS0,115200n8/g" < grub.cfg | sed 's/console=ttyS0,115200n8 console=ttyS0,115200n8/console=ttyS0,115200n8/g' > tmp ;\ mv -f tmp grub.cfg ;\ chmod 664 grub.cfg ;\ done popd umount mnt Obtain applicable Kernel configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All this work lets the following script work. This is an ``expect`` script and can match and send in response-specific data. It configures the VM and generates the critical ``.config`` file needed to compile the Linux Kernel. The reason all this effort is taken to generate the ``.config`` inside the VM but not compile it is purely one of resource and time efficiency. As a script: .. code:: #!/usr/bin/expect -f set timeout -1 set log [lindex $argv 0] spawn qemu-system-x86_64 -s -cpu host -nographic -accel kvm -m 1G Arch-Linux-x86_64-basic.qcow2 expect "login: " send "arch\r" expect "Password: " send "arch\r" expect "$ " send "sudo pacman -Syu --noconfirm --needed base-devel xmlto inetutils bc git cpio perl\r" expect "$ " send "cd linux\r" expect "$ " send "sudo make localyesconfig\r" expect { "(NEW)" { send "\r" exp_continue } "$ " { send "sudo poweroff\r" expect eof exit } } Finally, we copy out the ``.config`` from the VM to the host's Linux Kernel repo. However, a few tweaks are known to be needed, and these are performed below. Further changes may be accomplished through the use of commands like ``make xconfig``. .. code:: bash ./mount_vm.sh cp -r mnt/home/arch/linux/.config linux/.config sync umount mnt pushd $LINUX_REPO_PATH sed -i '/CONFIG_SATA_AHCI=$/d' .config sed -i '/CONFIG_DEBUG_INFO=$/d' .config sed -i '/CONFIG_DEBUG_INFO_DWARF5=$/d' .config sed -i '/CONFIG_GDB_SCRIPTS=$/d' .config sed -i '/CONFIG_GDB_INFO_REDUCED=$/d' .config sed -i '/CONFIG_KGDB=$/d' .config sed -i '/CONFIG_FRAME_POINTER=$/d' .config sed -i '/CONFIG_KVM_GUEST=$/d' .config sed -i '/CONFIG_RANDOMIZE_BASE=$/d' .config echo "CONFIG_SATA_AHCI=y" >> .config echo "CONFIG_DEBUG_INFO=y" >> .config echo "CONFIG_DEBUG_INFO_DWARF5=y" >> .config echo "CONFIG_GDB_SCRIPTS=y" >> .config echo "CONFIG_GDB_INFO_REDUCED=n" >> .config echo "CONFIG_KGDB=y" >> .config echo "CONFIG_FRAME_POINTER=y" >> .config echo "CONFIG_KVM_GUEST=y" >> .config echo "CONFIG_RANDOMIZE_BASE=n" >> .config Compile Linux ~~~~~~~~~~~~~ The last setup step is to compile the kernel as shown below. This is CPU and memory intensive. It is also notorious for failing for odd reasons. Just because a configuration exists doesn't mean it can work. If you have difficulty here, it is best to reach out to the #kernel IRC channel for advice. `NOTE: Don’t forget to adjust the "-j" flag for your system. Many users will need to reduce the number of parallel jobs running.` `NOTE: This step may require manual intervention. Simply accept the default options for [Y]es or [N]o. Avoid [M]odule because for later steps kernel modules will be unavailable to load into Qemu from external kernel loading.` .. code:: bash make -j popd Develop and debug ----------------- At this juncture, the raw resources are present for development. Now, the rest needs to be shown through an example. This guide uses an example for ``fanotify(7)`` which is simple and easily adapted for other use cases. Term 1 ~~~~~~ .. code:: bash ./debug_vm.sh # # git clone https://gitlab.com/anadon/monitor-for-free-space cd monitor-for-free-space make ./mnt_monitor / 5000 ^Z bg 1 touch ~/_1 # Term 2 ~~~~~~ .. code:: gdb $LINUX_REPO_PATH/vmlinux : target remote:1234 : continue : ^C : break fs/notify/fanotify/fanotify_user.c:process_access_response : continue : bt : step : continue And that’s it! Citations --------- - https://github.com/archlinux/arch-boxes - https://linuxlink.timesys.com/docs/how_to_use_kgdb - https://www.bodunhu.com/blog/posts/start-linux-kernel-hacking/ - https://lore.kernel.org/linux-newbie/CAFkJGRcWdHpbxjBrynncOb3apWO7d5+WP82WqcHeq=hsc65MAA@mail.gmail.com/ - https://stackoverflow.com/a/7810608/826573 - https://stackoverflow.com/a/2705678/826573 Special Thanks ~~~~~~~~~~~~~~ - ngn