What could be possibly worse that an almost unbeatable boss in a game or a tough maze that consume hours of gameplay with not much progress? How about a Linux kernel crash that makes you lose all the game progress with no apparent reason or feedback? Though rare, it is a real possibility that would make gamers quite annoyed, given that Linux is used more and more as a platform for playing games. Some technologies are available to collect logs and feedback the user in case such disastrous events happen, mostly related with kernel crashes handling mechanisms. The main ones available are kdump and pstore, but still there are work to be done in this area... In this talk we're going to present the basics about kernel crash handling, like how a kernel panic might happen, how to deal with that (with an overall discussion about kdump and pstore techs) and the kdumpst tool, developed specially to deal with this situation on Steam Deck (and generically on Arch Linux); also we're gonna discuss some missing pieces / ideas to make it even less likely gamers need to complain that their device just got hang for no reason! (c) X.Org Developer's Conference (XDC) 2023 October 17-19, 2023 A Coruña (Spain) https://indico.freedesktop.org/event/4/

1. To crash or not to crash:
If you do, at least recover fast!
Guilherme G. Piccoli (Igalia)
2023-10-19 / XDC
1

2. Context
Steam Deck game console -> SteamOS Linux distro
Arch Linux based
No official tool for Arch Linux to collect kernel crash
logs / data - that would be nice to have
What data do we want to collect? What's useful
enough?
Focus on being lightweight and don't waste resources
2

3. Kernel oops / panic
Oops is when kernel detects a bad condition / error
Panic is the kernel reaction, if set!
Potential panic causes:
NULL pointers (concurrency / SMP)
Lockups (soft/hard, hung tasks)
Artificial triggered (sysrq), BUG() clause
What can we do?
Reboot (and pretend nothing happened!)
Or...collect data: vmcore, dmesg (kernel log)
3

4. Collecting (all the) data: kdump
Jump to a new kernel (kexec), using pre-reserved
RAM area
Preserves memory content of the crashed kernel
Collects broken kernel's core dump (vmcore)
Pros: (Almost) full copy of crashed kernel memory
Post-mortem analysis (easy to share)
Cons: Pre-reserved memory required (>200M usually)
Risks during kernel booting / vmcore collecting
PCI devices reset: potential nightmare
4

5. Collecting (most) data: pstore
Saves the kernel log in a persistent storage/backend
during panic time
Multiple backends: RAM, UEFI, block device
Lightweight: used on embedded (and
chromebooks)
Pros: very fast and (hopefully) transparent process
{Small amount / No} memory required
Doesn't require kexec support
Cons: does not save a full vmcore
Some backends may have limited space (UEFI)
5

6. Presenting: kdumpst
Kdump/pstore alone are kernel techs
They require userspace counter-parts
is an Arch Linux pstore/kdump tool
Available on
Supports GRUB and initcpio / dracut (systemd-boot plans)
Defaults to pstore mode (but supports kdump);
currently only ramoops backend (UEFI plans)
Customizable: sysctls, `crashkernel`, log
compression
Present on SteamOS as Deck's crash collecting tool
kdumpst
AUR
6

7. THANKS
Feel free to reach me on IRC (gpiccoli - OFTC/Libera)
7