Linux Kernel Debugging and Measurement (LNXKRN2)

Unix-like systems, Linux - programming

Linux provides a wide range of advanced tools for kernel debugging and tracing, allowing developers to gain deep insights into kernel behavior without disrupting its operation. This course focuses on effective use of tools like ftrace, eBPF, and other mechanisms for detailed performance analysis, function tracing, and interrupt debugging. Participants will learn how to instrument kernel code and leverage both static and dynamic tracepoints to gather valuable data on kernel operations.

This course is designed for embedded systems developers and system programmers aiming to enhance their debugging skills and optimize Linux kernel performance.

Location, current course term

Contact us

Custom Customized Training (date, location, content, duration)

The course:

Hide detail
  • Observability of the kernel code
    1. Difference from user-space observability
    2. How to modify code for observability
    3. What helps us avoid the modifications
  • Kernel logging subsystem 
    1. Writing to the message ring buffer 
    2. Configuring debug prints in Makefile
    3. Using dynamic debugging feature
  • Using the ftrace subsystem
    1. Interface in debugfs
    2. Listing and tracing functions
    3. Bonus: Using trace-cmd command
  • Kernel function dynamic tracing
    1. Tracing function enter and exit 
    2. Measuring function call duration
    3. Exploring the function graph
  • Static kernel tracepoints 
    1. Tracing well-known events 
    2. How trace points are added
    3. Advantages of named trace points
  • The eBPF subsystem
    1. Kernel bytecode interpreter
    2. Perf event integration
    3. Compiling eBPF tools
  • Creating custom tracers with bpftrace 
    1. Tracing language similar to SystemTap
    2. Using per-cpu and per-task variables
    3. Printing latency histograms
  • Profiling with bpftrace 
    1. Detecting missed deadlines
    2. Investigating causes of problems
    3. Measuring suspicious activities 
  • Examples of debugging and tracing 
    1. Examples shipped with bpftrace and BCC 
    2. Measuring scheduler and preemption 
    3. Tracing interrupts and handlers 
  • More uses of eBPF in debugging 
    1. BCC and data processing in Python 
    2. CO-RE and eBPF on embedded systems 
    3. Other languages for eBPF tracing 
  • Other potentially useful tools 
    1. KASan, KCSAN, KMemleak, Lockdep 
    2. KGDB, Kdump, perf, SystemTap 
    3. SystemTap for kernel modifications 
Assumed knowledge:
Knowledge of the Unix environment and programming in C language.
Schedule:
3 days (9:00 AM - 5:00 PM )
Language: