KTR(4) BSD Programmer's Manual KTR(4)NAMEktrSYNOPSIS
options KTR
options KTR_MASK
options KTR_COMPILE
options KTR_SIZE
options KTR_OLDCPU
DESCRIPTION
The ktr facility provides a low overhead execution tracing mechanism to
kernel code. The functionality is similar to printf(), however the format
string is not interpreted at run time (it is interpreted when the trace
buffer is decoded), thus the overhead for a trace point is very low
(about 20 clocks on the Intel architecture if the trace class is enabled,
or 4 clocks if the trace class is disabled at run time).
The traces are kept in a circular buffer on a per-cpu basis. Each trace
entry in the buffer is 32 bytes, although not all bytes must be written
at a given trace point. When a new trace is made it overwrites the oldest
entry in the trace buffer. A trace entry consists of a 2 word time stamp,
a one word pointer to the format string for the trace entry, and up to 5
arguments.
The size of trace buffer for each CPU is determined at compile time by
the KTR_SIZE option. KTR_SIZE is a byte count and must be a power of 2;
the minimum trace buffer size is 32 bytes and the maximum is determined
by available kernel virtual memory in the per-cpu data area (on the Intel
architecture this limits the trace buffer size to 2MB, or 132K trace en-
tries). The default kernel trace buffer size is 32K, or 1024 entries.
The trace buffer for the first (boot) CPU in a configuration is always
allocated in the BSS area, the trace areas for additional CPUs are dynam-
ically allocated when they are initialized.
The time stamp is the high resolution CPU cycle counter if one is avail-
able, otherwise the contents of the kernel global time structure (seconds
and microseconds as of the last tick or call to microtime) is used.
The conditional facilities are based on the classes defined in sys/ktr.h.
Up to 32 classes may be defined, each class may be enabled or disabled at
compile time or run time. Each trace point must have a trace class asso-
ciated with it; a set of macros are provided that do not require a class,
they use the default trace class KTR_GEN.
The preprocessor symbol KTR_COMPILE defines which trace classes will be
compiled into the kernel. Only classes that are enabled via 1 bits in
KTR_COMPILE will generate trace code. To compile all possible tracing
into the kernel set KTR_COMPILE to ~0, note that this will result in a
fairly large kernel as each trace point will generate a small amount of
code plus the trace string itself. There is no runtime or kernel size
overhead for trace points which are disabled via KTR_COMPILE.
The kernel global variable ktr_mask defines which trace classes are ac-
tive at run time. There are certain conditions when all tracing is dis-
abled by a kernel facility setting ktr_mask to 0 (usually temporarily):
the in-kernel debugger ( kdebug(4)) disables tracing when it is active,
when the system panics tracing is disabled, and when the CPU is not capa-
ble of running the tracing instructions (although the KTR_OLDCPU option
may mitigate this condition on some architectures). A common use of
ktr_mask in a debugging situation is to set it to 0 when some (possibly
complex) error condition has been detected, usually in a device driver;
after the problem is reproduced the trace buffer then contains detailed
history leading up to the problem. Run time enabling of trace classes
that have not been compiled in has no effect.
The entire trace facility can be disabled by omitting
options KTR from the configuration; this is similar to setting
KTR_COMPILE to 0, however it also removes all support code (such as the
trace decoder found in kdebug(8) ).
The trace facility can be disabled in binary only kernels; in this case
the support code and trace points are not compiled out (since the source
code is not available), rather ktr_mask and ktr_size are set to 0 to pre-
vent any space from being allocated for the trace buffer and prevent any
of the compiled in trace points from attempting to write to the trace
buffer.
TRACE POINTS
Trace points are implemented via C preprocessor macros. The actual code
that makes the trace entry is architecture specific and is usually writ-
ten in assembly code (using C asm statements). Macros are available for
use in C and assembly modules.
For C modules the trace macros are called as:
CTRn(class, desc, [args])
The n above refers to the number of arguments that will be supplied (not
including class and desc ), and may be from 0 through 5. Class defines
which class (or classes) this trace is a member of, the predefined class-
es are in sys/ktr.h. Device drivers should generally used the KTR_DEV
class, although private classes may be defined by using unassigned class
bits if necessary. Desc is the printf()() style format string. A maximum
of 5 % substitutions may be used in a single trace, and only the substi-
tutions %x, %d, and %s should generally be used; the best rule of thumb
is to not use a % substitution that will require more than one word (ex-
cept for %s). Widths may be specified, but precisions may not, and width
specifications are ignored for %s. The %b format (which is only available
in the kernel printf() function) is not recognized.
The arguments may be any valid C expressions (including functions calls,
although this is usually not good practice since it may change the behav-
ior of the program when traces are enabled vs. disabled). Keep in mind
that simpler expressions will result in smaller/faster trace code at a
given trace point (with the fastest trace point being one that takes no
arguments).
An example C trace point:
CTR1(KTR_DEV, "lpopen(dev=%x)", dev);
An abbreviated version of the trace macros that leaves out the class is
available, it is called as:
TRn(desc, [args]);
This implicitly uses the KTR_GEN class.
Trace macros are available for assembly modules as well, these are
ATR(desc), CATR(class, desc, label), SEG_ATR(desc, segment),
SEG_ATR_DESC(desc, segment), and CATRD(class, desc, label). When using
the assembly macros the conditional compilation must be hand coded, a
typical assembly trace point (for the Intel architecture) looks like:
#if KTR_COMPILE & KTR_SMP
CATR(KTR_SMP, "AP_STARTED, unit id=%x", 1f);
movl LAPIC_ID,%eax
movl %eax,KTR_PARM1(%ebx)
1:
#endif
VIEWING THE TRACE BUFFER
Once the trace entries are made they may be viewed multiple ways, the
most common is the tdump(8) utility, this reads and decodes the trace
buffer using the kvm(3) kernel memory interface (running kernel, crash
dump, or ICE probe). Example output from tdump:
bash# tdump
C Idx Delta-T Trace
= === ======== =====
0 360 ff syscall: run user p=0xf07e5600
0 35f 7e syscall p=0xf07e5600 err=0 ret=f017542c
0 35e 11c syscall p=0xf07e5600 #0xc7
0 35d 209 syscall: re-entering kernel
0 35c 3eb syscall: run user p=0xf07e5600
0 35b 13f syscall p=0xf07e5600 err=0 ret=2a
0 35a 477 splx done ../../i386/isa/com.c:2097 cpl=7fffa7a0
0 359 b5 splx ../../i386/isa/com.c:2097 cpl=7fffb7ba -> 7fffa7a0
0 358 1a2 spltty ../../i386/isa/com.c:2048 7fffa7a0 | 7fffb7ba
[...]
In situations where a system has crashed or is unable to write a dump,
kdebug(8) may be used to decode the trace buffer:
0> t
Newest entry at clock 311a5899992c
0 398 0: disable tracing for kdebug
0 397 b638dd: delaying debugger entry 100ms
0 396 f0e: kdebug: entry
0 395 d3: splx done ../../i386/isa/com.c:2212 cpl=7ffffffa
0 394 79: splx ../../i386/isa/com.c:2212 cpl=ffffffff -> 7ffffffa
It is also possible to manually look at the trace entries with cross sys-
tem (serial or ICE probe) gdb(1).
MULTIPROCESSOR
On multiprocessor systems, each processor has its own trace buffer. The
trace decode utility performs a merge sort of the trace entries from the
trace buffers; merging is done by sorting on the time stamp entries, this
can produce confusing traces when the CPU cycle counters are not synchro-
nized (which generally should not happen unless something is broken).
In a multiprocessor system KTR_SIZE defines the size of the trace buffer
for each CPU.
OPTIONS
The kernel configuration options used by the ktr facility are:
KTR Enables the ktr facility when defined.
KTR_SIZE Size of each CPU's trace buffer in bytes, must be a power of
2 and no smaller than 32 bytes. Defaults to 32768.
KTR_MASK Default run time mask (used to initialize ktr_mask ). De-
faults to ~(KTR_SPL | KTR_CLK).
KTR_COMPILE Defines which trace classes are enabled at compile time. De-
faults to ~(KTR_SPL).
KTR_OLDCPU If defined an alternate implementation is used (if avail-
able). On the Intel architecture this generates code that
will run on a pre-Pentium CPU (this option is not available
for binary only kernels).
HISTORY
The ktr facility first appeared in BSD/OS 3.0.
BSDI BSD/OS December 10, 1997 4