CRASH(8)CRASH(8)NAMEcrash - Analyze Linux crash dump data or a live system
SYNOPSIScrash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
DESCRIPTION
Crash is a tool for interactively analyzing the state of the Linux sys‐
tem while it is running, or after a kernel crash has occurred and a
core dump has been created by the netdump, diskdump, LKCD, kdump, xen‐
dump or kvmdump facilities. It is loosely based on the SVR4 UNIX crash
command, but has been significantly enhanced by completely merging it
with the gdb(1) debugger. The marriage of the two effectively combines
the kernel-specific nature of the traditional UNIX crash utility with
the source code level debugging capabilities of gdb(1).
In the dumpfile form, both a NAMELIST and a MEMORY-IMAGE argument must
be entered. In the live system form, the NAMELIST argument must be
entered if the kernel's vmlinux file is not located in a known loca‐
tion, such as the /usr/lib/debug/lib/modules/<kernel-version> direc‐
tory.
The crash utility has also been extended to support the analysis of
dumpfiles generated by a crash of the Xen hypervisor. In that case,
the NAMELIST argument must be that of the xen-syms binary. Live system
analysis is not supported for the Xen hypervisor.
The crash utility command set consists of common kernel core analysis
tools such as kernel stack back traces of all processes, source code
disassembly, formatted kernel structure and variable displays, virtual
memory data, dumps of linked-lists, etc., along with several commands
that delve deeper into specific kernel subsystems. Appropriate gdb
commands may also be entered, which in turn are passed on to the gdb
module for execution. If desired, commands may be placed in either a
$HOME/.crashrc file and/or in a .crashrc file in the current directory.
During initialization, the commands in $HOME/.crashrc are executed
first, followed by those in the ./.crashrc file.
The crash utility is designed to be independent of Linux version depen‐
dencies. When new kernel source code impacts the correct functionality
of crash and its command set, the utility will be updated to recognize
new kernel code changes, while maintaining backwards compatibility with
earlier releases.
OPTIONS
NAMELIST
This is a pathname to an uncompressed kernel image (a vmlinux
file), or a Xen hypervisor image (a xen-syms file) which has
been compiled with the "-g" option. If using the dumpfile form,
a vmlinux file may be compressed in either gzip or bzip2 for‐
mats.
MEMORY-IMAGE
A kernel core dump file created by the netdump, diskdump, LKCD
kdump, xendump or kvmdump facilities.
If a MEMORY-IMAGE argument is not entered, the session will be
invoked on the live system, which typically requires root privi‐
leges because of the device file used to access system RAM. By
default, /dev/crash will be used if it exists. If it does not
exist, then /dev/mem will be used; but if the kernel has been
configured with CONFIG_STRICT_DEVMEM, then /proc/kcore will be
used. It is permissible to explicitly enter /dev/crash,
/dev/mem or /proc/kcore.
mapfile
If the NAMELIST file is not the same kernel that is running
(live system form), or the kernel that was running when the sys‐
tem crashed (dumpfile form), then the System.map file of the
original kernel should be entered on the command line.
-h [option]
--help [option]
Without an option argument, display a crash usage help message.
If the option argument is a crash command name, the help page
for that command is displayed. If it is the string "input", a
page describing the various crash command line input options is
displayed. If it is the string "output", a page describing com‐
mand line output options is displayed. After the help message
is displayed, crash exits.
-s Proceed directly to the "crash>" prompt without displaying any
version, GPL, or crash initialization data during startup.
-i file
Execute the command(s) contained in file prior to displaying the
"crash>" prompt for interactive user input.
-d num Set the internal debug level. The higher the number, the more
debugging data will be printed when crash initializes and runs.
-S Use /boot/System.map as the mapfile.
-e vi | emacs
Set the readline(3) command line editing mode to "vi" or
"emacs". The default editing mode is "vi".
-f Force the usage of a compressed vmlinux file if its original
name does not start with "vmlinux".
-k Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo
file.
-t Display the system-crash timestamp and exit.
-L Attempt to lock all of its virtual address space into memory by
calling mlockall(MCL_CURRENT|MCL_FUTURE) during initialization.
If the system call fails, an error message will be displayed,
but the session continues.
-c tty-device
Open the tty-device as the console used for debug messages.
-p page-size
If a processor's page size cannot be determined by the dumpfile,
and the processor default cannot be used, use page-size.
-m option=value
--machdep option=value
Pass an option and value pair to machine-dependent code. These
architecture-specific option/pairs should only be required in
very rare circumstances:
X86_64:
physbase=<physical-address>
irq_eframe_link=<value>
max_physmem_bits=<value>
vm=orig (pre-2.6.11 virtual memory address ranges)
vm=2.6.11 (2.6.11 and later virtual memory address ranges)
vm=xen (Xen kernel virtual memory address ranges)
vm=xen-rhel4 (RHEL4 Xen kernel virtual address ranges)
PPC64:
vm=orig
vm=2.6.14 (4-level page tables)
IA64:
phys_start=<physical-address>
init_stack_size=<size>
vm=4l (4-level page tables)
ARM:
physbase=<physical-address>
-x Automatically load extension modules from a particular direc‐
tory. If a directory is specified in the CRASH_EXTENSIONS shell
environment variable, then that directory will be used. Other‐
wise /usr/lib64/crash/extensions (64-bit architectures) or
/usr/lib/crash/extensions (32-bit architectures) will be used;
if they do not exist, then the ./extensions directory will be
used.
--memory_module modname
Use the modname as an alternative kernel module to the crash.ko
module that creates the /dev/crash device.
--memory_device device
Use device as an alternative device to the /dev/crash, /dev/mem
or /proc/kcore devices.
--no_kallsyms
Do not use kallsyms-generated symbol information contained
within kernel module object files.
--no_modules
Do not access or display any kernel module related information.
--no_ikconf
Do not attempt to read configuration data that was built into
kernels configured with CONFIG_IKCONFIG.
--no_data_debug
Do not verify the validity of all structure member offsets and
structure sizes that it uses.
--no_kmem_cache
Do not initialize the kernel's slab cache infrastructure, and
commands that use kmem_cache-related data will not work.
--no_elf_notes
Do not use the registers from the ELF NT_PRSTATUS notes saved in
a compressed kdump header for backtraces.
--kmem_cache_delay
Delay the initialization of the kernel's slab cache infrastruc‐
ture until it is required by a run-time command.
--readnow
Pass this flag to the embedded gdb module, which will override
its two-stage strategy that it uses for reading symbol tables
from the NAMELIST.
--smp Specify that the system being analyzed is an SMP kernel.
-v
--version
Display the version of the crash utility, the version of the
embedded gdb module, GPL information, and copyright notices.
--cpus number
Specify the number of cpus in the SMP system being analyzed.
--osrelease dumpfile
Display the OSRELEASE vmcoreinfo string from a kdump dumpfile
header.
--hyper
Force the session to be that of a Xen hypervisor.
--p2m_mfn pfn
When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
is typically analyzed with either the Xen hypervisor or the dom0
kernel. It is also possible to analyze any of the guest domU
kernels if the pfn_to_mfn_list_list pfn value of the guest ker‐
nel is passed on the command line along with its NAMELIST and
the dumpfile.
--xen_phys_start physical-address
Supply the base physical address of the Xen hypervisor's text
and static data for older xendump dumpfiles that did not pass
that information in the dumpfile header.
--zero_excluded
If a kdump dumpfile has been filtered to exclude various types
of non-essential pages, any attempt to read them will fail.
With this flag, reads from any of those pages will return zero-
filled memory.
--no_panic
Do not attempt to find the task that was running when the kernel
crashed. Set the initial context to that of the "swapper" task
on cpu 0.
--more Use /bin/more as the command output scroller, overriding the
default of /usr/bin/less and any settings in either ./.crashrc
or $HOME/.crashrc.
--less Use /usr/bin/less as the command output scroller, overriding any
settings in either ./.crashrc or $HOME/.crashrc.
--CRASHPAGER
Use the output paging command defined in the CRASHPAGER shell
environment variable, overriding any settings in either
./.crashrc or $HOME/.crashrc.
--no_scroll
Do not pass run-time command output to any scrolling command.
--no_crashrc
Do not execute the commands in either $HOME/.crashrc or
./.crashrc.
--mod directory
When loading the debuginfo data of kernel modules with the mod
-S command, search for their object files in directory instead
of in the standard location.
--reloc size
When analyzing live x86 kernels that were configured with a CON‐
FIG_PHYSICAL_START value that is larger than its CONFIG_PHYSI‐
CAL_ALIGN value, then it will be necessary to enter a relocation
size equal to the difference between the two values.
--minimal
Bring up a session that is restricted to the log, dis, rd, sym,
eval, set and exit commands. This option may provide a way to
extract some minimal/quick information from a corrupted or trun‐
cated dumpfile, or in situations where one of the several kernel
subsystem initialization routines would abort the crash session.
--kvmhost [32|64]
When examining an x86 KVM guest dumpfile, this option specifies
that the KVM host that created the dumpfile was an x86 (32-bit)
or an x86_64 (64-bit) machine, overriding the automatically
determined value.
--kvmio <size>
override the automatically-calculated KVM guest I/O hole size.
COMMANDS
Each crash command generally falls into one of the following cate‐
gories:
Symbolic display
Displays of kernel text/data, which take full advantage of the
power of gdb to format and display data structures symbolically.
System state
The majority of crash commands consist of a set of "kernel-
aware" commands, which delve into various kernel subsystems on a
system-wide or per-task basis.
Utility functions
A set of useful helper commands serving various purposes, some
simple, others quite powerful.
Session control
Commands that control the crash session itself.
The following alphabetical list consists of a very simple overview of
each crash command. However, since individual commands often have sev‐
eral options resulting in significantly different output, it is sug‐
gested that the full description of each command be viewed by executing
crash-h <command>, or during a crash session by simply entering help
command.
* "pointer to" is shorthand for either the struct or union com‐
mands. It displays the contents of a kernel structure or union.
alias creates a single-word alias for a command.
ascii displays an ascii chart or translates a numeric value into its
ascii components.
bt displays a task's kernel-stack backtrace. If it is given the -a
option, it displays the stack traces of the active tasks on all
CPUs. It is often used with the foreach command to display the
backtraces of all tasks with one command.
btop translates a byte value (physical offset) to its page number.
dev displays data concerning the character and block device assign‐
ments, I/O port usage, I/O memory usage, and PCI device data.
dis disassembles memory, either entire kernel functions, from a
location for a specified number of instructions, or from the
start of a function up to a specified memory location.
eval evaluates an expression or numeric type and displays the result
in hexadecimal, decimal, octal and binary.
exit causes crash to exit.
extend dynamically loads or unloads crash shared object extension mod‐
ules.
files displays information about open files in a context.
foreach
repeats a specified command for the specified (or all) tasks in
the system.
fuser displays the tasks using the specified file or socket.
gdb passes its argument to the embedded gdb module. It is useful
for executing gdb commands that have the same name as crash com‐
mands.
help alone displays the command menu; if followed by a command name,
a full description of a command, its options, and examples are
displayed. Its output is far more complete and useful than this
man page.
irq displays data concerning interrupt request numbers and bottom-
half interrupt handling.
kmem displays information about the use of kernel memory.
list displays the contents of a linked list.
log displays the kernel log_buf contents in chronological order.
mach displays data specific to the machine type.
mod displays information about the currently installed kernel mod‐
ules, or adds or deletes symbolic or debugging information about
specified kernel modules.
mount displays information about the currently-mounted filesystems.
net display various network related data.
p passes its arguments to the gdb "print" command for evaluation
and display.
ps displays process status for specified, or all, processes in the
system.
pte translates the hexadecimal contents of a PTE into its physical
page address and page bit settings.
ptob translates a page frame number to its byte value.
ptov translates a hexadecimal physical address into a kernel virtual
address.
q is an alias for the "exit" command.
rd displays the contents of memory, with the output formatted in
several different manners.
repeat repeats a command indefinitely, optionally delaying a given num‐
ber of seconds between each command execution.
runq displays the tasks on the run queue.
search searches a range of user or kernel memory space for given value.
set either sets a new context, or gets the current context for dis‐
play.
sig displays signal-handling data of one or more tasks.
struct displays either a structure definition or the contents of a ker‐
nel structure at a specified address.
swap displays information about each configured swap device.
sym translates a symbol to its virtual address, or a static kernel
virtual address to its symbol -- or to a symbol-plus-offset
value, if appropriate.
sys displays system-specific data.
task displays the contents of a task_struct.
timer displays the timer queue entries, both old- and new-style, in
chronological order.
union is similar to the struct command, except that it works on kernel
unions.
vm displays basic virtual memory information of a context.
vtop translates a user or kernel virtual address to its physical
address.
waitq walks the wait queue list displaying the tasks which are blocked
on the specified wait queue.
whatis displays the definition of structures, unions, typedefs or
text/data symbols.
wr modifies the contents of memory on a live system. It can only
be used if /dev/mem is the device file being used to access sys‐
tem RAM, and should obviously be used with great care.
When crash is invoked with a Xen hypervisor binary as the NAMELIST, the
command set is slightly modified. The *, alias, ascii, bt, dis, eval,
exit, extend, gdb, help, list, log, p, pte, rd, repeat, search, set,
struct, sym, sys, union, whatis, wr and q commands are the same as
above. The following commands are specific to the Xen hypervisor:
domain displays the contents of the domain structure for selected, or
all, domains.
doms displays domain status for selected, or all, domains.
dumpinfo
displays Xen dump information for selected, or all, cpus.
pcpus displays physical cpu information for selected, or all, cpus.
vcpus displays vcpu status for selected, or all, vcpus.
FILES
.crashrc
Initialization commands. The file can be located in the user's
HOME directory and/or the current directory. Commands found in
the .crashrc file in the HOME directory are executed before
those in the current directory's .crashrc file.
ENVIRONMENT
EDITOR Command input is read using readline(3). If EDITOR is set to
emacs or vi then suitable keybindings are used. If EDITOR is
not set, then vi is used. This can be overridden by set vi or
set emacs commands located in a .crashrc file, or by entering -e
emacs on the crash command line.
CRASHPAGER
If CRASHPAGER is set, its value is used as the name of the pro‐
gram to which command output will be sent. If not, then command
output is sent to /usr/bin/less -E -X by default.
CRASH_MODULE_PATH
Specifies an alternative directory tree to search for kernel
module object files.
CRASH_EXTENSIONS
Specifies a directory containing extension modules that will be
loaded automatically if the -x command line option is used.
NOTES
If crash does not work, look for a newer version: kernel evolution fre‐
quently makes crash updates necessary.
The command set scroll off will cause output to be sent directly to the
terminal rather than through a paging program. This is useful, for
example, if you are running crash in a window of emacs.
AUTHOR
Dave Anderson <anderson@redhat.com> wrote crash.
Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@red‐
hat.com> wrote this man page.
SEE ALSO
The help command within crash provides more complete and accurate docu‐
mentation than this man page.
http://people.redhat.com/anderson - the home page of the crash utility.
netdump(8), gdb(1)CRASH(8)
