lockinfo(8)lockinfo(8)NAMElockinfo - Displays kernel locking statistics for SMP and NUMA plat‐
forms
SYNOPSIS
/usr/sbin/lockinfo [-since_boot] [-class=lock_class] [-top=count]
[-sort=statistic] [-rad=rad_id] [-percpu] [command [cmd_args]]
OPTIONS
Specifies a particular lock class name (for example, thread.lock) for
which trace information is to be displayed. This option adds to the
general information displayed about all lock classes detailed informa‐
tion about where in the kernel code the specified lock class is
asserted.
For an alternative way to request trace information, see the
entry for the -top option. Displays statistics for each lock on
a per-CPU basis. Displays lock statistics for the specified
RAD.
This option is useful only on NUMA systems. The lockinfo command
prints an error message if the specified RAD does not exist.
The -rad option is available in the version of the lockinfo
software included in the Tru64 UNIX product kit, starting with
Version 5.1B. However, this version of lockinfo can also be used
on Versions 5.1 and 5.1A. Displays locking statistics for the
system since it was booted. To use this option, the system must
have been booted with the lockmode attribute set to 4. See
sys_attrs_generic(5) for more information about the lockmode
attribute. Sorts output statistics from highest to lowest for
the specified statistic. The value for statistic can be one of
the following, which also correspond to the column headings in
the lockinfo display: The number of tries for asserting the
lock. (Default) The number of attempts on read. The maximum
number of readers in the critical path (for a C class lock) or
the maximum number of cycles spent holding the lock (for an S,
RWS, and MCS class lock). The number of lock misses. The num‐
ber of blocks encountered while waiting for the lock. The maxi‐
mum amount of time (in seconds) spent waiting for a lock. The
total (in seconds) of all times spent waiting for the lock. The
lock miss percent (corresponds to the percent misses column in
the display).
You can also specify none for statistic. In this case, lockinfo
sorts output in the order in which the lock classes are defined
in the kernel source. (This is not a particularly useful
option.) Causes lockinfo to do an initial pass of statistics
gathering and then perform lock class tracing for each, in turn,
of the asserted locks that sort in the top count from the first
pass. When you specify the -top option, any command you pass to
lockinfo will execute count+1 times.
To request trace information for one specific lock class, see
the entry for the -class option.
OPERANDS
The command to be executed by lockinfo. Any arguments to the preceding
command.
The command and cmd_args operands are used to limit the length of time
that lockinfo runs. Typically, sleep is specified for command and some
number of seconds is specified for cmd_args.
DESCRIPTION
The lockinfo utility collects and displays locking statistics for the
kernel SMP locks. It uses the /dev/lockdev pseudo driver to collect
data. Locking statistics can be gathered when the lockmode attribute
for the generic subsystem is set to 2 (the default for SMP systems), 3,
or 4. Be sure to review RESTRICTIONS for important information about
support for this utility.
When you enter a lockinfo command, the utility first opens the lockdev
pseudo driver and turns on lock statistics gathering. Then, the utility
forks and executes the specified command. After command completes, the
utility turns off lock statistics gathering (closes lockdev), collects
the data, and sends it to stdout. The output data shows the SMP lock‐
ing that was done by the operating system during the execution time for
command.
If you do not include command to set a time limit on the interval for
gathering statistics, the statistics shown are for the length of time
since the last system boot.
To gather statistics with lockinfo, you typically follow these steps:
Start up a system work load and wait for it to get to a steady state.
Start lockinfo with sleep as the specified command and some number of
seconds as the specified cmd_args. This causes lockinfo to gather sta‐
tistics for the length of time it takes the sleep command to execute.
Based on the first set of results, use lockinfo again to request more
specific information about any lock class that shows results, such as a
large percentage of misses, that you believe is very likely to cause a
system performance problem.
For example, the following command causes lockinfo to collect locking
statistics for 60 seconds: # lockinfo sleep 60
The output from this command might look like this:
hostname: sysname.node.corp.com lockmode: 2 (SMP default)
processors: 4 start time: Wed Jun 9 14:38:05 1999 end time:
Wed Jun 9 14:39:05 1999 command: sleep 60
tries reads trmax misses percent sleeps waitmax
waitsum
misses seconds
seconds bsBuf.bufLock (S)
5718642 0 45745 194509 3.4 0 0.00007
0.63226 lock.l_lock (S)
5579643 0 40985 75656 1.4 0 0.00005
0.16531 thread.lock (S)
1989132 0 24817 21795 1.1 0 0.00003
0.03864 vnode.v_lock (S)
1578583 0 49207 1527 0.1 0 0.00002
0.00443 vdT.vdIoLock (S)
1412449 0 149797 81078 5.7 0 0.00017
0.51438 BsBufHashLock (S)
1377903 0 42586 89312 6.5 0 0.00006
0.24563
. . .
inifaddr_lock (C)
1 1 1 0 0.0 0 0.00000
0.00000
total simple_locks = 28545191 percent unknown
= 0.0 total rws_locks = 1429 percent reads = 100.0 total complex_locks
= 2764296 percent reads = 33.2 percent unknown = 0.0
#
The first six lines of output specify the system, its lockmode
attribute setting, how many processors it has, the start and end times
of the statistics gathering, and the command that was run.
Following the initial six lines, a table with statistics data for each
lock class is displayed. See the description of the -sort option for
an explanation of the data in each column. In this example, the lock
class entries in the table are sorted by the number of tries (the
default sort order).
Note that each lock class name is tagged (in parentheses) with its lock
type. The following lock types are supported:
───────────────────────────────────────
Tag Lock Type
───────────────────────────────────────
(C) Complex lock
(MCS) Queued lock (NUMA system only)
(RCS) Reader/Writer spin lock
(S) Simple spin lock
───────────────────────────────────────
Finally, the display ends with some summary information: the total num‐
ber of different types of locks and the read percentages for each type.
When diagnosing a system problem, certain statistics are more important
than others to look at. The following results are most likely indicate
a problem: A large number of tries A percent misses value that is too
high.
“Too high” varies somewhat, depending on the lock class. A ker‐
nel developer who is testing VM code under development might
consider any percentage over 1 percent “too high” for certain
kinds of locks. A support representative testing released prod‐
uct software should look for percent misses values that exceed
the range of 5 to 7 percent. A large waitsum value
If you see these results, there are no hard and fast rules to tell you
whether the lock contention is poor code design in applications that
are running on the system or whether there is a problem in the operat‐
ing system software. A locking problem is simply an indication that
there is high contention for a certain type of resource. If contention
exists for a lock related to I/O and a particular application is spawn‐
ing many processes that compete for the same files and directories,
application or database storage design adjustments might be in order.
Applications that use System V semaphores can sometimes encounter lock‐
ing contention if they create a very large number of semaphores in a
single semaphore set because the kernel uses locks on each set of sema‐
phores. In this case, performance improvements might be realized by
changing the application to use more semaphore sets, each with a
smaller number of semaphores. Contention for locks on other kinds of
resources, such as memory, is less likely to be remedied by changing
application code; however, further investigation has to be done before
you can rule out a problem in the application.
Keep in mind that even when lock contention can be reduced by changing
applications and this option might be the only short-term solution, it
does not necessarily mean that the problem should not be reported to
central engineering. For example, if the applications were not causing
locking problems when run on an SMP system but do on a NUMA system with
comparable CPU and memory resources, support representatives should
report the case to central engineering. In this case, kernel developers
will want to investigate system algorithms that better handle the con‐
tention.
In the first example of lockinfo output, a few locks show high values
in the tries, percent misses, and waitsum columns. The following exam‐
ple uses lockinfo to show the code paths where one of these,
bsBuf.bufLock, is asserted with high frequency: # lockinfo
-class=sBuf.bufLock sleep 60 hostname: sysname.node.corp.com
lockmode: 2 (SMP default) processors: 4 start time: Wed
Jun 9 15:02:39 1999 end time: Wed Jun 9 15:03:39 1999 command:
sleep 60
Locks asserted by PC for lock class: bsBuf.bufLock
count miss caller : line # return
: line #
-------------------------------------------------------------------------------
733418 41275 bs_pinpg_one_int: 4494
bs_pinpg_clone: 4125 704579 49182 bs_pinpg_one_int: 4460
bs_pinpg_clone: 4125 697209 0 find_page: 5986
bs_pinpg_one_int: 4368 680910 30359 bs_unpinpg: 5461
log_donerec_nunpin: 3149 544828 12869 bs_q_lazy: 2006
bs_q_list: 1142 496294 15537 bs_unpinpg: 5670
log_donerec_nunpin: 3149 357840 0 find_page: 5986
bs_refpg_int: 2800 115502 21058 bs_derefpg: 3982
bmtr_update_rec_int: 2473
. . .
7 0 seq_ahead_cont: 6879
bs_pinpg_one_int: 4568
2 0 bs_derefpg: 3982 frag_group_deal‐
loc: 1637
tries reads trmax misses percent sleeps waitmax waitsum
misses seconds seconds
bsBuf.bufLock (S) 5322157 0 45745 366848 6.9 0
0.00009 1.77041
. . .
inifaddr_lock (C)
1 1 1 0 0.0 0 0.00000 0.00000
total simple_locks = 26331214 percent unknown
= 0.0 total rws_locks = 1265 percent reads = 100.0 total com‐
plex_locks = 2544907 percent reads = 33.2 percent unknown = 0.0
Based on the information returned about high frequency code path asser‐
tions for the sBuf.bufLock lock, the kernel developer can then look for
ways to reduce the amount of locking for this class. Strategies for
reaching this goal might include one or more of the following: Changing
the code to use a read/write spin lock rather than a simple spin lock
Reducing lock hold times by moving some work outside the time that the
lock is being held Making more radical changes in kernel algorithms to
reduce the frequency of lock assertions or the amount of time that
locks are held
The support representative with training in operating system internals
might be interested in tracing high frequency code path assertions to
better determine if a system performance problem requires submission of
a problem report on the operating system software or if changes need to
be made in third-party or site-specific applications.
See EXAMPLES for examples of lockinfo commands that include the -percpu
and -rad options.
NOTES
Privileges for using the lockinfo command are based on permissions for
the /dev/lockdev file. By default, permissions on this file mean that
using the lockinfo command requires superuser privilege. System admin‐
istrators can change the permissions on the /dev/lockdev special file
to allow “others” to gather lockinfo statistics without granting a par‐
ticular user superuser privileges.
The lockinfo command is indirectly invoked by the sys_check script. The
owner and group for the /dev/lockdev file should not be changed; in
other words, the file must be owned by root and remain in group mem.
Furthermore, user root and group mem must have read permission on the
file. For security reasons, do not run lockinfo in setuid root mode or
add nonprivileged users to group mem. (The latter action gives nonpriv‐
ileged users read permissions on physical memory.)
The version of lockinfo software included in Tru64 UNIX Version 5.1B
and higher releases includes NUMA support (gathering of statistics on a
per-RAD basis). Support representatives can copy this version of lock‐
info to NUMA platforms running Tru64 UNIX Version 5.1 or Version 5.1A,
but should be careful to protect the version of lockinfo already
installed on the system by using one of the following strategies: Copy
the new version to and run it from a directory other than /usr/sbin.
(For example, copy the new version of lockinfo to /usr/local/bin.)
Copy the new version of lockinfo to /usr/sbin but under a different
utility name (such as lockinfo_2).
Protecting the version of lockinfo that originally shipped with the
system is important because of the sys_check dependency.
RESTRICTIONS
Running the lockinfo command can impact system performance. However,
the performance impact exists only for the length of time that statis‐
tics are being gathered (the length of time it takes for the specified
command to execute).
Only one instance of lockinfo can be running on the system at a partic‐
ular time. This is because a process cannot open the /dev/lockdev
pseudo device when it is already opened by another process.
The lockinfo command is intended for use mainly by other facilities
distributed with the operating system, kernel developers, and support
representatives. It is not intended for use by customers who are not
working with a support representative to diagnose a performance prob‐
lem. Furthermore, lockinfo should not be invoked by customer scripts
and applications for the following reasons: The utility displays infor‐
mation that is difficult to interpret without training in operating
system internals; many lock class names are not intuitively obvious;
that is, the name does not suggest an association with a particular
kernel subsystem or the type of system resource being locked. The
utility interface and output is subject to change, without advance
notice, from one release to another and by patches that might be
applied to the utility.
EXIT STATUS
Success. An error occurred.
ERRORS
Lock class class_name not asserted
Explanation: The name specified for the -class option is a supported
lock class for the system. However, no trace information is available;
during the time that locking statistics were being gathered, the speci‐
fied lock was not asserted.
User Action: Verify class names in output from lockinfo that is entered
without the -class option. You also might need to better synchronize
the statistics-gathering period with a time that the lock class is
being asserted in order to generate trace information. This may take a
few tries or an increase in the number of seconds that statistics are
gathered.
lock statistics not supported in lockmode 0 or 1
Explanation: The lockmode attribute must be set to 2, 3. or 4 for the
lockinfo command to work. The default for SMP systems is 2, so someone
reset the value to 0 or 1 when the system was booted.
User Action: The attribute value must be changed to 2, 3, or 4 and the
system rebooted before lockinfo will work.
open: Device busy
Explanation: Another process has the /dev/lockdev device open. The
other process might be running lockinfo directly or be running the
sys_check script (which indirectly invokes lockinfo).
User Action: Wait until the first instance of lockinfo has stopped run‐
ning and try the command again.
option since_boot only supported in lockmode 4
current lockmode: lockmode_value
Explanation: The system must have been booted with lockmode set to 4 in
order to start gathering locking statistics at system boot time.
Because of performance implications for production systems, this is
usually done only on a development system.
User action: Reset lockmode to 4 and reboot the system or, if that is
not possible, use lockmode to gather statistics for short time periods
after performance problems are being noticed.
top count too big count. Max value 1024
Explanation: A number larger than 1024 was specified for the -top argu‐
ment.
User action: Reduce the value. You typically gather trace information
for only the top few lock classes that display the largest values for
the specified sort criteria.
rad number rad_id is invalid
Explanation: The specified RAD number is less than 0 or greater than
the number of RADs on the system.
User action: Re-enter the lockinfo command with a corrected RAD identi‐
fier.
Rad rad_id has no active processors
Explanation: The specified RAD exists on the system but the RAD has no
active processors. Therefore, there are no locking statistics to dis‐
play for that RAD. Note that the lockinfo command does not find proces‐
sors if they are installed but not running. The CPUs in the specified
RAD might have been in the process of being taken on or off line when
lockinfo was run.
User action: Make sure that you specified the number for a RAD in which
you expect CPUs to be active. If not, re-enter the command with a cor‐
rected RAD number. To check on processor status, use the psrinfo com‐
mand.
top number of countlarger then number of lock types number
Explanation: The -top argument exceeded the supported number of lock
class names.
User action: Reduce the value. You usually gather trace information for
only the top few lock classes that display the largest values for the
specified sort criteria.
Unknown class name: class_name
Explanation: An invalid lock class name was specified for the -class
option.
User Action: Verify class names in output from lockinfo when entered
without the -class option. Check the spelling and letter case for sup‐
ported names and re-enter the command with a corrected -class argument.
EXAMPLES
The following lockinfo command gathers locking statistics for each pro‐
cessor over a period of 60 seconds: # lockinfo-percpu sleep 60 host‐
name: sysname.node.corp.com lockmode: 4 (SMP DEBUG with
kernel mode preemption enabled) processors: 4 start time: Wed
Jun 9 14:45:08 1999 end time: Wed Jun 9 14:46:08 1999 command:
sleep 60
tries reads trmax misses percent sleeps wait‐
max waitsum
misses sec‐
onds seconds bsBuf.bufLock (S)
0 1400786 0 45745 47030 3.4 0
0.00007 0.15526
1 1415828 0 45367 47538 3.4 0
0.00006 0.15732
2 1399462 0 33076 48507 3.5 0
0.00005 0.15907
3 1398336 0 31753 48867 3.5 0
0.00005 0.15934
-----------------------------------------------------------------------
ALL 5614412 0 45745 191942 3.4 0
0.00007 0.63099
lock.l_lock (S)
0 1360769 0 40985 18460 1.4 0
0.00005 0.04041
1 1375384 0 20720 18581 1.4 0
0.00005 0.04124
2 1375122 0 20657 18831 1.4 0
0.00009 0.04198
-----------------------------------------------------------------------
ALL 5483049 0 40985 74688 1.4 0
0.00009 0.16526
. . .
inifaddr_lock (C)
0 0 0 1 0 0.0 0
0.00000 0.00000
1 1 1 1 0 0.0 0
0.00000 0.00000
2 0 0 1 0 0.0 0
0.00000 0.00000
3 0 0 1 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 1 1 1 0 0.0 0
0.00000 0.00000
total simple_locks = 28100338 percent unknown
= 0.0 total rws_locks = 1466 percent reads = 100.0 total com‐
plex_locks = 2716146 percent reads = 33.2 percent unknown = 0.0
# The following lockinfo command gathers locking statistics for a par‐
ticular RAD on a NUMA platform. In this example, only cumulative sta‐
tistics across all CPUs in the RAD are displayed.
# /usr/sbin/lockinfo -rad=4 sleep 10 hostname: sys‐
name.node.corp.com locktype: MCS Locks lockmode: 2
(SMP default) Tracing: RAD 4 only processors: 4 start
time: Thu May 17 09:15:09 2001 end time: Thu May 17
09:15:19 2001 command: sleep 10
tries reads trmax misses misses sleeps
waitmax waitsum
percent sec‐
onds seconds processor.callout_lock (M)
1298 0 0 0 0.0 0
0.00000 0.00001 thread.lock (M)
336 0 0 0 0.0 0
0.00000 0.00000 processor.lock (M)
226 0 0 1 0.4 0
0.00000 0.00016 pag.idle_lock (M)
124 0 0 0 0.0 0
0.00000 0.00002 wait_queue.lock (M)
114 0 0 0 0.0 0
0.00000 0.00000 unknown_simple_lock (M)
76 0 0 0 0.0 0
0.00000 0.00000 misc_tcp_lock (M)
70 0 0 0 0.0 0
0.00000 0.00000 pag.lock (M)
10 0 0 0 0.0 0
0.00000 0.00000 cam_softc (M)
10 0 0 0 0.0 0
0.00000 0.00000 vm_control.vm_free_lock (M)
10 0 0 0 0.0 0
0.00000 0.00000 cam_x_pqhead1 (M)
8 0 0 0 0.0 0
0.00000 0.00000 pmap.lock (M)
6 0 0 0 0.0 0
0.00000 0.00000 proc.p_lock (M)
2 0 0 0 0.0 0
0.00000 0.00000 nxm_vp_lock (M)
2 0 0 0 0.0 0
0.00000 0.00000 task.ipc_translation_lock (M)
1 0 0 0 0.0 0
0.00000 0.00000 nxm_thread_lock (M)
1 0 0 0 0.0 0
0.00000 0.00000 kern_port.port_data_lock (M)
1 0 0 0 0.0 0
0.00000 0.00000 port_hash_bucket.lock (M)
1 0 0 0 0.0 0
0.00000 0.00000
total mcs_locks = 2296 percent unknown
= 0.0 total rws_locks = 0 percent reads = 0.0 total com‐
plex_locks = 0 percent reads = 0.0 percent unknown = 0.0
The following lockinfo command gathers locking statistics for a
particular RAD on a NUMA platform. In this example, per-CPU sta‐
tistics are included as well as the total for all CPUs in the
RAD.
# /usr/sbin/lockinfo -rad=4 -percpu sleep 10 hostname:
sysname.node.corp.com locktype: MCS Locks lockmode:
2 (SMP default) Tracing: RAD 4 only processors: 4
start time: Thu May 17 09:16:33 2001 end time: Thu May
17 09:16:43 2001 command: sleep 10
tries reads trmax misses misses sleeps wait‐
max waitsum
percent sec‐
onds seconds processor.callout_lock (M)
16 110 0 0 0 0.0 0
0.00000 0.00000
17 92 0 0 0 0.0 0
0.00000 0.00000
18 100 0 0 0 0.0 0
0.00000 0.00000
19 992 0 0 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 1294 0 0 0 0.0 0
0.00000 0.00003
thread.lock (M)
16 54 0 0 0 0.0 0
0.00000 0.00000
17 30 0 0 0 0.0 0
0.00000 0.00000
18 162 0 0 0 0.0 0
0.00000 0.00000
19 90 0 0 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 336 0 0 0 0.0 0
0.00000 0.00000
processor.lock (M)
16 37 0 0 0 0.0 0
0.00000 0.00000
17 18 0 0 0 0.0 0
0.00000 0.00000
18 108 0 0 0 0.0 0
0.00000 0.00000
19 63 0 0 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 226 0 0 0 0.0 0
0.00001 0.00022
pag.idle_lock (M)
16 21 0 0 0 0.0 0
0.00000 0.00000
17 12 0 0 0 0.0 0
0.00000 0.00000
18 54 0 0 0 0.0 0
0.00000 0.00000
19 37 0 0 1 2.7 0
0.00001 0.00001
-----------------------------------------------------------------------
ALL 124 0 0 1 0.8 0
0.00001 0.00003
wait_queue.lock (M)
16 21 0 0 0 0.0 0
0.00000 0.00000
17 2 0 0 0 0.0 0
0.00000 0.00000
18 54 0 0 0 0.0 0
0.00000 0.00000
19 37 0 0 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 114 0 0 0 0.0 0
0.00000 0.00000
unknown_simple_lock (M)
16 56 0 0 0 0.0 0
0.00000 0.00000
17 0 0 0 0 0.0 0
0.00000 0.00000
18 20 0 0 0 0.0 0
0.00000 0.00000
19 0 0 0 0 0.0 0
0.00000 0.00000
-----------------------------------------------------------------------
ALL 76 0 0 0 0.0 0
0.00000 0.00000
. . .
total mcs_locks = 2292 percent unknown
= 0.0 total rws_locks = 0 percent reads = 0.0 total com‐
plex_locks = 0 percent reads = 0.0 percent unknown = 0.0
FILES
Pseudo driver that is opened by the lockinfo utility for statistics
gathering.
SEE ALSO
Commands: sched_stat(8), sys_check(8)
Others: sys_attrs_generic(5)lockinfo(8)
