A program which raises a signal during normal usage can be tedious to debug, if the signal happens after a long period of normal execution.
For example, if the signal occurs during the execution of a routine which is called often, a breakpoint on that routine can be hit hundreds or thousands of times before the call which leads to the signal occurs.
There are several ways to fix this. One is to use a Ladebug
variable as a counter:
(ladebug) where
>0 0x3ff814855e0 in do_SEGVv ...
#1 :
:
(ladebug) set $count = 0
(ladebug) when quiet in do_SEGV { set $count = $count + 1 }
[#4: when quiet ...
(ladebug) run
Thread received signal SEGV
stopped at ....
(ladebug) p $count
21
(ladebug) stop in do_SEGV if $count >= 21
[2] : stop in void do_SEGV ...
(ladebug) set $count = 0;
(ladebug) r
[#2: stopped at [void do_SEGV...
:
Here do_SEGV is the leaf routine (the ultimate callee)
in which the the SEGV happened, or the user routine up the stack
just before a call into system libraries.
At the end of this sequence, you are in the routine which caused the signal, and in the actual instance of the call which is about to cause the signal.
Another way is to instrument your application so that it keeps track of call numbers on its own. This is particularly attractive if the program runs for a long time, as the approach above requires at least three runs: one not under debugger control, which make you realize that a signal was being raised; another to count the calls; and a third to get you ready to debug. The instrumentation will determine the necessary count on the first run, and you can debug on the second, instead of the third.
The "counter" tool is an ATOM-based tool which will instrument
your program. The inserted routines will count each call of
each routine, and can catch signals not handled by your application.
When it catches a signal, it will report the counts for recently-called
routines:
== Here's what you had recently executed when
== your application got signal 11 ('Segmentation fault')
File c_corefile.c, routine do_SEGV, call number 21
File c_corefile.c, routine other, call number 2
File c_corefile.c, routine middle, call number 1
== A set of breakpoints to get you close to the
== event which caused the signal would be
==
== set $count = 0
== when quiet in do_SEGV { set $count = $count + 1 }
== stop in do_SEGV if $count >= 21
==
The version here catches SIGSEGV, SIGCHLD and SIGSYS; it should be obvious how to modify the code to catch a different set.
The instrumented version of your application can be used in daily use or in testing. The instrumentation adds about 50% to the run-time of the application and 10 percent to the size of the image, which for many users may not matter.
Please note that "counter" is a prototype. If your application does something which makes normal "C" code unrunable, prevents I/O or destroys the signal-catching mechanism, then this tool will not work correctly.
This prototype is based on the "Ring Tool" prototype, but has not been as thoroughly tested.
Using the prototype requires the following steps:
$ atom -gp -A1 your-executable counter.inst.c \
counter.anal.c -o your-executable.counter
The processing will take a certain amount of time. For example,
to instrument Ladebug itself takes about thirty seconds on an Alpha server.
name_table.dat
which your instrumented application will need to function correctly.
The source code below assumes that file is in the current directory,
but can easily be modified so that the file could be somewhere else:
just change the macro NAME_TABLE_NAME in
counter.inst.c. This file contains the file and procedure
names for the instrumented procedures in your executable.
name_table.dat is available.
/* Instrument for traceback on signals.
*/
#include <stdio.h>
#include <string.h>
#include <cmplrs/atom.inst.h>
#include <regex.h>
#include <errno.h>
/* Define the name tables. The table holds pointers to strings,
* which is one "*"; it's an array, which is two, and this is a
* pointer to the table, for three "*" in a row.
*/
typedef struct name_table_struct {
int size;
const char ***file_names;
const char ***proc_names;
} name_table_t;
name_table_t name_table;
#define MAX_NAME_SIZE 300
#define NAME_TABLE_NAME "./name_table.dat"
/* Import the instrumentation patterns specified by the user. If none
* were specified, use the empty set defined in the template file.
*/
#ifdef DEB_COUNTER_PATTERNS_FILE
#include "counter.inst.patterns.h"
#else
#include "counter.inst.patterns.template.h"
#endif
#define DEB_COUNTER_REG_ERROR_MSG_LEN 512
static const int deb_counter_true = 1;
static const int deb_counter_false = 0;
static const char * deb_counter_last_file_name_skipped = NULL;
static int deb_counter_program_is_threaded = 0;
/* Set this to 1 to see debugging output
*/
static const int deb_counter_debug_flag = 0;
/* This procedure compares a string against an expression pattern.
* It return deb_counter_true (1) if the string matches the pattern
* and deb_counter_false (0) if it does not.
*/
static int deb_counter_expression_match(
const char *string,
const char *pattern)
{
int error;
regex_t preg;
char msg[DEB_COUNTER_REG_ERROR_MSG_LEN];
error = regcomp( &preg,
pattern,
REG_EXTENDED | REG_NOSUB );
if (error) {
regerror(error, &preg, msg, DEB_COUNTER_REG_ERROR_MSG_LEN);
printf("%s\n", msg);
return deb_counter_false;
}
if( 0 == regexec( &preg, string, 1, 0, 0 ) ) return deb_counter_true;
return deb_counter_false;
}
/* This procedure compares the name of a procedure with the known
* set of restrictor patterns in order to determine if instrumentation
* of said procedure should be skipped. It returns deb_counter_true (1)
* if the procedure should be skipped and deb_counter_false (0) if it
* should be instrumented.
*/
static int deb_counter_skip_procedure( const char *name )
{
int index = 0;
if( name == NULL ) return deb_counter_true;
/* Compare the procedure name against the specified expressions in
* the procedure patterns array.
*/
while( deb_counter_proc_patterns[index][0] != ';' ) {
if( deb_counter_expression_match( name, deb_counter_proc_patterns[index] ))
return deb_counter_true;
index = index + 1;
}
return deb_counter_false;
}
/* This procedure compares the name of a file with a known set of
* restrictor patterns in order to determine if instrumentation of
* procedures within said file procedure should be skipped. It returns
* deb_counter_true (1) if the file's procedures should be skipped and
* deb_counter_false (0) if they should be instrumented.
*/
static int deb_counter_skip_source_file( const char *name )
{
int index = 0;
/* Skip non-user routines, such as those in files not compiled with "-g",
* like our own calls to demangle names. We don't have to worry about
* our use of "printf", though, as printf is in another library ("Object"
* in ATOM terms).
*/
if( !name ) return deb_counter_true; /* No file name means no "-g" */
/* Because we call this more than once on the same file, we
* cache the file name pointer for faster compares.
*/
if( deb_counter_last_file_name_skipped == name ) return deb_counter_true;
while( deb_counter_file_patterns[index][0] != ';' ) {
if( deb_counter_expression_match( name, deb_counter_file_patterns[index] )) {
deb_counter_last_file_name_skipped = name;
return deb_counter_true;
}
index = index + 1;
}
return deb_counter_false;
}
int write_table_line( FILE *f, int index,
const char *file_name,
const char *proc_name )
{
static int am_ok = 1;
int result;
if (!am_ok) return 0;
if (!f) {
am_ok = 0;
printf( "== No file to write to\n" );
return 0;
}
if (file_name == NULL || proc_name == NULL) {
am_ok = 0;
printf( "== Names to write are null\n" );
return 0;
}
if (strlen( file_name ) > MAX_NAME_SIZE ||
strlen( proc_name ) > MAX_NAME_SIZE ) {
am_ok = 0;
printf( "== names are too long\n" );
return 0;
}
result = fprintf( f, "%d %s %s\n", index, file_name, proc_name );
return (result > 0);
}
void note_names( int proc_num, const char* filenm, const char* procname )
{
static FILE *f;
static int am_ok = 1;
if (!am_ok) return; /* Silent after first error */
if (!f) {
f = fopen( NAME_TABLE_NAME, "w" );
if (!f) {
am_ok = 0;
printf( "== Unable to open temporary data file, processing\n" );
printf( " will be incomplete. error was: %d\n", errno );
}
}
if (!write_table_line( f, proc_num, filenm, procname )) {
am_ok = 0;
}
}
unsigned InstrumentAll(int argc, char **argv)
{
Obj *o;
Proc *p;
Block *b;
Inst *i;
int proc_num;
/* Offer threaded programs a -toolargs=-threads option, to profile each
* thread independently.
*/
if (argc == 2 && strcmp(argv[1],"-threads") == 0) {
deb_counter_program_is_threaded = deb_counter_true;
}
/* Prototypes
*/
AddCallProto( "ProcTrace( int )" );
AddCallProto( "MainInit( int, int, const char* )" );
/* Traverse the "Objects" and instrument the application.
*/
proc_num = 0;
for (o = GetFirstObj(); o != NULL; o = GetNextObj(o)) {
const char* objName;
if (BuildObj(o)) return 1; /* Error */
objName = GetObjName(o);
if( !deb_counter_program_is_threaded ) {
if( objName &&
deb_counter_expression_match( objName, "libpthread.so$") ) {
printf(
"%s %s\n",
"Warning:",
"application appears to be multi-threaded, but the");
printf(
"%s %s\n",
" ",
"thread safe option was not specified.");
}
}
if( GetObjInfo(o, ObjModifyHint) == OBJ_READONLY) {
if( deb_counter_debug_flag ) {
printf( "Skipped instrumentation processing for object: %s\n",
( !objName ? "<unknown>" : objName ) );
}
ReleaseObj(o);
continue;
} else {
if( deb_counter_debug_flag ) {
printf( "Instrumenting object file: %s\n",
( !objName ? "<unknown>" : objName ) );
}
}
deb_counter_last_file_name_skipped = NULL;
for (p = GetFirstObjProc(o); p != NULL; p = GetNextProc(p)) {
const char *name = ProcName(p);
const char *filenm = ProcFileName(p);
if( name == NULL ) continue;
if( deb_counter_skip_source_file( filenm ) ||
deb_counter_skip_procedure ( name ) ) {
if( deb_counter_debug_flag ) {
printf( "Skipped proc: %s from file: %s\n",
name,
( !filenm ? "<unknown>" : filenm ) );
}
continue;
}
/* Count this procedure and build up the name vector.
*/
proc_num++; /* Index starts at 1! */
note_names( proc_num, filenm, name );
if( deb_counter_debug_flag ) {
printf( "*Instrument proc %d: %s from file: %s\n",
proc_num, name, filenm );
}
/* For user routines, count the calls.
*/
AddCallProc(p, ProcBefore, "ProcTrace", proc_num );
}
WriteObj( o );
deb_counter_last_file_name_skipped = NULL;
}
/* Add a call to initialize now that we know how
* many procedures there are.
*/
AddCallProgram( ProgramBefore, "MainInit",
deb_counter_program_is_threaded,
proc_num,
NAME_TABLE_NAME );
return(0);
}
/* Count routines and dump if asked.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <cmplrs/atom.anal.h>
#include <demangle.h>
#include <errno.h>
#include <signal.h>
/* Names and counts.
*/
static int proc_count = 0;
static int *counts = NULL;
const char **file_table_ptr = NULL;
const char **proc_table_ptr = NULL;
/* "Last-few" routines executed.
*/
#define LAST_FEW 100
static int last_index = 0;
static int last_few[LAST_FEW];
static char *no_file = "<unknown file>";
static char *no_proc = "<unknown proc>";
/* True and False values for prototype code
*/
static const int deb_counter_true = 1;
static const int deb_counter_false = 0;
/* Set this to 1 to turn on debug printing.
*/
#define DEBUGGING 0
/* Hold buffer for processed filenames
*/
#define WORKBUFLEN 128
static char file_name_work_buf[WORKBUFLEN];
/* NULL FILE pointer used in calls to fflush() to force writing
of output streams
*/
FILE * null_file_ptr = NULL;
#define MAX_NAME_SIZE 300
int read_table_line( FILE *f, int *index_ptr,
const char **file_name_ptr,
const char **proc_name_ptr )
{
char buf1[MAX_NAME_SIZE];
char buf2[MAX_NAME_SIZE];
char *n1;
char *n2;
int i;
int result;
int len;
static int am_ok = 1;
if (!am_ok) return;
if (!f) {
am_ok = 0;
printf( "== No file to read from\n" );
return 0;
}
if (index_ptr == NULL || file_name_ptr == NULL || proc_name_ptr == NULL) {
am_ok = 0;
printf( "== Pointers to write to are null\n" );
return 0;
}
result = fscanf( f, "%d %s %s", index_ptr, buf1, buf2 );
len = strlen( (char*)&buf1[0]);
n1 = (char *) malloc( len + 1);
for (i = 0; i < len; i++) {
n1[i] = buf1[i];
}
n1[len] = '\0';
len = strlen( (char*)&buf2[0]);
n2 = (char *) malloc( len + 1 );
for (i = 0; i < len; i++) {
n2[i] = buf2[i];
}
n2[len] = '\0';
*file_name_ptr = n1;
*proc_name_ptr = n2;
return 1;
}
int read_name_table( int proc_num, const char *name_table_name )
{
int table_index;
int i;
FILE *f = fopen( name_table_name, "r" );
if (!f) {
printf( "== Could not open name-table file, errno is %d\n",
errno );
return 0;
}
file_table_ptr = (const char **) malloc (proc_num * sizeof(const char *));
proc_table_ptr = (const char **) malloc (proc_num * sizeof(const char *));
if (DEBUGGING) {
printf( "-- About to read %d name table records\n", proc_num );
}
for (i = 1; i <= proc_num; i++ ){
file_table_ptr[i] = NULL;
proc_table_ptr[i] = NULL;
table_index = 0;
read_table_line( f, &table_index,
&file_table_ptr[i],
&proc_table_ptr[i] );
if (DEBUGGING) {
printf( "-- Record %d, \"%s\", \"%s\"\n",
table_index, file_table_ptr[i], proc_table_ptr[i] );
}
if (i != table_index) {
printf( "== Error in reading from table\n" );
return;
}
}
if (DEBUGGING)
printf( "Read %d table entries\n", proc_num );
}
int already_seen( int which )
{
static int already_been_seen[LAST_FEW];
static int next_seen = 0;
int i;
for( i = 0; i < next_seen; i++ ) {
if (already_been_seen[i] == which)
return 1; /* already seen */
}
/* Haven't seen this one before
*/
already_been_seen[next_seen++] = which;
return 0;
}
/* The heart of it all; tell the user the execution counts
* for recently-executed routines.
*/
void where_was_i( int sig )
{
int i;
int which;
int max_value;
int max_index;
printf( "\n\t== Here's what you had recently executed when" );
printf( "\n\t== your application got signal %d ('", sig );
printf( "%s')\n\n", __sys_siglist[sig] );
fflush(stdout);
max_value = 0;
max_index = 0;
for (i = last_index - 1; i >= 0; i-- ) {
which = last_few[i];
if (which != 0 /* Actual record exists */
&& counts[which] > 0 /* for which we have counts */
&& !already_seen( which )) { /* and it's not the N-1th call */
printf( "\t File %s, routine %s, call number %d\n",
file_table_ptr[which],
proc_table_ptr[which],
counts[which] );
if (counts[which] > max_value) {
max_value = counts[which];
max_index = which;
}
}
}
for (i = LAST_FEW - 1; i > last_index; i-- ) {
which = last_few[i];
if (which != 0 && counts[which] > 0 && !already_seen( which )) {
printf( "\t File %s, rtn %s, call number %d\n",
file_table_ptr[which],
proc_table_ptr[which],
counts[which] );
if (counts[which] > max_value) {
max_value = counts[which];
max_index = which;
}
}
}
if (max_value == 1) {
printf( "\n\t== A breakpoint to get you close to the event " );
printf( "\n\t== which caused the signal would be " );
printf( "\n\t== " );
printf( "\n\t== stop in %s", proc_table_ptr[max_index] );
} else if (max_value > 1) {
printf( "\n\t== A set of breakpoints to get you close to the" );
printf( "\n\t== event which caused the signal would be " );
printf( "\n\t== " );
printf( "\n\t== set $count = 0 " );
printf( "\n\t== when quiet in %s { set $count = $count + 1 }",
proc_table_ptr[max_index] );
printf( "\n\t== stop in %s if $count >= %d ",
proc_table_ptr[max_index], max_value );
}
printf ("\n\n");
fflush(stdout);
}
/* Catch, inform and resignal.
*/
struct sigaction old_sigaction;
void sig_handler(int sig, siginfo_t *sip, void *extra)
{
int ret;
where_was_i( sip->si_signo );
/* Restore the old signal action
*/
ret = sigaction( sig, &old_sigaction, 0);
if (ret) {
perror("sigaction");
exit(1);
}
/* Re-signal
*/
kill( getpid(), sig );
fflush(stdout);
}
/* Prepare to catch and resignal SIGSEGV, SIGCHLD and SIGSYS, could add others.
*/
void MainInit( int program_is_threaded,
int proc_num,
const char *name_table_name )
{
int i;
struct sigaction sa;
int ret;
/* Initialize counts.
*/
proc_count = proc_num;
counts = (int *) malloc ( (proc_num + 1) * sizeof( int * ));
for (i = 0; i <= proc_num; i++) {
counts[i] = 0;
}
for (i = 0; i < LAST_FEW; i++) {
last_few[i] = 0;
}
/* Initialize tables.
*/
read_name_table( proc_num, name_table_name );
/* Catch and resignal SIGSEGV
*/
sigemptyset(&sa.sa_mask);
sa.sa_sigaction = sig_handler;
sa.sa_flags = SA_CLDNOTIFY|SA_SIGINFO;
ret = sigaction( SIGSEGV, &sa, &old_sigaction );
if (ret) {
perror("sigaction");
exit(1);
}
/* Catch and resignal SIGCHLD
*/
sigemptyset(&sa.sa_mask);
sa.sa_sigaction = sig_handler;
sa.sa_flags = SA_CLDNOTIFY|SA_SIGINFO;
ret = sigaction( SIGCHLD, &sa, &old_sigaction );
if (ret) {
perror("sigaction");
exit(1);
}
/* Catch and resignal SIGSYS
*/
sigemptyset(&sa.sa_mask);
sa.sa_sigaction = sig_handler;
sa.sa_flags = SA_CLDNOTIFY|SA_SIGINFO;
ret = sigaction( SIGSYS, &sa, &old_sigaction );
if (ret) {
perror("sigaction");
exit(1);
}
}
/* Count a procedure's invocations in a low-overhead way.
*/
void ProcTrace( int my_index )
{
/* Count
*/
counts[ my_index ]++;
/* Remember this was called
*/
if (last_index >= LAST_FEW) last_index = 0;
last_few[last_index++] = my_index;
}
/* Now that the user cares, demangle a name.
*
* WARNING: this returns a pointer to a static buffer; do NOT
* save or write to the result.
*/
#define NAMELEN 200
#define NAMELEN_WITH_GUARD (NAMELEN+1)
static char *deb_counter_demangle_name( char *mangled_name )
{
static char demangled_name[ NAMELEN_WITH_GUARD ];
int status;
if( mangled_name == NULL )
return no_file;
#ifdef DEB_COUNTER_NO_DEMANGLE
/* Call to MLD_demangle_string() has proven unsafe to use with
* threads programs (segv).
*/
return mangled_name;
#else
demangled_name[NAMELEN] = (char)0; /* guard */
status = MLD_demangle_string( mangled_name,
demangled_name, NAMELEN,
MLD_SHOW_DEMANGLED_NAME );
return &demangled_name[0];
#endif
}
/* The following text describes which procedures are not instrumented in
* the Ladebug debugger and why they were excluded.
*
* Most memory management related procedures (malloc, free, realloc),
* including Ladebug's specialized memory management routines
* (ladebugMalloc*, ladebugRealloc* and ladebugFree*) are not instrumented
* as they have a dramatic impact on application performance.
*
* The procedure DebugCtl::operator void*(void) (__opPv__8DebugCtlXv) is
* not instrumented because this routine gets called when Ladebug "calls"
* the dump routines from the command line, before they can set the "quiet"
* flag. If we didn't do this, the history buffer would start filling up
* with entries for call/line 12/ret of this routine.
*
* Any procedures related to this tool (deb_counter*) are not instrumented
* as they are not interesting to the user.
*
* Any procedures for which the corresponding source file is not known are
* excluded from instrumentation as they offer little useful information
* for the user.
*
* Any procedures from software outside the Ladebug source pool, such as
* Rogue Wave, C++ name demangling, libdpi and Standard C++ routines.
*
* Any compiler generated procedures deemed generally uninteresting, such
* as interludes, initialization routines, etc. Compiler generated "throw"
* routines are instrumented as they are often interesting.
*
* Any Ladebug "give()" procedures are not instrumented as they are trivial
* in nature and thus uninteresting.
*/
#define DEB_COUNTER_MAX_PATTERN_LEN 100
/* Add source file name/directory path expressions to this array if you
* wish to have procedures from these areas skipped during instrumentation.
* The delimiter pattern (";") must be kept as the final entry.
*
* Note that the file/path names come from the application's Symbol Table
* info (man stdump), so the manner in which the application is compiled
* (relative vs. absolute path names) can have an effect on directory name
* matching.
*/
static char deb_counter_file_patterns[][DEB_COUNTER_MAX_PATTERN_LEN] = {
"/rogue-alpha-.*/rw/", /* Ladebug's Rogue Wave files */
"/libdpi/", /* Ladebug's libdpi usage */
"/usr/include/cxx/", /* C++ iostreams, etc. */
"alpha_unwind.c", /* System context unwind file */
"ots_packed_arith.c", /* Ladebug's libots usage */
"/cvt(ascii|num).c", /* Ladebug's libcvtas usage */
";" /* end of list delimiter */
};
/* Add procedure name expressions to this array if you wish to have
* procedures whose name contains the expression skipped during
* instrumentation. The delimiter pattern (";") must be kept as the
* final entry.
*
* Note that the procedure names come from the application's Symbol
* Table info (man stdump), so issues such as C++ name mangling must
* be addressed in these expressions.
*/
static char deb_counter_proc_patterns[][DEB_COUNTER_MAX_PATTERN_LEN] = {
"^__INTER__", /* C++ interlude procs */
"^__(fini|init)_sti__", /* Compiler generated procs */
"^__t.*ptrthunk__Xv$", /* C++ compiler generated procs */
"^free$", /* Standard C memory mgmt proc */
"^(m|c|re)alloc$", /* Standard C memory mgmt procs */
"^ots_", /* Ladebug's libots usage */
"^longdouble_", /* Libcvtas longdouble procs */
"^give__.*Xv$", /* Ladebug Ptr notation proc */
"^__opPv__8DebugCtlXv$", /* Ladebug DebugCtl operation */
"ladebug(Malloc|Free|Realloc)", /* Ladebug memory mgmt procs */
";" /* end of list delimiter */
};
/* This file provides a template for creating the counter.inst.patterns.h
* file used by the Ladebug counter tool. In the case where a user
* chooses not to create a counter.inst.patterns.h file, this file is
* used to provide a default (empty) definition.
*
* The counter.inst.patterns.h defines regular expressions which, when
* encoutered during instrumention of an application, are to be
* skipped (not instrumented).
* There are two sets of expressions that can be specified, those to be
* matched against filenames and those to be matched against procedure
* names.
*
* To create your own pattern description file for an application,
* copy this file to "counter.inst.patterns.h" and edit that file
* to add your own expressions.
*
* For an example of the expression definitions used when instrumenting
* the Ladebug debugger, see the file counter.inst.patterns.example.h.
*/
#define DEB_COUNTER_MAX_PATTERN_LEN 100
/* Add source file name/directory path expressions to this array if you
* wish to have procedures from these areas skipped during instrumentation.
* The delimiter pattern (";") must be kept as the final entry.
*
* Note that the file/path names come from the application's Symbol Table
* info (man stdump), so the manner in which the application is compiled
* (relative vs. absolute path names) can have an effect on directory name
* matching.
*/
static char deb_counter_file_patterns[][DEB_COUNTER_MAX_PATTERN_LEN] = {
";" /* end of list delimiter */
};
/* Add procedure name expressions to this array if you wish to have
* procedures whose name contains the expression skipped during
* instrumentation. The delimiter pattern (";") must be kept as the
* final entry.
*
* Note that the procedure names come from the application's Symbol
* Table info (man stdump), so issues such as C++ name mangling must
* be addressed in these expressions.
*/
static char deb_counter_proc_patterns[][DEB_COUNTER_MAX_PATTERN_LEN] = {
";" /* end of list delimiter */
};
Click here to send comments or problems to the Ladebug team.