Displaying Trace Data

Once you determine the consumer of interest, you can retrieve the data corresponding to any unconsumed buffers by specifying the address of the state structure to the ::dtrace dcmd. The following example shows the output of the ::dtrace dcmd on an anonymous enabling of syscall:::entry with the action trace(execname):

> ::dtrace_state
    ADDR MINOR     PROC NAME                 FILE
cbfb7a40     2        - <anonymous>             -

> cbfb7a40::dtrace
CPU     ID                    FUNCTION:NAME
  0    344                resolvepath:entry   init                           
  0     16                      close:entry   init                           
  0    202                      xstat:entry   init                           
  0    202                      xstat:entry   init                           
  0     14                       open:entry   init                           
  0    206                     fxstat:entry   init                           
  0    186                       mmap:entry   init                           
  0    186                       mmap:entry   init                           
  0    186                       mmap:entry   init                           
  0    190                     munmap:entry   init                           
  0    344                resolvepath:entry   init                           
  0    216                    memcntl:entry   init                           
  0     16                      close:entry   init                           
  0    202                      xstat:entry   init                           
  0     14                       open:entry   init                           
  0    206                     fxstat:entry   init                           
  0    186                       mmap:entry   init                           
  0    186                       mmap:entry   init                           
  0    186                       mmap:entry   init                           
  0    190                     munmap:entry   init
...

The ::dtrace dcmd handles errors in the same way that dtrace ( 1M ) does: if drops, errors, speculative drops, or the like were encountered while the consumer was executing, ::dtrace will emit a message corresponding to the dtrace ( 1M ) message.

The order of events as displayed by ::dtrace is always oldest to youngest within a given CPU. The CPU buffers themselves are displayed in numerical order. If an ordering is required for events on different CPUs, trace the timestamp variable.

You can display only the data for a specific CPU by specifying the c option to ::dtrace:

> cbfb7a40::dtrace -c 1
CPU     ID                    FUNCTION:NAME
  1     14                       open:entry   init
  1    206                     fxstat:entry   init
  1    186                       mmap:entry   init
  1    344                resolvepath:entry   init
  1     16                      close:entry   init
  1    202                      xstat:entry   init
  1    202                      xstat:entry   init
  1     14                       open:entry   init
  1    206                     fxstat:entry   init 
  1    186                       mmap:entry   init  
...

Notice that ::dtrace only processes in-kernel DTrace data. Data that has been consumed from the kernel and processed (through dtrace ( 1M ) or other means) will not be available to be processed with ::dtrace. To assure that the most amount of data possible is available at the time of failure, use a ring buffer buffering policy. See Chapter 11, Buffers and Buffering for more information on buffer policies.

The following example creates a very small (16K) ring buffer and records all system calls and the process making them:

# dtrace -P syscall'{trace(curpsinfo->pr_psargs)}' -b 16k -x bufpolicy=ring
dtrace: description 'syscall:::entry' matched 214 probes

Looking at a crash dump taken when the above command was running results in output similar to the following example:

> ::dtrace_state
    ADDR MINOR     PROC NAME                 FILE
cdccd400     3 d15e80a0 dtrace           ced065f0

> cdccd400::dtraceCPU     ID                    FUNCTION:NAME
  0    139                    getmsg:return   mibiisa -r -p 25216            
  0    138                     getmsg:entry   mibiisa -r -p 25216            
  0    139                    getmsg:return   mibiisa -r -p 25216            
  0    138                     getmsg:entry   mibiisa -r -p 25216            
  0    139                    getmsg:return   mibiisa -r -p 25216            
  0    138                     getmsg:entry   mibiisa -r -p 25216            
  0    139                    getmsg:return   mibiisa -r -p 25216            
  0    138                     getmsg:entry   mibiisa -r -p 25216            
  0    139                    getmsg:return   mibiisa -r -p 25216            
  0    138                     getmsg:entry   mibiisa -r -p 25216            
  0     17                     close:return   mibiisa -r -p 25216            
...
  0     96                      ioctl:entry   mibiisa -r -p 25216            
  0     97                     ioctl:return   mibiisa -r -p 25216            
  0     96                      ioctl:entry   mibiisa -r -p 25216            
  0     97                     ioctl:return   mibiisa -r -p 25216            
  0     96                      ioctl:entry   mibiisa -r -p 25216            
  0     97                     ioctl:return   mibiisa -r -p 25216            
  0     96                      ioctl:entry   mibiisa -r -p 25216            
  0     97                     ioctl:return   mibiisa -r -p 25216            
  0     16                      close:entry   mibiisa -r -p 25216            
  0     17                     close:return   mibiisa -r -p 25216            
  0    124                   lwp_park:entry   mibiisa -r -p 25216            
  1     68                     access:entry   mdb -kw                        
  1     69                    access:return   mdb -kw                        
  1    202                      xstat:entry   mdb -kw                        
  1    203                     xstat:return   mdb -kw                        
  1     14                       open:entry   mdb -kw                        
  1     15                      open:return   mdb -kw                        
  1    206                     fxstat:entry   mdb -kw                        
  1    207                    fxstat:return   mdb -kw                        
  1    186                       mmap:entry   mdb -kw                        
...
  1     13                     write:return   mdb -kw                        
  1     10                       read:entry   mdb -kw                        
  1     11                      read:return   mdb -kw                        
  1     12                      write:entry   mdb -kw                        
  1     13                     write:return   mdb -kw                        
  1     96                      ioctl:entry   mdb -kw                        
  1     97                     ioctl:return   mdb -kw                        
  1    364                    pread64:entry   mdb -kw                        
  1    365                   pread64:return   mdb -kw                        
  1    366                   pwrite64:entry   mdb -kw                        
  1    367                  pwrite64:return   mdb -kw                        
  1    364                    pread64:entry   mdb -kw                        
  1    365                   pread64:return   mdb -kw                        
  1     38                        brk:entry   mdb -kw                        
  1     39                       brk:return   mdb -kw                        
>

Note that CPU 1's youngest records include a series of write ( 2 ) system calls by an mdb -kw process. This result is likely related to the reason for the system failure because a user can modify running kernel data or text with mdb ( 1 ) when run with the k and w options. In this case, the DTrace data provides at least an interesting avenue of investigation, if not the root cause of the failure.