38.5. Analyzing the Data

38.5. Analyzing the Data

38.5. Analyzing the Data

Periodically, the OProfile daemon, oprofiled, collects the samples and writes them to the /var/lib/oprofile/samples/ directory. Before reading the data, make sure all data has been written to this directory by executing the following command as root:

opcontrol --dump

Each sample file name is based on the name of the executable. For example, the samples for the default event on a Pentium III processor for /bin/bash becomes:

\{root\}/bin/bash/\{dep\}/\{root\}/bin/bash/CPU_CLK_UNHALTED.100000

The following tools are available to profile the sample data once it has been collected:

Use these tools, along with the binaries profiled, to generate reports that can be further analyzed.

Warning

The executable being profiled must be used with these tools to analyze the data. If it must change after the data is collected, backup the executable used to create the samples as well as the sample files. Please note that the sample file and the binary have to agree. Making a backup isn't going to work if they do not match. oparchive can be used to address this problem.

Samples for each executable are written to a single sample file. Samples from each dynamically linked library are also written to a single sample file. While OProfile is running, if the executable being monitored changes and a sample file for the executable exists, the existing sample file is automatically deleted. Thus, if the existing sample file is needed, it must be backed up, along with the executable used to create it before replacing the executable with a new version. The oprofile analysis tools use the executable file that created the samples during analysis. If the executable changes the analysis tools will be unable to analyze the associated samples. Refer to Section 38.4, “Saving Data” for details on how to backup the sample file.

38.5.1. Using opreport

The opreport tool provides an overview of all the executables being profiled.

The following is part of a sample output:


Profiling through timer interrupt           
TIMER:0|  
samples|      %| 
------------------     
25926 97.5212 no-vmlinux
359  1.3504 pi        
65  0.2445 Xorg        
62  0.2332 libvte.so.4.4.0        
56  0.2106 libc-2.3.4.so        
34  0.1279 libglib-2.0.so.0.400.7        
19  0.0715 libXft.so.2.1.2        
17  0.0639 bash         
8  0.0301 ld-2.3.4.so         
8  0.0301 libgdk-x11-2.0.so.0.400.13         
6  0.0226 libgobject-2.0.so.0.400.7         
5  0.0188 oprofiled         
4  0.0150 libpthread-2.3.4.so         
4  0.0150 libgtk-x11-2.0.so.0.400.13         
3  0.0113 libXrender.so.1.2.2         
3  0.0113 du         
1  0.0038 libcrypto.so.0.9.7a         
1  0.0038 libpam.so.0.77         
1  0.0038 libtermcap.so.2.0.8         
1  0.0038 libX11.so.6.2         
1  0.0038 libgthread-2.0.so.0.400.7         
1  0.0038 libwnck-1.so.4.9.0


Each executable is listed on its own line. The first column is the number of samples recorded for the executable. The second column is the percentage of samples relative to the total number of samples. The third column is the name of the executable.

Refer to the opreport man page for a list of available command line options, such as the -r option used to sort the output from the executable with the smallest number of samples to the one with the largest number of samples.

38.5.2. Using opreport on a Single Executable

To retrieve more detailed profiled information about a specific executable, use opreport:

opreport <mode> <executable>

<executable> must be the full path to the executable to be analyzed. <mode> must be one of the following:

-l

List sample data by symbols. For example, the following is part of the output from running the command opreport -l /lib/tls/libc-<version>.so:


samples  %        symbol name 
12       21.4286  __gconv_transform_utf8_internal 
5         8.9286  _int_malloc 
4         7.1429  malloc 
3         5.3571  __i686.get_pc_thunk.bx 
3         5.3571  _dl_mcount_wrapper_check 
3         5.3571  mbrtowc 
3         5.3571  memcpy 
2         3.5714  _int_realloc 
2         3.5714  _nl_intern_locale_data 
2         3.5714  free 
2         3.5714  strcmp 
1         1.7857  __ctype_get_mb_cur_max 
1         1.7857  __unregister_atfork 
1         1.7857  __write_nocancel 
1         1.7857  _dl_addr 
1         1.7857  _int_free 
1         1.7857  _itoa_word 
1         1.7857  calc_eclosure_iter 
1         1.7857  fopen@@GLIBC_2.1 
1         1.7857  getpid 
1         1.7857  memmove 
1         1.7857  msort_with_tmp 
1         1.7857  strcpy 
1         1.7857  strlen 
1         1.7857  vfprintf 
1         1.7857  write

The first column is the number of samples for the symbol, the second column is the percentage of samples for this symbol relative to the overall samples for the executable, and the third column is the symbol name.

To sort the output from the largest number of samples to the smallest (reverse order), use -r in conjunction with the -l option.

-i <symbol-name>

List sample data specific to a symbol name. For example, the following output is from the command opreport -l -i __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:


samples  %        symbol name 
12       100.000  __gconv_transform_utf8_internal

The first line is a summary for the symbol/executable combination.

The first column is the number of samples for the memory symbol. The second column is the percentage of samples for the memory address relative to the total number of samples for the symbol. The third column is the symbol name.

-d

List sample data by symbols with more detail than -l. For example, the following output is from the command opreport -l -d __gconv_transform_utf8_internal /lib/tls/libc-<version>.so:


vma      samples  %        symbol name 
00a98640 12       100.000  __gconv_transform_utf8_internal 
00a98640 1         8.3333 
00a9868c 2        16.6667 
00a9869a 1         8.3333 
00a986c1 1         8.3333 
00a98720 1         8.3333 
00a98749 1         8.3333 
00a98753 1         8.3333 
00a98789 1         8.3333 
00a98864 1         8.3333 
00a98869 1         8.3333 
00a98b08 1         8.3333

The data is the same as the -l option except that for each symbol, each virtual memory address used is shown. For each virtual memory address, the number of samples and percentage of samples relative to the number of samples for the symbol is displayed.

-x <symbol-name>

Exclude the comma-separated list of symbols from the output.

session:<name>

Specify the full path to the session or a directory relative to the /var/lib/oprofile/samples/ directory.

38.5.3. Getting more detailed output on the modules

OProfile collects data on a system-wide basis for kernel- and user-space code running on the machine. However, once a module is loaded into the kernel, the information about the origin of the kernel module is lost. The module could have come from the initrd file on boot up, the directory with the various kernel modules, or a locally created kernel module. As a result when OProfile records sample for a module, it just lists the samples for the modules for an executable in the root directory, but this is unlikely to be the place with the actual code for the module. You will need to take some steps to make sure that analysis tools get the executable.

For example on an AMD64 machine the sampling is set up to record "Data cache accesses" and "Data cache misses" and assuming you would like to see the data for the ext3 module:

$ opreport /ext3
CPU: AMD64 processors, speed 797.948 MHz (estimated)
Counted DATA_CACHE_ACCESSES events (Data cache accesses) with a unit mask of 0x00 (No unit mask) count 500000
Counted DATA_CACHE_MISSES events (Data cache misses) with a unit mask of 0x00 (No unit mask) count 500000
DATA_CACHE_ACC...|DATA_CACHE_MIS...|
samples|      %|  samples|      %|
------------------------------------
148721 100.000      1493 100.000 ext3

To get a more detailed view of the actions of the module, you will need to either have the module unstripped (e.g. installed from a custom build) or have the debuginfo RPM installed for the kernel.

Find out which kernel is running, "uname -a", get the appropriate debuginfo rpm, and install on the machine.

Then make a symbolic link so oprofile finds the code for the module in the correct place:

          # ln -s   /lib/modules/`uname -r`/kernel/fs/ext3/ext3.ko /ext3
        

Then the detailed information can be obtained with:

# opreport image:/ext3 -l|more
warning: could not check that the binary file /ext3 has not been modified since the profile was taken. Results may be inaccurate.
CPU: AMD64 processors, speed 797.948 MHz (estimated)
Counted DATA_CACHE_ACCESSES events (Data cache accesses) with a unit mask of 0x00 (No unit mask) count 500000
Counted DATA_CACHE_MISSES events (Data cache misses) with a unit mask of 0x00 (No unit mask) count 500000
samples  %        samples  %        symbol name
16728    11.2479  7         0.4689  ext3_group_sparse
16454    11.0637  4         0.2679  ext3_count_free_blocks
14583     9.8056  51        3.4159  ext3_fill_super
8281      5.5681  129       8.6403  ext3_ioctl
7810      5.2514  62        4.1527  ext3_write_info
7286      4.8991  67        4.4876  ext3_ordered_writepage
6509      4.3767  130       8.7073  ext3_new_inode
6378      4.2886  156      10.4488  ext3_new_block
5932      3.9887  87        5.8272  ext3_xattr_block_list
...

38.5.4. Using opannotate

The opannotate tool tries to match the samples for particular instructions to the corresponding lines in the source code. The resulting files generated should have the samples for the lines at the left. It also puts in a comment at the beginning of each function listing the total samples for the function.

For this utility to work, the executable must be compiled with GCC's -g option. By default, Red Hat Enterprise Linux packages are not compiled with this option.

The general syntax for opannotate is as follows:

opannotate --search-dirs <src-dir> --source <executable>

The directory containing the source code and the executable to be analyzed must be specified. Refer to the opannotate man page for a list of additional command line options.