Tuning P4 Performance

There are several enviroment variables and command line options that can be used to tune the performance of the ch_p4 device. Note that these environment variables must be defined for all processes that are created, not just the process that you are launching MPI programs from (i.e., setting these variables should be part of your .login or .cshrc startup files). The environment variables are:

{ P4_SOCKBUFSIZE}.

Specifies the socket buffer size in bytes. Increasing this value can improve performance on some system.

{ P4_WINSHIFT}.

This is another socket parameter that is supported on only a few platforms. We recommend leaving it alone.

{ P4_GLOBMEMSIZE}.

This is the amount of memory in bytes reserved for communication with shared memory (when mpich is configured with -comm=shared). Increase this if you get error messages about p4_shmalloc returning NULL.

{ P4_RSHCOMMAND}.

This specifies the name of the remote shell command that is used to start new processes (as part of MPI_Init). If no value is given, the remote shell command determined by configure will be used. If a program name is given, it must accept the same command-line arguments as the program selected by configure.

TCP Tuning. The command line option -p4sctrl takes a parameter that specifies various socket options. These are in the form of name=value, separated by colons. With the exception of bufsize, users should normally not change the defaults for these. The names and their meanings are

bufsize

Socket buffer size, in 1K (1024) bytes. E.g., bufsize=32 requests 32K byte socket buffers. The default is 16.

winsize

Winshift size. Available only on systems that define TCP_WINSHFT and is ignored otherwise.

netsendw

Use select to wait for write to proceed. Values are y (default) or n.

netreadw

Use select to wait for read to proceed. Values are y (default) or n.

writev

Use writev to send the header (MPI envelope) and data in a single message. Values are y (default) or n.

readb

Switch a socket to blocking mode while wait to read instead of busy-waiting or using select. Values are y or n (default).

stat

Print out statistics on write and read operations. Useful only for experts!

    mpirun -np 2 mpptest -p4sctrl bufsize=64:writev=n 

SMP Tuning By default, the ch_p4 device uses the sched_yield system call (if it is available) to avoid spin waiting when in a blocking receive. You can override this with the --disable-yield to configure. See http://www.mcs.anl.gov/mpi/mpich1/micronotes/yield for some graphs showing the effect of using (or not using) sched_yield.