Running programs with mpirun

To run an MPI program, use the mpirun command, which is located in /usr/local/mpich-1.2.5/bin. For almost all systems, you can use the command

    mpirun -np 4 a.out 

On exit, mpirun returns the status of one of the processes, usually the process with rank zero in MPI_COMM_WORLD.

Running mpich Jobs under MPD

Because the MPD daemons are already in communication with one another before the job starts, job startup is much faster than with the ch_p4 device. The program mpirun runs in a separate (non-MPI) process that starts the MPI processes running the specified executable. It serves as a single-process representative of the parallel MPI processes in that signals sent to it, such as ^Z and ^C are conveyed by the MPD system to all the processes. The output streams stdout and stderr from the MPI processes are routed back to the stdout and stderr of mpirun. As in most MPI implementations, mpirun's stdin is routed to the stdin of the MPI process with rank 0. The mpirun command for the ch_p4mpd device has a number of special command-line arguments.

-np <num>

Run the program with <num> processes. This will be the size of MPI_COMM_WORLD at startup. The machines on which the processes will run are determined by the ring of MPD daemons.

-s

Close stdin. This allows the mpirun command to be run in the background without suspending in a way for tty input.

-g <size>

Start <size> processes under each MPD daemon, up until the number of processes specified with the -np argument have been started.

-m <file>

Treat <file> as a list of machines on which to start the processes. It is important that MPD daemons already be running on those machines. An alternate method is to use the -MPDLOC- argument. If you run a job with

       mpirun -np 8 a.out -MPDLOC- node1 node2 node3 node4 
  

then two copies of a.out will be run on each of the specified machines. For large numbers of machines, a compact form can be used. For example

       mpirun -np 80 a.out -MPDLOC- node%d:1-40,61-100 
  

runs on the machines node1 ... node40, node61, ... node100.

-wdir <directory>

Make <directory> the working directory for the application processes.

-jidfile <file>

Write the job id for the job into <file>.

-whole

The collection and merging of stdout is by default done in a way that maximizes efficiency. The disadvantage is that sometimes lines of output from different processes can be interleaved on the same line. With this argument the user guarantees that all program output sent to stdout is followed by a newline character, and especially that the last line of output is terminated by a newline character. In this case the processes that collect output will wait for the newlines before forwarding output, thus ensuring that lines are kept separate. If this argument is specified and the condition is not met, then the program may hang.

-l

Label lines of stdout and stderr output with the rank of the process that does it. Either with this option or without it, all stdout and stderr output from the application processes is routed back to the stdout and stderr of the mpirun process itself.

-1

By default, the first application process is started on the same node that mpirun is run on. With this option the first process is started on the machine where the next MPD daemon in the ring is running. This option allows you, for example, to run mpirun on a ``login'' or ``development'' node on your cluster but to start all the application processes on ``computation'' nodes.

-y

Perform certain initialization required by a job using Myrinet for communication inside the cluster.

% mpirun 
Usage: mpirun <args> executable <args_to_executable> 
Arguments are: 
  [-np num_processes_to_run] (required)  
  [-help] (this message) 
  [-s]  (close stdin; can run in bkgd w/o tty input problems) 
  [-g group_size]  (start group_size procs per mpd) 
  [-m machine_file]  (filename for allowed machines) 
  [-l]  (line labels; unique id for each process's output 
  [-1]  (do NOT start first process locally) 
  [-y]  (run as Myrinet job) 
  [-whole]  (stdout is guaranteed to stay in whole lines) 
  [-wdir dirname] (set wording directory for application) 
  [-jidfile file] (place job id in file file) 

Passing Environment Variables to Programs under MPD

By default, the mpirun program only passes a few environment variables to the program. This speeds program startup, particularly since in most cases, few applications require environment variables. However, for those that do (including the MPE_LOG_FORMAT variable used to control the format of MPE logfiles), mpirun supports the command line argument -MPDENV-. This causes mpirun and MPD to start processes with the environment variables listed after -MPDENV-. For example,

    mpirun -np N cpilog -MPDENV- MPE_LOG_FORMAT=SLOG