In order to use WS GRAM, the container must be started with Transport Level security. The "-nosec" option should *not* be used with globus-start-container.

WS GRAM requires that the sudo command is installed and functioning on the service host where WS GRAM software will execute.

Authorization rules will need to be added to the sudoers file to allow the WS GRAM service account to execute (without a password) the scheduler adapter in the accounts of authorized GRAM users. For configuration details, see the Configuring sudo section.

Platform Note: On AIX, sudo is not installed by default, but it is available as source and rpm here: AIX 5L Toolbox for Linux Applications

WS GRAM depends on a local mechanism for starting and controlling jobs. Included in the WS GRAM software is a Fork scheduler, which requires no special software installed to execute jobs on the local host. However, to enable WS GRAM to execute and manage jobs to a batch scheduler, the scheduler software must be installed and configured prior to configuring WS GRAM.

WS GRAM depends on scheduler adapters to translate the WS GRAM job description document into commands understood by the local scheduler, as well as monitor the jobs.

Scheduler adapters included in the GT 4.0 release are: PBS, Condor, LSF

Other third party scheduler adapters available for GT 4.0.x releases:

For configuration details, see the Configuring scheduler adapters section.

Though staging directives are processed by RFT (see next section), RFT uses GridFTP servers underneath to do the actual data movement. As a result, there must be at least one GridFTP server that shares a file system with the execution nodes. There is no special process to get staged files onto the execution node before the job executable is run. See the Non-default GridFTP server section of this admin guide for details on how to configure WS GRAM for your GridFTP servers used in your execution environment.

WS GRAM depends on RFT to perform file staging and cleanup directives in a job description. For configuration details, see the RFT admin guide Important: Jobs requesting these functions will fail if RFT is not properly setup.

When the credentials of the service account and the job submitter are different (multi user mode), then GRAM will prepend a call to sudo to the local adapter callout command. Important: If sudo is not configured properly, the command and thus job will fail.

As root, here are the two lines to add to the /etc/sudoers file for each GLOBUS_LOCATION installation, where /opt/globus/GT4.0.0 should be replaced with the GLOBUS LOCATION for your installation:

# Globus GRAM entries
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.0.0/libexec/globus-gridmap-and-execute 
                -g /etc/grid-security/grid-mapfile
                /opt/globus/GT4.0.0/libexec/globus-job-manager-script.pl *
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.0.0/libexec/globus-gridmap-and-execute 
                -g /etc/grid-security/grid-mapfile
                /opt/globus/GT4.0.0/libexec/globus-gram-local-proxy-tool *
            

The globus-gridmap-and-execute program is used to ensure that GRAM only runs programs under accounts that are in the grid-mapfile. In the sudo configuration, it is the first program called. It looks up the account in the grid-mapfile and then runs the requested command. It is redundant if sudo is properly locked down. This tool could be replaced with your own authorization program.

The WS GRAM scheduler adapters included in the release tarball are: PBS, Condor and LSF. To install, follow these steps (shown for pbs):

    % cd $GLOBUS_LOCATION\gt4.0.0-all-source-installer
                
                % make gt4-gram-pbs
                
                % make install
                
            

Using PBS as the example, make sure the scheduler commands are in your path (qsub, qstat, pbsnodes).

For PBS, another setup step is required to configure the remote shell for rsh access:

                
                % cd $GLOBUS_LOCATION/setup/globus
                
                % ./setup-globus-job-manager-pbs --remote-shell=rsh
                
            

The last thing is to define the GRAM and GridFTP file system mapping for PBS. A default mapping in this file is created to allow simple jobs to run. However, the actual file system mappings for your compute resource should be entered to ensure:

Done! You have added the PBS scheduler adapters to your GT installation.

Note for future GT builds with scheduler adapters: scheduler adapters can be enabled by adding --enable-wsgram-pbs to the configure line when building the entire toolkit.

                % configure --prefix=$GLOBUS_LOCATION --enable-wsgram-pbs ...
                % make
                % make install
            

To run the container using just a user proxy, instead of host creds, edit the $GLOBUS_LOCATION/etc/globus_wsrf_core/global_security_descriptor.xml file, and either comment out the credentials section...

                <?xml version="1.0" encoding="UTF-8"?>
                <securityConfig xmlns="http://www.globus.org">
                <!--
                <credential>
                <key-file value="/etc/grid-security/containerkey.pem"/>
                <cert-file value="/etc/grid-security/containercert.pem"/>
                <credential>
                -->
                <gridmap value="/etc/grid-security/grid-mapfile"/>
                <securityConfig>
            

or replace the credentials section with a proxy file location...

                <?xml version="1.0" encoding="UTF-8"?>
                <securityConfig xmlns="http://www.globus.org">
                <proxy-file value="<PATH TO PROXY FILE>"/>
                <gridmap value="/etc/grid-security/grid-mapfile"/>
                <securityConfig>
            

Running in personal mode (user proxy), another GRAM configuration setting is required. For GRAM to authorize the RFT service when performing staging functions, it needs to know the subject DN for verification. Here are the steps:

                % cd $GLOBUS_LOCATION/setup/globus
                % ./setup-gram-service-common --staging-subject=
                "/DC=org/DC=doegrids/OU=People/CN=Stuart Martin 564720"
            

You can get your subject DN by running this command:

                % grid-cert-info -subject
            

By default, the GridFTP server is assumed to run as root on localhost:2811. If this is not true for your site then change it by editing the GridFTP host and/or port in the GRAM and GridFTP file system mapping config file: $GLOBUS_LOCATION/etc/gram-service/globus_gram_fs_map_config.xml.

By default, the globus services will assume 8443 is the port the Globus container is using. However the container can be run under a non-standard port, for example:

                % globus-start-container -p 4321
            

If you wish to specify a non-standard gridmap file in a multi-user installation, two basic configurations need to be changed:

Example: global_security_descriptor.xml

                ...
                
                <gridmap value="/opt/grid-mapfile"/>
                
                ...
            

sudoers

                ...
                
                # Globus GRAM entries
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.0.0/libexec/globus-gridmap-and-execute 
                -g /opt/grid-mapfile
                /opt/globus/GT4.0.0/libexec/globus-job-manager-script.pl *
                globus  ALL=(username1,username2) 
                NOPASSWD: /opt/globus/GT4.0.0/libexec/globus-gridmap-and-execute 
                -g /opt/grid-mapfile
                /opt/globus/GT4.0.0/libexec/globus-gram-local-proxy-tool *
                
                ...
            

RFT is used by GRAM to stage files in and out of the job execution environment. In the default configuration, RFT is hosted in the same container as GRAM and is assumed to have the same service path and standard service names. This need not be the case. For example, the most likely alternative scenario is that RFT would be hosted seperately in a container on a different machine. In any case, both the RFT and the Delegation Service endpoints need to be adjustable to allow this flexibility. The following options can be passed to the setup-gram-service-common script to affect these settings:

            --staging-protocol=<protocol>
            --staging-host=<host>
            --staging-port=<port>
            --staging-service-path=<RFT and Delegation factory service path>
            --staging-factory-name=<RFT factory service name>
            --staging-delegation-factory-name=<name of Delegation factory service used by RFT>
        

for example

            % setup-gram-service-common \
            --staging-protocol=http
            --staging-host=somemachine.fakedomain.net
            --staging-port=8444
            --staging-service-path=/tomcat/services/
            --staging-factory-name=MyReliableFileTransferFactoryService
            --staging-delegation-factory-name=MyDelegationFactoryServiceForRFT
        

will internally cause the GRAM service code to construct the following EPR addresses:

            http://somemachine.fakedomain.net:8444/tomcat/services/MyReliableFileTransferFactoryService
            
            http://somemachine.fakedomain.net:8444/tomcat/services/MyDelegationFactoryServiceForRFT
        

The file $GLOBUS_LOCATION/etc/gram-service/jndi-config.xml contains configuration information that is common to every local resource manager.

More precisely, the configuration data it contains pertains to the implementation of the GRAM WSRF resources (factory resources and job resources), as well as initial values of WSRF resource properties that are always published by any Managed Job Factory WSRF resource.

The data is categorized by service, because according to WSRF, in spite of the service/resource separation of concern, a given service will use only one XML Schema type of resource. In practice it is therefore clearer to categorize the configuration resource implementation by service, even if theoretically speaking a given resource implementation could be used by several services. For more information, refer to the Java WS Core documentation.

Here is the decomposition, in JNDI objects, of the common configuration data, categorized by service. Each XYZHome object contains the same Globus Core-defined information for the implementation of the WSRF resource, such as the Java implementation class for the resource (resourceClass datum), the Java class for the resource key (resourceKeyType datum), etc.

When a SOAP call is made to a GRAM factory service in order to submit a job, the call is actually made to a GRAM service-resource pair, where the factory resource represents the local resource manager to be used to execute the job.

There is one directory gram-service-<manager>/ for each local resource manager supported by the GRAM installation.

For instance, let's assume the command line:

% ls etc | grep gram-service-

gives the following output:

gram-service-Fork
gram-service-LSF
gram-service-Multi

In this example, the Multi, Fork and LSF job factory resources have been installed. Multi is a special kind of local resource manager which enables the GRAM services to support multijobs.

The JNDI configuration file located under each manager directory contains configuration information for the GRAM support of the given local resource manager, such as the name that GRAM uses to designate the given resource manager. This is referred to as the GRAM name of the local resource manager.

For instance, $GLOBUS_LOCATION/etc/gram-service-Fork/jndi-config.xml contains the following XML element structure:

    <service name="ManagedJobFactoryService">
        <!-- LRM configuration:  Fork -->
        <resource
            name="ForkResourceConfiguration"
            type="org.globus.exec.service.factory.FactoryResourceConfiguration">
            <resourceParams>
                [...]
                <parameter>
                    <name>
                        localResourceManagerName
                    </name>
                    <value>
                        Fork
                    </value>
                </parameter>           
                <!-- Site-specific scratchDir
Default: ${GLOBUS_USER_HOME}/.globus/scratch
                <parameter>
                    <name>
                        scratchDirectory
                    </name>
                    <value>
                        ${GLOBUS_USER_HOME}.globus/scratch
                    </value>
                </parameter>           
                -->
            </resourceParams>
        </resource>        
    </service>

In the example above, the name of the local resource manager is Fork. This value can be used with the GRAM command line client in order to specify which factory resource to use when submitting a job. Similarly, it is used to create an endpoint reference to the chosen factory WS-Resource when using the GRAM client API.

In the example above, the scratchDirectory is set to ${GLOBUS_USER_HOME}/.globus/scratch. This is the default setting. It can be configured to point to an alternate file system path that is common to the compute cluster and is typically less reliable (auto purging), while offering a greater amount of disk space (thus "scratch").

By default, the GLUECE: resource property (which contains GLUE data) is sent to the default Index Service:

You can configure which resource properties are sent in WS GRAM's registration.xml file, $GLOBUS_LOCATION/etc/gram-service/registration.xml. The following is the relevant section of the file (as it is set by default):

            <Content xsi:type="agg:AggregatorContent"
            xmlns:agg="http://mds.globus.org/aggregator/types">
            
            <agg:AggregatorConfig xsi:type="agg:AggregatorConfig">
            
            <agg:GetResourcePropertyPollType
                xmlns:glue="http://mds.globus.org/glue/ce/1.1">
            <!-- Specifies that the index should refresh information
            every 60000 milliseconds (once per minute) -->
            <agg:PollIntervalMillis>60000</agg:PollIntervalMillis>
            
            <!-- specifies the resource property that should be
            aggregated, which in this case is the GLUE cluster
            and scheduler information RP -->
            
            <agg:ResourcePropertyName>glue:GLUECE</agg:ResourcePropertyName>
            
            </agg:GetResourcePropertyPollType>
            </agg:AggregatorConfig> 
            <agg:AggregatorData/>
            </Content>
        

SoftEnv is a system designed to make it easier for users to define what applications they want to use, and easier for administrators to make applications available to users. SoftEnv has evolved from the original implementation called Soft designed at Northeastern University in 1994.

In some environments like TeraGrid it's desirable to make use of SoftEnv before a job is submitted to leverage the use of an exactely defined software environment the job will run in.

Because this feature is very specific and may not be available on many systems, support for SoftEnv is disabled by default in normal job submissions. There is a parameter in the JNDI configuration of WS GRAM to enable SoftEnv support in job submissions.

SoftEnv support must be enabled on a per-scheduler basis because the internal mechanisms to support SoftEnv vary between the different types of schedulers. Currently only the scheduler Fork, PBS and LSF can be configured to have SoftEnv support enabled, Condor not yet. To enable this feature the parameter 'enableDefaultSoftwareEnvironment' in the scheduler specific JNDI configuration must be set to 'true'. For example to enable SoftEnv support in the Fork scheduler, set the 'enableDefaultSoftwareEnvironment' in $GLOBUS_LOCATION/etc/gram-service-Fork/jndi-config.xml to 'true'.

Enabled SoftEnv support means that a users default environment will be created from his .soft file before each job submission automatically. The user doesn't need to provide extra SoftEnv keys in the extensions element of a job description. This is not done if the SoftEnv feature is disabled.

For more information and examples, please look in the SoftEnv section of the user guide.

For the scheduler Fork SoftEnv needs to be installed on the host the container is running on. For PBS and LSF SoftEnv needs to be installed on the hosts where the jobs are executed.

To enable communities to define their own system-wide variables and enable their users to use them in their job descriptions, a new generic variable/value config file was added where these variables can be defined. If a job description document contains one of these variables that file will be used to resolve any matching variables.

A new service parameter in the JNDI container registry defines the path to the variable mapping file. The mapping is done for each scheduler. This file is checked periodically (configurable frequency) to see if it has changed. If so, it is reread and the new content replaces the old.

For example for the scheduler Fork there are the following entries in $GLOBUS_LOCATION/etc/gram-service-Fork/jndi-config.xml which can be configured to determine the location and the refresh period of the variable mapping file:

<parameter>
  <name>
    substitutionDefinitionsFile
  </name>
  <value>
    /root/vdt-stuff/globus/etc/gram-service-Condor/substitution definition.properties
  </value>
</parameter>
<parameter>
  <name>
    substitutionDefinitionsRefreshPeriod
  </name>
  <value>
    <!-- MINUTES -->
    480
  </value>
</parameter>

The use of variables in the job description document that are not defined in the variable mapping file leads to the following error during job submission: 'No value found for RSL substitution variable <variableName>'

WS-GRAM provides mechanisms to provide access to audit and accounting information associated with jobs that are submitted to local resource manager (LRM) like PBS, LSF, Condor by WS-GRAM. GRAM is not a local resource manager but rather a protocol engine for communicating with a range of different local resource managers using a standard message format. In some scenarios it is desirable to get an overview over the usage of the underlying LRM like

The following three usecases give a better overview about the meaning and purpose of auditing and accounting:

The audit record of each job is stored in a DBMS and contains

While audit and accounting records may be generated and stored by different entities in different contexts, we assume here that audit records are generated by the GRAM service itself and accounting records by the LRM to which the GRAM service submits jobs. Accounting records could contain all information about the duration and the resource-usage of a job. Audit records are stored in a database indexed by a Grid job identifier (GJID), while accounting records are maintained by the LRM indexed by a local job identifier (JID).

GRAM Service GJID creation

The WS-GRAM service returns an EPR that is used to control the job. The EPR is an XML document and cannot effectively be used as a primary key for a database table. It needs to be converted from an EPR to an acceptable GJID format. A utility class EPRUtil.java is included GT releases beginning from version 4.0.5 and can be used by both the GRAM service before storing the audit record and the GRAM client before getting audit information from the audit database.

To connect the two sets of records, both audit and accounting records, we require that GRAM records theJID in each audit record that it generates. It is then straightforward for an audit service to respond to requests like 'give me the charge of the job with JID x' by first selecting matching record(s) from the audit table and then using the local JID(s) to join to the accounting table of the LRM to access relevant accounting record(s).

We propose a Web Service interface for accessing audit and accounting information. OGSA-DAI is a WSRF service that can create a single virtual database from two or more remote databases. In the future, other per-job information like job performance data could be stored using the GJID or local JID as an index, and then made available in the same virtual database. The rest of this chapter focuses on how to configure WS-GRAM to enable Audit-Logging. A case study for TeraGrid can be read here

OGSA-DAI is available here: http://www.globus.org/toolkit/docs/4.0/techpreview/ogsadai/

Audit logging in WS-GRAM is done 3 times in a job's lifecycle: When the processing starts, when the job is submitted to the local resource manager and when it's fully processed or when it fails.

More information about how to use this data to get e.g. accounting information of a job, how to query that audit database for information via a Web Services interface etc. please go here

Database

Currently database schemas for PostgreSQL and MySQL are provided to create the audit database table.

Simple string extension elements are converted into single-element arrays with the name of the unqualified tag name of the extension element as the array's key name in the Perl job description hash. Simple string extension elements can be considered a special case of the string array construct in the next section.

For example, adding the following element to the <extensions> element of the job description as follows:

will cause the $description->myparam() to return the following value:

String arrays are a simple iteration of the simple string element construct. If you specify more than one simple string element in the job description, these will be assembled into a multi-element array with the unqualified tag name of the extension elements as the array's key name in the Perl job description hash.

For example:

will cause the $description->myparams() to return the following value:

Name/value extension elements can be thought of as string arrays with an XML attribute 'name'. This will cause the creation of a two-dimensional array with the unqualified extension element tag name as the name of the array in the Perl job description hash.

For example:

will cause the $description->myvars() to return the following value:

In addition to the globus_gram_job_manager update package, the globus_gram_job_manager_setup_pbs update package is required to take advantage of the PBS node selection extensions.

Node selection constraints in PBS can be specified in two ways, generally using a construct intended to eventually apply to all resource managers which support node selection, or explicitly by sepcifiying a simple string element. The former will be more portable, but the later will appeal to those familiar with specifying node constraints for PBS jobs.

To specify PBS node selection constraints explicitly, one can simply constuct a single, simple string extension element named nodes with a value that conforms to the #PBS -l nodes=... PBS job description directive. The Globus::GRAM::ExtensionsHandler module will make this available to the PBS adapter script by invoking $description->{nodes}. The updated PBS adapter package checks for this value and will create a directive in the PBS job description using this value.

To use the generic construct for specifying node selection constraints, use the resourceAllocationGroup element:

Extension elements specified according to the above pseudo-schema will be converted to an appropriate nodes parameter which will be treated as if an explicit nodes extension element were specified. Multiple resourceAllocationGroup elements may be specified. This will simply append the constraints to the nodes paramater with a '+' separator. Note that one cannot specify both hostType/hostCount and hostName elements. Similarly, one cannot specify both processCount and processesPerHost elements.

Here are some examples of using resourceAllocationGroup:

For starters, this module logs various things to the log file specified in the logfile extension element. If you place this element at the start of the extensions you are creating support for, then you can look at the specified log file to get some idea of what the handler is doing. You can add new logging lines by using the $self->log() function. This simply takes a string that gets appended to the log file with a prefix of "<date string> EXTENSIONS HANDLER:".

There are three main subroutines that are used to handle parsing events and process them accordingly: Char(), StartTag(), and EndTag(). More handlers can be specified for other specific events when creating the XML::Parser instance in new() (see the XML::Parser documentation for details). Descriptions of what the three main subroutines do currently are layed out bellow. Modify the subroutines as neccesary to achieve your goal.

Char() doesn't do anything but collect CDATA found between the current element start and end tags. You can access the CDATA for the current element by using $self->{CDATA}.

StartTag() is responsible for collecting the attributes associated with the element. It also increments the counter which keeps track of the number of child elements to the current extension element, and pushes the current element name onto the @scope queue for later use.

EndTag() is used for taking the CDATA collected by Char() and creating new Perl job description hash entries. This is most likely where you will need to do most of your work when adding support for new extension elements. Two useful variables are $currentScope and $parentScope. These indicate the current element that is being parsed and the parent of the element being parsed respectively. This is useful for establishing a context from which to work from. The @scope queue is poped at the end of this subroutine.

There is not much to say here. Each adapter is different. Spend some time trying to understand what the adapter does and then make and test your changes. Any new hash entries you created in ExtensionsHandler.pm can be accessed by calling $description->entryname(), where 'entryname' is the name of the entry that was added. See the construct documentation above for more details.