Table of Contents
Globus XIO is a new component as of Globus 3.2. Previous versions of Globus provided an API called globus_io. XIO is a replacement for globus_io; however a compatibility layer exists to allow full backward compatibility for applications written against globus_io. For users familiar with globus_io the learning curve for XIO will be very small. The APIs use the same event model and only significantly differ in connection establishment.
No changes need to be made in applications using version 2.x releases of the C common libraries.
If you are running a version 2.2 or earlier, it is deprecated, unsupported, has major bugs leading to stability problems, has known potential security exploits via the wuftpd server, and has a protocol incompatibility with later versions due to an error in the security code. Your GT 2.2 clients will not work with newer servers (GT 2.4.0 and greater) and new clients will not work with GT 2.2 servers. You should immediately upgrade to GT 4.0.0.
If you are running GT 2.4.0 or greater, the migration for GridFTP is relatively painless. There were only new features added. No changes were made to the existing protocol or APIs, so any existing client or application built using our client APIs will work unchanged. You may install new clients and servers on an as-opportunity-permits basis and will have no problems. Any combination of old/new client/server will work.
To upgrade your server, either install it in a $GLOBUS_LOCATION different than the GT 2 installation (either as
part of an entire GT 4 installation or by just doing make gridftp). Alternately, you can statically link the new
server to avoid versioning issues and replace the existing executable. The configuration files are very different,
so you will need to update the configuration.
To upgrade your client, simply build the client and use the new client as you would the old one.
Below is a list of new functionality available in GT 3.2 and higher (note that the MLST/MLSD feature is used by RFT in GT 3.2 and higher and is required if you want to be able to specify a directory to move):
New Functionality in 3.2:
Server Improvements
- Structured File Info
- MLST, MLSD
- checksum support
- chmod support
globus-url-copy changes
- File globbing support
- Recursive dir moves
- RFC 1738 support
- Control of restart
- Control of DC security
Users of GT2 RLS must create the RLI database, which was not required in some GT2 versions
of RLS. To create the RLI database run the RLI database script for your particular DBMS. See the
database configuration instructions for more details. This is only required if the RLS
is deployed as an RLI (e.g., rls_server true).
In pre-WS GRAM, jobs are submitted to a job manager process started by a Gatekeeper process. The Gatekeeper process is typically started out by an inetd server, which forks a new gatekeeper per job. In WS GRAM, jobs are started by the ManagedExecutionJobService, which is a Java service implementation running within the globus service container.
The gatekeeper searches the $GLOBUS_LOCATION/etc/grid-services directory to determine which services it will start. Typically there is one job manager service entry file in that directory per scheduler type.
In pre-WS GRAM, the GRAM client is required to delegate a proxy credential to the Gatekeeper so that the job manager can send authenticated job state change messages.
In WS GRAM, delegation is done as needed using the DelegationFactoryService. Jobs may be passed references to delegated credentials as part of the job description.
In pre-WS gram, communication between the client and gatekeeper is done using GSI-wrapped messages. Communication between the the client and job manager are sent using SSL. The job manager uses the delegated credential for file streaming or staging as well. Mutual authentication is done on all connections. All communications consist of a single request-response pattern. Network connections and security contexts are never cached between messages.
In WS GRAM, communication may be secured using TLS, ws secure messaging, or ws secure conversation, depending on service configuration. When doing authentication, the service will use the credentials of the container, or for secure message or conversation, a service-specific credential. It will not use a delegated credential when communicating with the client.
The gatekeeper process is started as a root service out of inetd. It then uses the grid-mapfile decide which local user it should setuid() to before starting the job manager process, based on the credential used to submit the job request. The user may optionally propose a non-default user-id by specifying it in the gatekeeper contact string. The job manager process runs entirely under the local user account.
In WS GRAM, the job management service runs within a container shared with other services. The container is run under a non-privileged account. All commands which need to be run as a particular user (such as interactions with the scheduler) are started via sudo. Authorization is done via the globus-gridmap-and-execute program.
In pre-WS gram, all file system and scheduler interactions occur within a perl module called by the globus-job-manager-script.pl program. Scheduler-specific perl modules implement a number of methods which are used by the job manager:
- submit
- poll
- cancel
- signal
- make_scratchdir
- remove_scratchdir
- stage_in
- stage_out
- cache_cleanup
- remote_io_file_create
- proxy_relocate
- proxy_update
Only a small set of these script methods are used in the WS GRAM implementation. The subset used is:
- submit
- poll (called only once per job and only for fork/condor jobs to merge output)
- cancel
- cache_cleanup
Some of the functionality has been moved into other services for reliability or performance reasons. Other functions have been removed altogether.
- poll: SEG
- signal: dropped
- make_scratchdir: rft
- remove_scratchdir: rft
- stage_in: rft
- stage_out: rft
- remote_io_file_create: rft or resource property queries
- proxy_relocate: delegation service
- proxy_update: delegation service
In pre-WS gram, each job submitted would cause the following processes to be created:
- gatekeeper (short lived)
- job manager (lives the duration of the job)
- perl script (short lived 4 or more instances depending on job type)
- perl script poll called periodically
In WS GRAM, each job causes the following processes to be created
- sudo + perl script--(typically 2 times: submit, cache_cleanup)
- for fork jobs, one fork-starter process (blocked waiting for a signal) for the duration of the job
Additionally, there will be a per-scheduler instance of the SEG program, monitoring a log file for job state changes. Migration from pre-WS GRAM to WS-GRAM
Typical interactions with the pre-WS gram service were done with either the globusrun or command or the globus-job suite of scripts (globus-job-submit, globus-job-run, globus-job-get-output, globus-job-status, globus-job-clean). The main difference between these sets of commands is that globusrun required a job description in RSL format, and the globus-job-submit and globus-job-run scripts would construct that based on command line options.
In WS GRAM, the globusrun-ws command implements the functionality of globusrun using the XML Job Description language in place of the RSL format job description of pre-WS GRAM. It also allows specifying parts of the Job Description with simple command line arguments (for executable and arguments), similar to what one would do with globus-job-run. Like globusrun, the globusrun-ws program supports both the interactive and batch submission of GRAM jobs.
Table 1. Command Line Option Comparison
| Description | pre-WS GRAM globusrun option | WS GRAM globusrun-ws option |
|---|---|---|
| Interactive Multirequest Control | -i | NO EQUIVALENT |
| Job Description File Path | -f <rsl filename> | -file <rsl filename> | -f <filename> | -job-description-file <filename> |
| Quiet operation | -q | -quiet | -q | -quiet |
| File streaming of stdout and stderr *see note 1* | -o (Implies -q) | -s | -streaming (Implies -q, sometimes -staging-delegate) |
| Enable embedded GASS Server | -s | -server (Implies -o and -q) | NO EQUIVALENT |
| Enable writing to the embedded GASS Server | -w | -write-allow (Implies -s and -q) | NO EQUIVALENT |
| Specify Service Contact | -r <resource-manager> | -resource <resource-manager> (Specifies Gatekeeper contact) | -F, -Ft, or -Ff; Use either factory service contact (-F), Factory Type (-Ft) or Factory EPR file (-Ff) |
| Do not terminate job when SIGINT is received. | -n | -no-interrupt | -n | -no-cleanup |
| Destroy a job based on a job - contact | -k <job contact> | -kill <job contact> | -kill -j <filename> | -kill -job-epr-file <filename> |
| Get current job status | -status <job contact> | -status -j <filename> | -status -job-epr-file <filename> |
| Batch mode job submission | -b | -batch or -F | -fast-batch | -batch | -b |
| Refresh proxy | -refresh-proxy <job contact> | -y <job contact> | NO EQUIVALENT |
| Stop a job manager process, saving state | -stop-manager <job contact> | NO EQUIVALENT |
| Validate job description without submitting job | -p | -parse | -validate |
| Ping job manager | -a | -authenticate-only | NO EQUIVALENT |
| Dryrun | -d | -dryrun | NO EQUIVALENT |
Note 1: In pre-WS GRAM, streaming is done using https connections from the job manager to a GASS server embedded in the globusrun program. In WS GRAM, streaming is implemented by accessing a gridftp server configured to run along with the service container.
globusrun-ws has additional options to deal with file streaming, monitoring an existing job, authentication and authorization, http timeouts, default termination time, encryption, etc.
This table describes the migration path for applications which use the C language interface to pre-WS gram. This table covers the globus_gram_client API.
Table 2. C API Migration Table
| GT2 API Command | GT4 API Command |
|---|---|
| globus_gram_client_callback_allow() | globus_notification_create_consumer() |
| globus_gram_client_register_job_request() | ManagedJobFactoryPortType_GetResourceProperty_epr_register() |
| globus_gram_client_job_request() | ManagedJobFactoryPortType_GetResourceProperty_epr() |
| globus_gram_client_register_job_cancel() | ManagedExecutableJobPortType_Destroy_epr_register() |
| globus_gram_client_job_cancel() | ManagedExecutableJobPortType_Destroy_epr() |
| globus_gram_client_job_status() | ManagedExecutableJobPortType_GetResourceProperty_epr() with the property name {http://www.globus.org/namespaces/2004/10/gram/job/types}state |
| globus_gram_client_register_job_status() | ManagedExecutableJobPortType_GetResourceProperty_epr_register() with the property name {http://www.globus.org/namespaces/2004/10/gram/job/types}state |
| globus_gram_client_job_refresh_credentials() | globus_delegation_client_util_delegate_epr |
| globus_gram_client_register_job_refresh_credentials() | globus_delegation_client_util_delegate_epr_register() |
| globus_gram_client_register_job_signal() | ManagedExecutableJobPortType_release_epr_register() |
| globus_gram_client_job_signal() | ManagedExecutableJobPortType_release_epr() |
| globus_gram_client_register_job_callback_registration() | ManagedExecutableJobPortType_Subscribe_epr_register() |
| globus_gram_client_job_callback_register() | ManagedExecutableJobPortType_Subscribe_epr() |
| globus_gram_client_register_job_callback_unregistration() | SubscriptionManager_Destroy_epr_register() |
| globus_gram_client_job_callback_unregister() | SubscriptionManager_Destroy_epr() |
| globus_gram_client_callback_disallow() | globus_notification_destroy_consumer() |
| globus_gram_client_job_contact_free() | wsa_EndpointReferenceType_destroy() |
| globus_gram_client_error_string() | globus_error_get(result) |
| globus_gram_client_set_credentials() | globus_soap_message_handle_set_attr() with the property name GLOBUS_SOAP_MESSAGE_USER_CREDENTIAL_KEY and the value the gss_cred_id_t |
| globus_gram_client_ping() | XXX? Maybe factory get resource properties? |
| globus_gram_client_register_ping() | XXX? Maybe factory get resource properties? |
| globus_gram_client_debug() | set GLOBUS_SOAP_MESSAGE_DEBUG environment variable to MESSAGES to see XML messages sent/received |
| globus_gram_client_version() | NO EQUIVALENT |
| globus_gram_client_attr_init() | globus_soap_message_attr_init() |
| globus_gram_client_attr_destroy() | globus_soap_message_attr_destroy() |
| globus_gram_client_attr_set_credential() | globus_soap_message_handle_set_attr() with the property name GLOBUS_SOAP_MESSAGE_USER_CREDENTIAL_KEY and the value the gss_cred_id_t |
| globus_gram_client_attr_get_credential() | globus_soap_message_attr_get() with the property name GLOBUS_SOAP_MESSAGE_USER_CREDENTIAL_KEY. Migration from Prews GRAM to ws-GRAM |
Pre-WS GRAM uses a custom language for specifying a job description. WS GRAM uses an xml based language for this same purpose. In pre-WS GRAM, relations (such as count=5) can occur in any order within the RSL; in WS GRAM, the relations must be in the order in the XML schema definition. The RSL attribute description below is in the order defined by the XML schema
Table 3. RSL Migration Table
| GT2 RSL Attribute | GT4 job description element |
|---|---|
| (username = NAME) | <localUserId>NAME<localUserId> |
| (two_phase = TWO_PHASE_TIMEOUT) *See Note 1* | <holdState>Pending</holdState> |
| (executable = EXE) | <executable>EXE</executable> |
| (directory = DIR) | <directory>DIR</directory> |
| (arguments=ARG1 ... ARGN) | <argument>ARG1</argument> ... <argument>ARGN</argument> |
| (environment = (ENV_VAR_1 ENV_VAL_1) ... (ENV_VAR_N ENV_VAL_N)) | <environment> <name>ENV_VAR_1</name> <value>ENV_VAL_1</name> ... <name>ENV_VAR_N</name> <value>ENV_VAR_N</value> </environment> |
| (stdin = LOCAL_FILE_PATH) *See Note 2* | <stdin>file:///LOCAL_FILE_PATH</stdin> |
| (stdout = LOCAL_FILE_PATH) *See Note 2* | <stdout>file:///LOCAL_FILE_PATH</stdout> |
| (stderr = LOCAL_FILE_PATH) *See Note 2* | <stderr>file:///LOCAL_FILE_PATH</stderr> |
| (count = NUMBER) | <count>NUMBER</count> |
| (library_path = PATH_ELEMENT_1 ... PATH_ELEMENT_N) | <libraryPath>PATH_ELEMENT_1</libraryPath> ... <libraryPath>PATH_ELEMENT_N</libraryPath> |
| (host_count = NUMBER) | <hostCount>NUMBER</hostCount> |
| (project = PROJECT) | <project>PROJECT</project> |
| (queue = QUEUE) | <queue>QUEUE</queue> |
| (max_time = MINUTES) | <maxTime>MINUTES</maxTime> |
| (max_wall_time = MINUTES) | <maxWallTime>MINUTES</maxWallTime> |
| (max_cpu_time = MINUTES) | <maxCpuTime>MINUTES</maxCpuTime> |
| (max_memory = MEGABYTES) | <maxMemory>MEGABYTES</maxMemory> |
| (min_memory = MEGABYTES) | <minMemory>MEGABYTES</minMemory> |
| (job_type = JOBTYPE) | <jobType>JOBTYPE</jobType> |
| (file_stage_in = (REMOTE_GRIDFTP_URL_1 LOCAL_FILE_PATH_1) ... (REMOTE_GRIDFTP_URL_N LOCAL_FILE_PATH_N)) *See Note 4* | <fileStageIn> <transfer> <sourceUrl>REMOTE_GRIDFTP_URL_1</sourceUrl> <destinationUrl>file:///LOCAL_FILE_PATH_1</destinationUrl> </transfer> <transfer> <sourceUrl>REMOTE_GRIDFTP_URL_N</sourceUrl> <destinationUrl>file:///LOCAL_FILE_PATH_N</destinationUrl> </transfer> </fileStageIn> |
| (file_stage_out = (LOCAL_FILE_PATH_1 REMOTE_GRIDFTP_URL_1) ... (LOCAL_FILE_PATH_N REMOTE_GRIDFTP_URL_N)) *See Note 4* | <fileStageOut> <transfer> <sourceUrl>file:///LOCAL_FILE_PATH_1</sourceUrl> <destinationUrl>REMOTE_GRIDFTP_URL_1</destinationUrl> </transfer> <transfer> <sourceUrl>file:///LOCAL_FILE_PATH_N</sourceUrl> <destinationUrl>REMOTE_GRIDFTP_URL_N</destinationUrl> </transfer> </fileStageOut> |
Note 1: The globusrun-ws program will automatically release the hold after receiving the indicated hold state. To simulate the two-phase submit timeout, an application could set the initial termination time of the resource. A hold state may be set for fileCleanUp state for two-phase commit end, but it is not possible to submit a job with both hold states.
Note 2: stdin, stdout, and stderr must only be a local file URL. Ftp and gridftp URLs can be handled by using a fileStageIn and fileStageOut elements (described below).
Note 3: Value job types for WS GRAM are multiple (the default), single, mpi, and condor.
Note 4: The WS GRAM service uses RFT to transfer files. This only supports gridftp and ftp file transfers. The local file path must be a mappable by an entry in the file system mapping file.
The following RSL attributes have no direct equivalent in WS GRAM:
- dry_run: Similar behavior can be obtained by using a job hold state of Pending and then destroying the job resource without releasing the hold.
- file_stage_in_shared: No support for the GASS cache, hence this is gone. Applications may use RFT to transfer files before submitting a batch of jobs.
- gass_cache: GASS cache is not used by WS GRAM, so there is no need for setting the cache path.
- gram_my_job: collective operations are enabled for every managed execution job service via rendezvous registration
- proxy_timeout: Delegated security proxies are handled via the DelegationFactory Service. Resource lifetime is controlled by the wsrl:SetTerminationTime operation
- remote_io_url: The WS GRAM service does not use GASS, so there is no equivalent to this.
- restart: There is no equivalent.
- rsl_substitution: The WS GRAM service does not support user-defined substitutions. Certain values may be referenced in some RSL values by a similar technique, but these are for system configuration parameters only. See the WS GRAM job description document for description of RSL variable syntax, values, and attributes where they may be used.
- save_state: All WS GRAM jobs are persistent, so there is no elements related to this.
- scratch_dir: This is now a deployment configuration option.
- stderr_position: Standard error streaming is now a feature of the globusrun-ws program instead of part of the WS GRAM service, so there is no equivalent element for restarting error streaming at a specific point.
- stdout_position: Standard output streaming is now a feature of the globusrun-ws program instead of part of the WS GRAM service, so there is no equivalent element for restarting output streaming at a specific point.
Here are some examples of converting some pre-WS GRAM RSLs to WS GRAM.
Table 4. RSL Migration Examples
| pre-WS GRAM RSL | WS GRAM Job Description |
|---|---|
(* Simple Job Request With Arguments *)
&(executable = /bin/echo)
(arguments = Hello, Grid)
|
<?xml version="1.0"?>
<!-- Simple Job Request With Arguments -->
<job xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job/description">
<ns1:executable>/bin/echo</ns1:executable>
<ns1:argument>Hello,</ns1:argument>
<ns1:argument>Grid</ns1:argument>
</job>
|
(* Multijob Request *)
+(
&(executable = /bin/echo)
(arguments = Hello, Grid From Subjob 1)
(resource_manager_name = resource-manager-1.globus.org)
(count = 1)
)
(
&(executable = mpi-hello)
(arguments = Hello, Grid From Subjob 2)
(resource_manager_name = resource-manager-2.globus.org)
(count = 2)
(jobtype = mpi)
)
|
<?xml version="1.0"?>
<!-- Multijob Request -->
<multiJob
<!-- namespace of GRAM resource ID elements -->
xmlns:gram="http://www.globus.org/namespaces/2004/10/gram/job"
<!-- namespace of WS-Addressing (EPR) elements -->
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/03/addressing">
<factoryEndpoint>
<wsa:Address>
<!-- URL for the factory service on resource-manager-0.globus.org --<
https://resource-manager-0.globus.org:8443/wsrf/services/ManagedJobFactoryService
</wsa:Address>
<wsa:ReferenceProperties>
<!-- ID for the "Multi" factory resource -->
<gram:ResourceID>Multi</gram:ResourceID>
</wsa:ReferenceProperties>
<wsa:ReferenceParameters/>
</factoryEndpoint>
<job>
<factoryEndpoint>
<wsa:Address>
<!-- URL for the factory service on resource-manager-1.globus.org -->
https://resource-manager-1.globus.org:8443/wsrf/services/ManagedJobFactoryService
</wsa:Address>
<wsa:ReferenceProperties>
<!-- ID for the "Fork" factory resource -->
<gram:ResourceID>Fork</gram:ResourceID>
</wsa:ReferenceProperties>
<wsa:ReferenceParameters/>
</factoryEndpoint>
<executable>/bin/echo</executable>
<argument>Hello,</argument>
<argument>Grid</argument>
<argument>From</argument>
<argument>Subjob</argument>
<argument>1</argument>
<count>1</count>
</job>
<job>
<factoryEndpoint>
<wsa:Address>
<!-- URL for the factory service on resource-manager-2.globus.org -->
https://resource-manager-2.globus.org:8443/wsrf/services/ManagedJobFactoryService
</wsa:Address>
<wsa:ReferenceProperties>
<!-- ID for the "Fork" factory resource -->
<gram:ResourceID>Fork</gram:ResourceID>
</wsa:ReferenceProperties>
<wsa:ReferenceParameters/>
</factoryEndpoint>
<executable>mpi-hello</executable>
<argument>Hello,</argument>
<argument>Grid</argument>
<argument>From</argument>
<argument>Subjob</argument>
<argument>2</argument>
<count>2</count>
<jobType>mpi</jobType>
</job>
</multiJob>
|
Although the basic functionality remains the same for MDS in GT4, the architecture, standards used, and implementation have changed significantly in GT2. The following table shows a mapping of some GT2 concepts to GT4 concepts.
Table 5. Comparison of GT2 and GT4
| Description | GT2 Version | GT4 Version |
|---|---|---|
| Format of data describing a resource | LDAP data hierarchy | XML data document |
| Query language | LDAP queries | XPath queries |
| Wire protocol for queries | LDAP | WS-ResourceProperties |
| APIs used for queries | LDAP APIs | WS Core APIs |
| Command-line clients used for queries | grid-info-search | wsrf-get-property, wsrf-get-properties,
wsrf-query |
| Available GUIs | Various LDAP browsers | WebMDS |
| Wire protocol for subscription/notification | Not supported | WS-Notification |
| APIs used for subscription/notification | Not supported | WS Core APIs |
| Security support | SAML-based security using X.509 user, proxy and host certificates | HTTPS-based security using X.509 user, proxy and host certificates |
| Queryable index of aggregated information | GIIS, which publishes data using the LDAP-related standards listed above | WS MDS Index Server, which publishes data using the WSRF-related standards listed above |
| Queryable source of non-aggregated information | GRIS, which uses information providers to gather data from services and then publishes that data the LDAP-related standards listed above | Individual web services, which publish data about their own resources using WSRF-related standards listed above. |
| Index registration mechanism | MDS servers (GRIS's and, in some cases, GIIS's) register themselves
with a GIIS. An MDS server is configured to register itself to a remote
index by editing the local MDS server's
grid-info-resource-register.conf file, providing
information about the location of the remote index to register to
and timeout values for the registration | WS MDS Index servers maintain aggregating service groups that
include registration information (timeout values, the mechanism to
use to acquire information, and additional mechanism-specific parameters)
The registration is accomplished by
adding an entry to an aggregating service group via the
mds-servicegroup-add command. In addition, services
may be configured to register themselves to the default index server
running in the same container. |
| Mechanism used by an index to collect information | GIIS's send LDAP queries to remote serves. | WS MDS Index servers use a plugin-based architecture to support several mechanisms to collect information. The Globus Toolkit supplies plugins that support collecting information via polling (resource property queries), subscription/notification, and by program execution. |
This version is not compatible with the GT2 version of CAS because of protocol changes. To migrate to this version, this component needs to be installed completely independent of any current GT2.x CAS installs.
No special procedures are required for GSI-OpenSSH installations migrating from GT2 to GT4. GSI-OpenSSH is backward compatible.