Multi-job Processing on grid
As described in Multi-job Processing, the CLC Server supports execution of multiple jobs at the same time on a job node or single server. This is also possible in grid setups. By providing resource information when submitting jobs to the grid, the underlying grid scheduler will be able to schedule multiple jobs on the execution nodes when appropriate.
Algorithms can either be exclusive, streaming, or non-exclusive. Exclusive algorithms are optimized to utilize the machine they are running on. They have very high I/O bandwidth, memory, or CPU requirements and therefore do not play well with other algorithms. Streaming algorithms are highly I/O intensive, and running two streaming algorithms on the same machine does not yield any advantages. For grid execution, streaming algorithms are treated as exclusive algorithms. See the list in Appendix Non-exclusive Algorithms for a list of non-exclusive algorithms.
Non-exclusive algorithms expose their CPU or thread usage, and this can be passed on to the grid scheduler when submitting jobs. In order to pass this information to the grid scheduler, the grid preset must be set to Resource Aware mode:
Go to:
Admin (
) | Job distribution (
)
and click 'Create New Preset...' or 'Edit...' and fill out the fields as described in Configure grid presets and set 'Grid Mode' to 'Resource Aware'. This will bring up a new field called 'Shared native specification'.
When submitting non-exclusive algorithms the Shared native specification field is used as submission argument. Exclusive algorithms are submitted using the regular native specification.
The Shared native specification must be filled out such that at least the COMMAND_THREAD_MAX variable is passed properly to the grid (for hints see Controlling the number of cores utilized).
Figure 6.16: Grid preset configured for multi-job processing on LSF.
Figure 6.8 shows a Resource Aware grid preset setup for LSF.