LinuxHPC.org/Cluster Builder 1.3
Job Scheduler
Job Scheduler
By LinuxHPC.org and Cluster Resources
| Job Scheduler |
|
A job scheduler is a software application that is in charge of unattended background executions, commonly known as batch processing. This type of software is often known as batch queue manager because batch queues are commonly used to prioritize execution of multiple jobs. Job schedulers are also sometimes known as a distributed resource manager or batch scheduler.
Features/Benefits
Basic features expected of job scheduler software are:
- Interfaces to define workflows and/or job dependencies
- Automatic submission of executions
- Interfaces to monitor the executions
- Priorities and/or queues to control the execution order of unrelated jobs
Main Concepts
There are many concepts that are central to almost every job scheduler implementation and that are widely recognized with minimal variations:
- Jobs
- Dependencies
- Job Streams
- Users
- Master/Agent architecture — the historic architecture for Job scheduling software. The Job Scheduling software is installed on a single machine (Master) while on production machines only a very small component is installed (Agent) that waits commands from the Master, executes them, and returns the exit code back to the Master.
- Cooperative architecture — a decentralized model where each machine is capable of helping with scheduling and can offload locally scheduled jobs to other cooperating machines.
Scheduling
Various schemes are used to decide which particular job to run. Parameters that might be considered include:
- job priority
- compute resource availability
- license key if job is using licensed software
- execution time allocated to user
- number of simultaneous jobs allowed for a user
- estimated execution time
- elapsed execution time
Implementations
There are many implementations of job scheduling software. A common approach is to apply an advanced scheduler to a job scheduler, using the job scheduler only for the resource management functions, achieving a full workload management solution with the greatest set of capabilities. One such combination that is popular due to increased capabilities, support and low costs, is combining Moab Workload Manager with the TORQUE * Resource Manager.
Advanced Schedulers provide added functions and capabilities above and beyond what is capable of common job schedulers, integrating advanced reservations, resource scheduling, policy and event engines to help process workloads faster, in line with set priorities and organizational objectives.
See Main Article: Advanced Scheduler
Advanced SchedulersCommercial implementations
Open-source implementations
Job Schedulers
Open-source implementations
Commercial implementations
- Batch Job Server - Batch Job Scheduling Software for Windows 2000, 2003 +
- LoadLeveler
- Platform LSF (Load Sharing Facility)
- N1 Grid Engine — commercial version of Sun GridEngine
- PBS Pro — commercial version of Portable Batch System