Deployments¶
Explanation¶
After listing all the steps of your Pipeline, and all the Work objects that will be needed, you will also need something that is going to pick up these jobs and run them. The answer to this are Workflow runners, which are replicated containers spawned on your specified cluster. These containers pick up specific Work per step, and runs their defined function in its given image. Workflow handles the spawning of these containers for you; when crafting your Pipeline Config, all you need to do is specify something called "Deployments". These are specifications telling Workflow what options to apply to create a service of replicated containers. These deployments are then spawned by specifying them in the "runs_on" attribute of a Pipeline or a Pipeline Step.
YAML Example¶
Here is an example Pipeline Config YAML file, with deployments:
version: 1
name: example-pipeline-config
defaults:
user: example-user
deployments:
- name: example-deployment-1
site: chime
image: chimefrb/workflow:latest
resources:
cores: 1
ram: "1G"
gpu: 1
replicas: 5
sleep: 5
lives: -1
- name: example-deployment-2
site: local
image: chimefrb/workflow:latest
resources:
cores: 2
ram: "256M"
gpu: 0
replicas: 50
sleep: 15
lives: 10
pipeline:
runs_on: example-deployment-1
steps:
- name: example-stage-1
stage: 1
matrix:
event: [111111, 222222]
work:
site: chime
command: ["ls", "${{ matrix.event }}"]
- name: example-stage-2
runs_on: example-deployment-1, example-deployment-2
stage: 2
matrix:
event: [333333, 444444, 555555]
work:
site: chime
command: ["ls", "${{ matrix.event }}"]
runs_on usage
In the Pipeline section, you can see a "runs_on" attribute at the Pipeline level, and one at the Step level. If a Step is missing the attribute, it will default to the Pipeline level's. If given a matrix for a Pipeline (thus spawning multiple replicated Pipelines, which is currently the only way to have multiple Pipelines for a Pipeline Config), the "runs_on" deployment specified will only be deployed once for all those Pipelines. Furthermore, these top-level "runs_on" deployments will spawn right at the start, reserving resources early. Additionally, you can see that multiple deployments can be referenced for one object; in this example, this means both Deployments will be spawned for that Step (at its time of execution), using the Deployment's given site's orchestration engine.
YAML Specification¶
Name¶
The name to be given to the Deployment, and will be used when spawning the service on the given site.
Site¶
The site that the Deployment will be spawned on, e.g. "chime", "canfar", "local", etc. The site will determine the orchestration engine used (Docker, Kubernetes, etc).
Image¶
The image executable to use, which should contain all the necessary code to run your Work's specified function/command, and must contain Workflow as a package dependency, so that the default entrypoint command "workflow run ..." will succeed.
Resources¶
The resource limits to be set on the replicated containers that are spawned for the deployment. The keys are "cores", "ram", and "gpu", which can be any non-negative value. For "ram", it must be specified in gigabytes (e.g. "12G") or megabytes (e.g. "128M"). For "cpu" and "gpu", it's specified in cores as an integer or float.
Replicas¶
The number of replicated Workflow runner containers to spawn when using this deployment. The maximum acceptable value for this SANIC_DEPLOYER_MAX_REPLICAS, an environmental variable set in the pipelines_managers service in the docker-compose.yaml.
Sleep¶
The amount of time in seconds for each replicated Workflow runner containers to sleep for after attempting a single Work object. Must be a non-negative number.
Lives¶
The amount of Work objects each replicated Workflow runner container will stay alive for before exiting (default is -1 for infinite). For example, if lives is set to 3, the Workflow runner will attempt 3 times to pick up and run a Work object, and whether it was a success or a failure, will exit after those 3 times.