Workspace¶

A workspace is yaml configuration which defines the operational behavior of the workflow system. For example, it defines the valid sites, authentication method, URLs for backend services, archiving and storage options for the plots and products produced by the workflow system, and many more.

A workspace is,

required to run workflow.
project specific or shared across multiple projects.
shared across multiple users or specific to a user / pipeline.
stored at the home directory under the path ~/.config/workflow.
provided by the project administrator.
can be a filepath, URL or a JSON string.

Command Line Interface¶

Workflow ships with a command line interface to manage workspaces.

workflow workspace --helpUsage: workflow workspace [OPTIONS] COMMAND [ARGS]...

Manage Workflow Workspaces.

Options:
--help Show this message and exit.

Commands:
ls List workspaces.
purge Purge all local workspaces.
read Read workspace config.
set Set the active workspace.
unset Unset active workspace.

Set a workspace¶

Your project administrator should provide you with a workspace configuration file. This file can be hosted as a remote url, a local file or be pushed upstream to be packaged into workflow itself.

To set a workspace run,

Bash

workflow workspace set development
workflow workspace set https://raw.githubusercontent.com/your/repo/development.yaml
workflow workspace set /path/to/development.yaml

CHIME/FRB Workspace

The CHIME/FRB workspace ships with the workflow package. To set the CHIME/FRB workspace run,

Bash

$ workflow workspace set chimefrb

Setting a workspace, copies the workspace configuration to ~/.config/workflow/workspace.yml location.

Reading a workspace¶

To read the active workspace configuration run,

workflow workspace read{
'sites': ['local', 'chime', 'kko', 'gbo', 'hco'],
'workspace': 'chimefrb',
'http': {
'baseurls': {
'buckets': ['http://local.link:8004', 'https://web.external/buckets'],
'configs': ['http://local.link:8007', 'https://web.external/pipelines'],
'loki': 'https://web.external/loki/loki/api/v1/push',
'pipelines': ['http://local.link:8007', 'https://web.external/pipelines'],
'products': 'https://web.external/read/products/',
'results': ['http://local.link:8005', 'https://web.external/results'],
'schedules': ['http://local.link:8007', 'https://web.external/pipelines']
}

For more information on the workspace configuration, see Workspace YAML format.

Unset a workspace¶

To unset the active workspace simply run,

Bash

workflow workspace unset

Important

Running workflow without a workspace set will result in an runtime error.

Workspace for Container Images¶

In order to setup workflow, pre-configured in container images, it is recommended to set the workflow set workspace command during the container build process. For example, in a Dockerfile:

Docker

FROM python:3.10-slim-buster
RUN set -ex \
    pip install workflow.core && \
    workflow workspace set development

Workspace YAML format¶

`workspace`¶

This key states the name of the workspace.

YAML

workspace: "test-workspace"

`sites`¶

Under the 'sites' key is where a list of strings naming the sites that are allowed when running Workflow. Eg:

YAML

sites:
  - local

`archive`¶

The 'archive' key describes the storage sites that are available. Valid keys are:

mounts, which should contain a key for every valid site listed under sites giving its root mount point.

YAML

archive:
mounts:
  local: "./"

`http`¶

The http key contains the configuration for requests that utilise http. It is here that we set the base URLs for Workflow's dependant services.

baseurls, this should contain a URL for each of the six services that Workflow is able to use. These comprise:
Buckets database
Results database
Pipelines API
Products API
Loki logs
MinIO object storage

An example of this section of the YAML file:

YAML

http:
  baseurls:
    buckets: http://localhost:8001
    results: http://localhost:8002
    pipelines: http://localhost:8003
    products: http://localhost:8004
    minio: http://localhost:8005
    loki: http://localhost:8005/loki/api/v1/push

`config`¶

archive
results, this key enables or disables the functionality to copy Work entries across to the Results database upon their completion. Valid values are either:
YAML
```
# To enable
results: true
# To disable
results: false
```
plots, this sets the archiving methods and storage type for plots produced when using Workflow. It has the following keys:
- methods, this is a list that sets the methods enabled by the workspace admin these can be a including any of the following: bypass, move, copy, delete.
- storage, this tells Workflow what storage type it will use to archive the plots, which can be either: s3, posix, or http (currently only has place-holding code).
YAML
```
plots:
  methods: ["move", "delete"]
  storage: "s3"
```
products, this sets the archiving methods and storage type for the products produced. It has the same keys as plots, above, and those have the same options.
- methods, same as above.
- storage, same as above.
  YAML
```
products:
  methods: ["bypass", "copy"]
  storage: "posix"
```
permissions, this is used by the archiver to determine how to set the permissions for each storage type. Although, 's3' is included here, MinIO handles permissions internally.
- s3
- user
- group
- posix
- command, this is an f-string template that defines the command that you wish to use to set the directory permissions.
- user, the user to use in the permissions command.
- group, the group to use in the permissions command.
  YAML
```
posix:
  group: "chime-frb-rw"
  command: "setfacl -R m g:{group}:rw {path}"
```

Below is an example of the config section of the workspace YAML.

YAML

config:
  archive:
    results: true
    plots:
      methods: ["move", "delete"]
      storage: "s3"
    products:
      methods: ["bypass", "copy"]
      storage: "posix"
    permissions:
      s3:
        user: "user1"
        group: "group1"
      posix:
        group: "group1"
        command: "setfacl -R -m g:{group}:r {path}"

slack, WIP; to be implemented.

logging¶

loki
tags, this sets the tags that are applied to the logs sent to Loki. These tags are used to index the logs for searching at a later date. This can be any key-value pair:

YAML

tags:
  project: chimefrb

Below is an example of the logging section of the workspace YAML.

YAML

logging:
  loki:
    tags:
      example_tag1: example_value1
      example_tag2: example_value2
      example_tag3: example_value3

Workspace¶

Command Line Interface¶

Set a workspace¶

Reading a workspace¶

Unset a workspace¶

Workspace for Container Images¶

Workspace YAML format¶

workspace¶

sites¶

archive¶

http¶

config¶

logging¶

`workspace`¶

`sites`¶

`archive`¶

`http`¶

`config`¶