Starting and monitoring the Datatrail Docker services¶
Deployment¶
Datatrail consists of multiple stacks that are created from two images:
maestro-datatrail and maestro-datatrail-client. The stacks are managed and
deployed from the frb-vsop node. The datatrail stack runs the datatrail
server, and the datatrail-daemons stack runs the datatrail daemons, both using
the maestro-datatrail image. More about those services can be found in the
datatrail repository documentation.
The datatrail-client stack runs the workers and APIs that are used to query
and interact with the database. These are launched on the Docker Swarm Manager
nodes, which are: frb-vsop.chime, outrigger_buffer.chime, aux.kko,
aux.gbo, and soon ha1.hco.
The compose files to start these stacks are located in
CHIMEFRB/datatrail and CHIMEFRB/datatrail-admin repositories. Which in the
following example have been cloned to directories of corresponding names:
datatrail-admin/stacks/chime-client.yaml
datatrail-admin/stacks/kko-client.yaml
datatrail-admin/stacks/gbo-client.yaml
datatrail-admin/stacks/hco-client.yaml
datatrail/stacks/daemon-compose.yaml
datatrail/stacks/server-compose.yaml
Warning
Always make sure you git pull the most recent changes before deploying.
To start or restart any of these services simply run the following.
Danger
Before taking down the datatrail stack, which runs the server, please
make sure to take down ALL other datatrail stacks - including those
running at the outrigger sites!
Additionally, the datatrail stack is now a core utility of the CHIME
telescope. Therefore, any down time must be duely communicated to the
collaboration.
# If directory does not exist:
git clone ssh://git@github.com/chimefrb/datatrail
git clone ssh://git@github.com/chimefrb/datatrail-admin
# Else, check you have the latest changes:
cd datatrail
git fetch
git pull
cd ../datatrail-admin
git fetch
git pull
cd ..
# Remove and redeploy stacks
docker stack rm datatrail-daemons datatrail-client datatrail
docker stack deploy -c datatrail/stacks/server-compose.yaml datatrail --with-registry-auth
docker stack deploy -c datatrail/stacks/daemon-compose.yaml datatrail-daemons --with-registry-auth
docker stack deploy -c datatrail/stacks/chime-client.yaml datatrail-client --with-registry-auth
Here we use the example of KKO; however, the process is the same for each outrigger and each has its
own stack file, named outrigger-client.yaml, for the example it is named kko-client.yaml.
# If directory does not exist:
git clone ssh://git@github.com/chimefrb/datatrail-admin
# Else, check you have the latest changes:
cd datatrail-admin
git fetch
git pull
cd ..
# Remove and redeploy stack
docker stack rm datatrail-client
docker stack deploy -c datatrail-admin/stacks/kko-client.yaml datatrail-client --with-registry-auth
Services per stack¶
The datatrail and datatrail-daemons stacks only run at CHIME, they are launched on
the frb-vsop.chime node. The datatrail-client stack runs at each site, and is
launched on the corresponding node.
datatrail stack¶
Launched from the server-compose.yaml file, the datatrail stack contains the following services:
datatrail_server- The main server that handles all the requests and database operations.datatrail_postgres_exporter- The exporter for the postgres database metrics.datatrail_autossh_{site}- The autossh service that connects the server to the scout servers at the outrigger sites.
datatrail-daemons stack¶
Launched from the daemon-compose.yaml file, the datatrail-daemons stack contains the following services:
datatrail-daemons_deletion_notification- Deletion notification service emits monthly reports of upcoming deletions of last replica of data. Note, last copy is currently not deleted.datatrail-daemons_deletion_stager_{site}- Deletion stager produces the deletion Works that are performed by the workers at each site.datatrail-daemons_healing_stager- Healing stager produces the healing Works that are performed by the workers at each site.datatrail-daemons_inventory- Inventory produces the inventory metrics that show the size of each larger dataset.datatrail-daemons_metric_pusher- Metric pusher pushes the metrics to the prometheus server.datatrail-daemons_replication_stager_{site}- Replication stager produces the replication Works that are performed by the workers at each site.datatrail-daemons_state_updater- State updater performs Works from the 'datatrail-state-updater' Bucket.
datatrail-client stack¶
Launched from the {site}-client.yaml file, the datatrail-client stack contains the following services:
datatrail-client_deletion_healer- Deletion healer performs Works in the 'datatrail-replica-heal' Bucket that are tagged with 'deletion' and have a site matching the location the service is running at.datatrail-client_event_registration- Event registration performs the registration of all data at the site.datatrail-client_reattempt_unregistered- Reattempt unregistered performs Works in the 'datatrail-unregistered-datasets' Results database.datatrail-client_replication_healer- Replication healer performs Works in the 'datatrail-replica-heal' Bucket that are tagged with 'replication' and have a site matching the location the service is running at.datatrail-client_replicator- Replicator performs the Works in the 'datatrail-replicator-{site}-minoc' Bucket.datatrail-client_update_canfar_certificates- Update canfar certificates updates the certificates for authenticating Canfar access.
Postgres Database¶
Datatrail is backed by a Postgres database that runs at frb-l4.chime. The configuration
for the database is in the CHIMEFRB/frb-devops repository, with the Docker Compose file
located at /l4-terminus/postgres/compose.yml.
CLI Tools¶
The tools to monitor datatrail is provided by the datatrail-admin CLI. The CLI contains
multiple groups of tools, the most notable are: deletion, registration, and replication.
The CLI can be run on the user's local machine, allowing them to quickly check the status of datatrail without having to ssh in to the node that it is running from. This gives the admin a quick overview of datatrail's status.
Datatrail :: Command Line Interface.
Options:
--help Show this message and exit.
Commands:
action Action tools.
commit Commit tools.
deletion Deletion tools.
query Query tools.
queue Queue tools.
registration Registration tools.
replication Replication tools.
utility Utility tools.
Grafana Panel¶
A Grafana page is also available which gives an overview of the status of Datatrail's various components. Navigate to https://grafana.chimenet.ca and use the magnifying glass to search for "Datatrail". Clicking on the result should yield a page like the one below:
