Deploying agents#

Agents are deployed to their specified home:

1agent_home = Source.FromLocal("/local/path/")
2smith = Agent(
3    home = agent_home,
4    runtime=ContainerRuntime.DOCKER,
5)
6smith.Deploy()

Previous deployments, such as those from an earlier version of Metasmith, can up overwritten by setting assertive=true.

1smith.Deploy(assertive=True)

Agents can also be deployed to remote machines through SSH by changing their home.

1agent_home = Source.FromSsh(
2    host="<host_name>",
3    path="/remote/path/",
4)

Tip

SSH setup

Additional setup commands, to be run before the excution of each workflow, can be specifed:

1smith = Agent(
2    # ...
3    setup_commands=[
4        'export TMPDIR="/home/$USER/tmp"',
5        'mkdir -p $TMPDIR',
6        'export APPTAINER_CACHEDIR="$TMPDIR"',
7        'export APPTAINER_TMPDIR="$TMPDIR"',
8    ]
9)

Note

For the Apptainer runtime, APPTAINER_CACHEDIR also selects where Metasmith stores its built container images (the .sif files and any unpacked .sandbox directories) — this is the store apptainer exec reads from at run time. When it is unset, Metasmith falls back to <agent_home>/container_images. If you export it (as above), point it at persistent storage with room for the images; a location that is wiped between sessions (e.g. a node-local /tmp) forces a re-pull on every run. The example above uses /home/$USER/tmp, which persists, so it is safe.

Container Runtime#

To maximize reproducibility and portability, Metasmith relies on OCI compliant containers to standardize the compute environment for itself and the tools that it runs. One of the following must be installed on the machine that the agent is deployed to:

  • Apptainer is typically used by research compute infrastructure. See the apptainer install notes for the recommended installation channel.

  • Docker is the de facto industry standard for containerization.

Important

Docker must be invocable without explicit use of sudo. Check to see that the following works.

--rm just tells docker to clean up after itself.

Terminal#
docker run --rm hello-world

There is this guide for linux systems

The agent must be configured to use the installed container runtime.

 1smith = Agent(
 2    ...
 3    runtime=ContainerRuntime.APPTAINER,
 4)
 5
 6# or
 7smith = Agent(
 8    ...
 9    runtime=ContainerRuntime.DOCKER,
10)

SLURM#

Provided Nextflow config presets are retrievable by name, including one for HPC platforms using SLURM.

 1with open("../secrets/slurm_account") as f:
 2    SLURM_ACCOUNT = f.readline()
 3
 4smith.RunWorkflow(
 5    task,
 6    config_file=smith.GetNxfConfigPresets()["slurm"],
 7    params=dict(
 8        slurmAccount=SLURM_ACCOUNT,
 9    )
10)

Tip

Here is the Nextflow documentation on

  • config <https://www.nextflow.io/docs/latest/config.html>

  • parameters <https://www.nextflow.io/docs/latest/cli.html#pipeline-parameters>

These are just paths to Nextflow config files, so feel free to point to your own config as well. Metasmith doesn’t require a specific Nextflow configuration.

1smith.RunWorkflow(
2    task,
3    config_file="path/to/my/config.nf",
4)

Tip

task is the result of smith.GenerateWorkflow(...)

On this page