My first agent#

This tutorial will demonstrate the minimal steps for using Metasmith. As an example, we will perform pangenome analysis on three E. coli genomes identified by their NCBI assembly accessions.

The Jupyter notebook for this tutorial can be obtained by:

Terminal#

$ msm get tutorials/my_first_agent.ipynb

Prerequisites#

Metasmith is installed along with either Docker or Apptainer, since we will be deploying an agent locally
A tutorial workspace has been setup for Jupyter notebooks

1 - Deploy an agent#

Agents, in this context, are virtual workers that perform complex tasks on the user’s behalf. To spwan an agent, Metasmith creates then deploys the agent to a home directory in which it will live. For this tutorial, the agent will live locally (on the same machine that Metasmith is installed on).

Tip

More on deploying agents, including to remote machines

Metasmith outsources the steps that compose an overall analysis pipeline to external software tools. To ensure that these tools can be reliably executed by an agent, self-contained software environments called “containers” are used. A container runtime downloads and manages the lifetime of these containers.

Metasmith can use the following container runtimes:

DOCKER
APPTAINER

Let’s create an agent called smith and give him a home in the current workspace under msm_home. We will instruct smith to manage containers with DOCKER.

agent_home = Source.FromLocal(WORKSPACE/"msm_home")
smith = Agent(
    home = agent_home,
    runtime=ContainerRuntime.DOCKER,
)

smith.Deploy()

2 - Register inputs#

Pangenome analysis seeks to compare a panel of genomes at the level of genes. Three E. coli genomes will be used as input, identified by their NCBI assembly accessions.

Metasmith accepts inputs in the form of files or values that become registered as DataInstances within a managed folder called a DataInstanceLibrary. Registering an input involves attaching a DataType that describes how it can be used. Computational steps specify a DataType for each of their inputs such that any DataInstance with a matching DataType can be consumed. Registering inputs as typed DataInstances enables Metasmith to determine which tools are capable of consuming them.

in_dir = WORKSPACE/"3pangenome.xgdb"
inputs = DataInstanceLibrary(in_dir)
inputs.Purge()  # clear the input folder, in case this is not the first time this cell was ran
inputs.AddTypeLibrary(MLIB/"data_types/ncbi.yml")
inputs.AddTypeLibrary(MLIB/"data_types/sequences.yml")
inputs.AddTypeLibrary(MLIB/"data_types/pangenome.yml")

group = inputs.AddValue("pangenome", "e coli", "pangenome::pangenome")
inputs.AddValue("DH10b",   "GCF_000019425.1", "ncbi::assembly_accession", parents={group})
inputs.AddValue("K12",     "GCF_000005845.2", "ncbi::assembly_accession", parents={group})
inputs.AddValue("EPI300",  "GCF_049667475.1", "ncbi::assembly_accession", parents={group})
inputs.Save()

Note

The concrete subtype ncbi::assembly_accession is required here — the more general ncbi::accession is abstract and will not match getNcbiAssembly’s input contract.

If you would rather start from a local .gbk file you have on disk, you can register it directly as sequences::gbk with inputs.AddItem(local_path, "sequences::gbk", parents={group}). Be aware that when an input already satisfies a downstream tool’s requirement, the planner will not schedule an upstream fetch for it — so mixing one local genome with two accessions will produce a workflow with a single-genome ppanggolin step rather than a three-genome one. Use accessions for all inputs, or local files for all inputs, when you want every genome to participate.

Tip

More on DataTypes, DataInstances, and DataInstanceLibraries

Using a try/except block to load the input library if is already created (instead of creating it each time) will enable automatic caching mechanisms in step 4 to reduce redundant computation:

in_dir = Path(WORKSPACE"/3pangenome.xgdb")
try:
    inputs = DataInstanceLibrary.Load(in_dir)
except:
    inputs = DataInstanceLibrary(in_dir)
    inputs.Purge() # just to be safe
    inputs.AddTypeLibrary(...)
    ...
    inputs.AddValue(...)
    ...

3 - Generate workflow#

Since computational steps transform input DataInstances into output DataInstances, they are called TransformInstances and are organized into special DataInstanceLibraries called a TransformInstanceLibrary. TransformInstances can be chained into workflows by matching the DataType of the upstream output to the DataType of the downstream input. The protocol of a TransformInstance is called a Transform and it may appear multiple times within a workflow.

When given a DataInstanceLibrary of inputs, a TransformInstanceLibrary of available tools, and target DataTypes, the agent can generate a workflow to produce DataInstances that match the target DataTypes, as long as a solution exists. Here, we request that targets of the type pangenome::heatmap be produced.

resources = [
    DataInstanceLibrary.Load(MLIB/f"resources/{n}")
    for n in ["containers", "lib"]
]

transforms = [
    TransformInstanceLibrary.Load(MLIB/f"transforms/{n}")
    for n in ["logistics", "pangenome"]
]

targets = TargetBuilder()
targets.Add("pangenome::heatmap")

task = smith.GenerateWorkflow(
    # divide the inputs into samples
    # we want all targets to be produced from each sample
    samples=inputs.AsSamples("ncbi::assembly_accession"),
    resources=resources,    # these are available for each sample, but need not be used
    transforms=transforms,
    targets=targets,
)

Tip

More on Transforms, and TransformInstances, and TransformInstanceLibraries

Each task has a code name or key composed of case sensitive letters and numbers that is calculated from the inputs and workflow steps. A task contains all the context required to execute a workflow.

The steps of the workflow can be rendered as a directed acyclic graph (DAG) or more commonly known as a flowchart. A DAG is a specific type of flowchart that has 2 properties:

“directed” indicates that for any two connected steps, data always flows from one to the other and never in reverse.
“acyclic” promises an implicit ording of steps such that once a step is performed, it will never be needed again.

Let’s take a look at the DAG for this generated workflow.

print(f'generated plan has [{len(task.plan.steps)}] steps')

workflow_diagram_path = f"{task.GetKey()}.dag.svg"
task.plan.RenderDAG(workflow_diagram_path)
print(f'diagram at [{workflow_diagram_path}]')

ipynbButtonLink(f"{workflow_diagram_path}", text="view workflow diagram")

Important

Notice how the ncbi::assembly_accession inputs are automatically “transformed” into sequences::gbk files by getNcbiAssembly to satisfy the input requirements of the pangenome analysis tool ppanggolin. All data types (shown in boxes) are valid as inputs or targets. Try different targets:

targets = TargetBuilder()
targets.Add("pangenome::ppanggolin_matrix")

# or
targets = TargetBuilder()
targets.Add("sequences::orfs")

Beware that there may not be a valid path for certain combinations of inputs and outputs, such as from sequences::gbk to sequences::orfs.

4 - Execute workflow#

To execute the workflow, it must first be staged to the agent’s home. This involves sending over the inputs, transform protocols, and translated nextflow workflow definition.

smith.StageWorkflow(task, on_exist="clear")

Since the agent home is local, you can view the results directly in the panel on the left.

metasmith_ws/
└──msm_home/
    ├── lib/
    ├── relay/
    ├── runs/
    │   └── ... # look for the tasks's key
    └── msm

A nextflow configuration is generated just before a run is triggered. We will use the “local” preset and lower the memory requirement to 2GB for all steps. The default resource estimates are liberal, but we know our task will only need to work with three genomes.

smith.RunWorkflow(
    task,
    config_file=smith.GetNxfConfigPresets()["local"],
    resource_overrides={
        "all": Resources(
            memory=Size.GB(2),
        )
    }
)

Tip

5 - Receive outputs#

We can ask for the location of a task from the agent and use it to load the produced DataInstanceLibrary that contains the workflow’s outputs.

results_path = smith.GetResultSource(task).GetPath()
results = DataInstanceLibrary.Load(results_path)

Once loaded, we can iterate through the results to find the heatmap since there should only be one output. We also make links to the main report files.

ipynbButtonLink(results_path/"_metadata/logs.latest/nxf_report.html")
ipynbButtonLink(results_path/"_metadata/logs.latest/nxf_timeline.html")

for path, type_name, endpoint in results.Iterate():
    if path.is_absolute(): continue # inputs have absolute paths
    ipynbButtonLink(results_path/path, f'view {type_name} {path.name}')

Next steps#

Other tutorials are available in the section panel on the left.