Representing data#

Data Types#

Motivations#

The interoperability problem that Metasmith seeks to solve can be summarized as determining if the output of one computational step can be immediately used as the input of another. The base case to be considered consists of three components: an upstream step, which produces an intermediate data product, that is then consumed by a downstream step. In bioinformatics, the conventional approach is to forego types and link the steps using the literal files that are produced. We will call this the “topology-centric” approach because interoperability is encoded by the connections between tools and intermediate files. The alternative proposed by Metasmith is to attach types to the inputs and outputs of each step. Outputs from the upstream step can be used as the inputs of the downstream step if the types match. We will call this the “contract-centric” approach because interoperability in each step’s contract, which describe the required inputs and promised outputs.

There are three key features that motivate the use of a contract-centric approach over a topology-centric approach.

  1. The contract of each step can be defined in isolation of other steps. While not sufficient on its own, this sets up the foundation for modular systems.

  2. Tools are means to an end. Obtaining correct results are often more important than the specific methods used. In this way, the contract-centric approach favours the user over the machine as it describes the data explicitly and leaves Metasmith to infer the topology. By contrast, the topology-centric approach describes the methods explicitly but leaves the no information about what is produced. For example, “a file produced by flye, megahit, spades, and hifiasm” is harder to comprehend than “a nucleotide sequence encoded in FASTA format”.

Definitions#

Metasmith describes data types using set of properties, which enables comparions between data to leverage set operations. For example, if we have two types: A = {1, 2} and B = {1, 2, 3}, then B may replace A since B can provide all properties that A can provide. Formally, if A is a subset of B (A ⊆ B), then A is substitutable by B. Practically, if the properties of an output A is a subset of the properties of required input B, then A can be used as the input B.

Data type objects within the Metasmith API are called Endpoints to avoid conflicts with python types and because they exist at either end of transforms.

1from metasmith.python_api import Endpoint

The following example shows how to create and compare endpoints.

1ball = Endpoint({"ball"})
2red_ball = Endpoint({"red", "ball"})
3red_ball.IsA(ball) # True, a red ball is a ball
4ball.IsA(red_ball) # False, a ball is not necessarily red

Data Type Library#

Endpoints can be gathered into a DataTypeLibrary and given a name for convenience.

1from metasmith.python_api import DataTypeLibrary

A DataTypeLibrary shares the same basic syntax as a python dict

1dtypes = DataTypeLibrary()
2dtypes["red_ball"] = Endpoint({"red", "ball"})

YAML#

Data type libraries can be persisted to disk and it may be more convenient to edit them as yaml files.

1dtypes.Save("dtypes.yml")
2dtypes = DataTypeLibrary.Load("dtypes.yml")

Note

To help with standardization across data type libraries, metadata can be included to describe the ontology of the data types.

In YAML form, properties of endpoints can be key-value pairs…

1from metasmith import examples
2dtypes = examples.DataTypeLibraries("template_keyval")

 1# ...
 2types:
 3    property_type_demo:
 4        properties:
 5            str: abc
 6            int: 1
 7            float: 0.3
 8            bool: True
 9            none: null
10            list:
11                - item 1
12                - item 2
13    contigs:
14        properties:
15            data: DNA sequence
16            format: FASTA
17    oci_image:
18        properties:
19            data: software container
20            format: OCI
21            provides:
22                - python==3.12
23                - some other tool

… or simple lists

1dtypes = examples.DataTypeLibraries("template_list")

 1# ...
 2types:
 3    property_type_demo:
 4        properties:
 5            - abc
 6            - 1
 7            - 0.3
 8            - True
 9            - null
10            - list item 1
11            - list item 2
12    contigs:
13        properties:
14            - data=DNA sequence
15            - format=FASTA
16    oci_image:
17        properties:
18            - data=software container
19            - format=OCI
20            - provides=python 3.12
21            - provides=some other tool

Below is a more realistic example themed after genomics.

1dtypes = examples.DataTypeLibraries("minimal_genomics")

 1ontology:
 2    doi: https://doi.org/10.1093/bioinformatics/btt113
 3    name: EDAM
 4    strict: false
 5    version: 1.25
 6schema: '1.0'
 7types:
 8    aa_sequences:
 9        properties:
10            data: Amino acid sequence
11            format: FASTA
12    contigs:
13        properties:
14            data: DNA sequence
15            format: FASTA
16    oci_image_blast:
17        properties:
18            data: software container
19            format: OCI
20            provides:
21                - blast
22    oci_image_prodigal:
23        properties:
24            data: software container
25            format: OCI
26            provides:
27                - prodigal
28    orf_annotations:
29        properties:
30            data: Protein features
31            format: CSV
32    protein_reference_fasta:
33        properties:
34            data: database reference
35            format: .faa

Data Instances#

A DataInstance refers to the piece of data that is described by an endpoint. Data instances are created when a file or folder is registered to a DataInstanceLibrary. On the filesystem, an XGDB is just a folder.

1from metasmith.python_api import DataInstanceLibrary
2xgdb = DataInstanceLibrary("./example.xgdb")

Note

For historical reasons, data instance libraries are shortened to “XGDB” (extended genome database) named after “PGDBs” (pathway genome database) from Pathway Tools and Metapathways.

Data type libraries must be added as namespaces to an XGDB before data instances can be added. The namespace "genomics" is added below.

1from metasmith import examples
2dtypes = examples.DataTypeLibraries("minimal_genomics")
3xgdb.AddTypeLibrary("genomics", dtypes)

The following information is required when adding new data instances:

  1. The path to the original file or folder

  2. The type of the data instance in the form "namespace::type"

1xgdb.AddItem("/path/to/original/contigs.fna", "genomics::contigs")
2xgdb.AddItem("/path/to/original/orfs.faa", "genomics::aa_sequences")
3xgdb.Save()

Once added, softlinks can by automatically generated for each input file within the XGDB. A prefix is added to ensure that file names are unique.

This creates the following directory structure:

example.xgdb/
├── _metasmith/
│   ├── index.yml
│   └── types/
│       └── genomics.yml
├── 1_contigs.fna           (symlink)
└── 2_orfs.faa              (symlink)

_metasmith/index.yml:

1# ...
2manifest:
3    /path/to/original/contigs.fna: genomics::contigs
4    /path/to/original/orfs.faa: genomics::aa_sequences
5# ...

_metasmith/types/genomics.yml:

 1# ...
 2types:
 3    contigs:
 4        properties:
 5            data: DNA sequence
 6            format: FASTA
 7    aa_sequences:
 8        properties:
 9            data: Amino acid sequence
10            format: FASTA
11# ...