Train a machine learning model on a collection#
Here, we iterate over the artifacts within a collection to train a machine learning model at scale.
import lamindb as ln
import anndata as ad
import numpy as np
💡 lamindb instance: testuser1/test-scrna
ln.track()
💡 notebook imports: anndata==0.9.2 lamindb==0.66.0 numpy==1.26.3 torch==2.1.2
💡 saved: Transform(uid='Qr1kIHvK506r5zKv', name='Train a machine learning model on a collection', short_name='scrna5', version='1', type=notebook, updated_at=2024-01-08 10:59:33 UTC, created_by_id=1)
💡 saved: Run(uid='izru8PESPmtAQ6TL4zJt', run_at=2024-01-08 10:59:33 UTC, transform_id=5, created_by_id=1)
Query our collection:
collection = ln.Collection.filter(
name="My versioned scRNA-seq collection", version="2"
).one()
collection.describe()
Show code cell output
Collection(uid='5DyMdnj3745Du6o7Qz6M', name='My versioned scRNA-seq collection', version='2', hash='BOAf0T5UbN_iOe3fQDyq', visibility=1, updated_at=2024-01-08 10:59:11 UTC)
Provenance:
📔 transform: Transform(uid='ManDYgmftZ8C5zKv', name='Standardize and append a batch of data', short_name='scrna2', version='1', type='notebook', updated_at=2024-01-08 10:59:00 UTC, created_by_id=1)
👣 run: Run(uid='62PXJ97djXO9FcEsR1A8', run_at=2024-01-08 10:59:00 UTC, transform_id=2, created_by_id=1)
👤 created_by: User(uid='DzTjkKse', handle='testuser1', name='Test User1', updated_at=2024-01-08 10:58:30 UTC)
⬇️ input_of (core.Run): ['2024-01-08 10:59:23 UTC']
Features:
var: FeatureSet(uid='37wxKWB1pQvXJdcMl246', n=36390, type='number', registry='bionty.Gene', hash='rMZltwoBCMdVPVR8x6nJ', updated_at=2024-01-08 10:58:51 UTC, created_by_id=1)
'MIR1302-2HG', 'FAM138A', 'OR4F5', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'OR4F29', 'None', 'OR4F16', 'None', 'LINC01409', 'FAM87B', 'LINC01128', 'LINC00115', 'FAM41C', 'None', ...
obs: FeatureSet(uid='54TCuGFi4fgLKTSx6qpP', n=4, registry='core.Feature', hash='8RXpCP5VJaMl5Ic2cVLv', updated_at=2024-01-08 10:58:52 UTC, created_by_id=1)
🔗 cell_type (40, bionty.CellType): 'dendritic cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'CD38-positive naive B cell', 'CD4-positive, alpha-beta T cell', 'classical monocyte', 'T follicular helper cell', ...
🔗 assay (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
🔗 tissue (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
🔗 donor (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...
external: FeatureSet(uid='vAl5hqvUdvMU0DKUVPOZ', n=2, registry='core.Feature', hash='EuMYcG5nh63G7mSNQTi9', updated_at=2024-01-08 10:59:10 UTC, created_by_id=1)
🔗 assay (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
🔗 organism (1, bionty.Organism): 'human'
Labels:
🏷️ organism (1, bionty.Organism): 'human'
🏷️ tissues (17, bionty.Tissue): 'blood', 'thoracic lymph node', 'spleen', 'lung', 'mesenteric lymph node', 'lamina propria', 'liver', 'jejunal epithelium', 'omentum', 'bone marrow', ...
🏷️ cell_types (40, bionty.CellType): 'dendritic cell', 'B cell, CD19-positive', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'cytotoxic T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'CD14-positive, CD16-negative classical monocyte', 'CD38-positive naive B cell', 'CD4-positive, alpha-beta T cell', 'classical monocyte', 'T follicular helper cell', ...
🏷️ experimental_factors (4, bionty.ExperimentalFactor): '10x 3' v3', '10x 5' v2', '10x 5' v1', 'single-cell RNA sequencing'
🏷️ ulabels (12, core.ULabel): 'D496', '621B', 'A29', 'A36', 'A35', '637C', 'A52', 'A37', 'D503', '640C', ...
🏷️ unordered_artifacts (2, core.Artifact): 'scrna/conde22.h5ad', 'None'
Create a map-style dataset#
Let us create a map-style dataset using using mapped(): a MappedCollection. This is what, for example, the PyTorch DataLoader expects as an input.
Under-the-hood, it performs a virtual inner join of the features of the underlying AnnData objects and thus allows to work with very large collections.
You can either perform a virtual inner join:
with collection.mapped(label_keys=["cell_type"], join="inner") as dataset:
print(len(dataset.var_joint))
749
Or a virtual outer join:
dataset = collection.mapped(label_keys=["cell_type"], join="outer")
len(dataset.var_joint)
36503
This is compatible with a PyTorch DataLoader because it implements __getitem__ over a list of backed AnnData objects.
The 5th cell in the collection can be accessed like:
dataset[5]
Show code cell output
[array([0., 0., 0., ..., 0., 0., 0.], dtype=float32), 12]
The labels are encoded into integers:
dataset.encoders
Show code cell output
[{'CD4-positive helper T cell': 0,
'classical monocyte': 1,
'non-classical monocyte': 2,
'effector memory CD8-positive, alpha-beta T cell, terminally differentiated': 3,
'T follicular helper cell': 4,
'gamma-delta T cell': 5,
'lymphocyte': 6,
'CD16-negative, CD56-bright natural killer cell, human': 7,
'B cell, CD19-positive': 8,
'progenitor cell': 9,
'conventional dendritic cell': 10,
'alveolar macrophage': 11,
'cytotoxic T cell': 12,
'CD8-positive, alpha-beta memory T cell': 13,
'plasmacytoid dendritic cell': 14,
'dendritic cell, human': 15,
'CD8-positive, CD25-positive, alpha-beta regulatory T cell': 16,
'regulatory T cell': 17,
'mucosal invariant T cell': 18,
'alpha-beta T cell': 19,
'effector memory CD4-positive, alpha-beta T cell': 20,
'naive thymus-derived CD4-positive, alpha-beta T cell': 21,
'group 3 innate lymphoid cell': 22,
'CD4-positive, alpha-beta T cell': 23,
'naive thymus-derived CD8-positive, alpha-beta T cell': 24,
'animal cell': 25,
'megakaryocyte': 26,
'germinal center B cell': 27,
'naive B cell': 28,
'dendritic cell': 29,
'mast cell': 30,
'effector memory CD4-positive, alpha-beta T cell, terminally differentiated': 31,
'macrophage': 32,
'CD14-positive, CD16-negative classical monocyte': 33,
'plasma cell': 34,
'CD38-positive naive B cell': 35,
'plasmablast': 36,
'memory B cell': 37,
'CD16-positive, CD56-dim natural killer cell, human': 38,
'CD8-positive, alpha-beta memory T cell, CD45RO-positive': 39}]
Create a pytorch DataLoader#
Let us use a weighted sampler:
from torch.utils.data import DataLoader, WeightedRandomSampler
# label_key for weight doesn't have to be in labels on init
sampler = WeightedRandomSampler(
weights=dataset.get_label_weights("cell_type"), num_samples=len(dataset)
)
dataloader = DataLoader(dataset, batch_size=128, sampler=sampler)
We can now iterate through the data loader:
for batch in dataloader:
pass
Close the connections in MappedCollection:
dataset.close()
In practice, use a context manager
with collection.mapped(label_keys=["cell_type"]) as dataset:
sampler = WeightedRandomSampler(
weights=dataset.get_label_weights("cell_type"), num_samples=len(dataset)
)
dataloader = DataLoader(dataset, batch_size=128, sampler=sampler)
for batch in dataloader:
pass