Hydrothermal Vent AI Benchmark: Unseen-Field Testing in 2026

A model can score 95% on hydrothermal-vent samples and still fail at the first vent field it has never seen. Random splits often place near-identical genomes, neighboring samples, and expedition-specific artifacts in both training and test sets.

A useful hydrothermal vent AI benchmark asks a harder question: can a model transfer physical, ecological, and evolutionary information to a new vent field, basin, expedition, or microbial lineage?

Key takeaways

• Geography usually matters more than randomness. DSV70’s 70 metagenomes from 21 vent fields support leave-one-field-out testing, but not casual random splits.
• No single dataset captures the system. WOA23 provides broad physical context, Argo and BGC-Argo provide water-column profiles, OOI contributes persistent observations, and DSV70 anchors genome-resolved biology.
• Start with late fusion. Separate physics, chemistry, imagery, and genome models before combining predictions. That makes missing data and leakage easier to diagnose.

Why random splits make vent AI look smarter than it is

Hydrothermal-vent data are clustered in space, time, and ancestry. Samples may come from the same chimney, expedition, sequencing run, or microbial population. A random split can distribute reads from one population—or images from one deployment—across both training and test sets.

The model then learns shortcuts:

• Sequencing-platform artifacts
• Expedition and laboratory signatures
• Vent-field identity
• Repeated sampling from one structure
• Closely related strains
• Image backgrounds associated with one research cruise

Consider a model trained to predict sulfur-oxidation genes. If most sulfur-rich samples came from one expedition, the model may learn that expedition’s library-preparation pattern or read-quality profile instead of the biology. It can perform brilliantly on a random test set, then fail on a new cruise where the same pathway is present but the technical signature has changed.

The first safeguard is grouped evaluation. In a leave-one-vent-field-out test, every sample from one field stays together:

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=21)

for train_idx, test_idx in cv.split(X, y, groups=vent_field_id):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]

The split should match the scientific question:

For DSV70, the headline experiment is simple: train on 20 fields, test on the 21st, and rotate until every field has been held out. Report the mean, median, and worst-field result. The worst field often reveals more than the average.

A fall from 95% random-split accuracy to 68% under geographic holdout isn’t automatically a failure. It may be the first honest measurement of generalization.

Build a data stack with distinct jobs

These resources operate at different spatial and temporal scales. Combining them without respecting those differences produces false precision.

WOA23 is a climatological background layer. Its gridded products cover temperature, salinity, oxygen, nitrate, phosphate, and silicate, with temperature extending to roughly 5,500 meters. It’s useful for environmental normalization and filling broad gaps, but it cannot describe the temperature a meter from an active chimney.

Argo and BGC-Argo provide observed profiles. Standard floats commonly cycle on roughly 10-day schedules and profile to about 2,000 dbar. Biogeochemical instruments can add oxygen, chlorophyll, backscatter, nitrate, pH, and irradiance. For vent work, they’re best treated as water-column context—not direct sensors of individual hydrothermal systems.

OOI fills a different role with persistent, high-frequency observations, including sensor time series, imagery, and chemical measurements. It can help connect a physical anomaly to habitat or biological change. Its data are rich but geographically concentrated, so the network shouldn’t be treated as a representative sample of the deep ocean.

DSV70 provides the genomic core. The reported resource contains 70 metagenomes from 21 hydrothermal vent fields, collected between 1993 and 2009, with approximately 3.56 terabases of raw reads and 7,422 medium- to high-quality MAGs: 6,063 bacterial and 1,359 archaeal. That supports functional annotation, phylogenomics, pangenome analysis, and leave-one-field-out testing.

It isn’t a uniform census. Sequencing technologies, sampling protocols, years, and research priorities differ across the collection. Those differences belong in the benchmark’s leakage audit, not in the model’s shortcut features.

Four benchmark tracks

A credible benchmark should establish focused tracks before introducing a large multimodal model.

1. Physics

Inputs can include gridded WOA23 variables, Argo or BGC-Argo profiles, OOI sensor values, depth, pressure, coordinates, and time.

Useful targets include missing-value reconstruction, oxygen anomalies, nutrient concentration, and short-term sensor forecasting. Start with persistence, climatology, interpolation, and gradient-boosted trees. Only then compare them with recurrent, convolutional, or transformer-based time-series models.

Report error and calibration, not just correlation. A neural model that barely beats climatology on a held-out field isn’t ready to support biological claims.

2. Ecology

Inputs may combine ROV or OOI imagery with temperature, oxygen, pH, substrate, and habitat labels. Targets can include habitat class, taxon presence, community composition, and disturbance detection.

Hydrothermal systems shouldn’t be reduced to “vent” versus “non-vent.” Useful labels distinguish active chimney, diffuse flow, peripheral habitat, plume, sediment, bare rock, organism-associated substrate, and background deep-sea environment. At Lucky Strike, community differences have been linked to vent exposure, topography, substrate hardness, and distance from active emissions. Peripheral areas can be more diverse than the hottest zones.

Use macro-F1, balanced accuracy, Matthews correlation coefficient, precision-recall AUC, and community-distance measures such as Bray-Curtis dissimilarity. These metrics prevent abundant classes from hiding failure on rare but important habitats.

3. Genomics

Inputs can be raw reads, assemblies, MAGs, gene families, taxonomic profiles, and functional annotations. Targets include vent-field classification, phylogenetic placement, gene-family presence, functional prediction, and pangenome similarity.

Track both technical and biological performance: MAG completeness and contamination, mapping rate, assembly statistics, taxonomic macro-F1, functional precision and recall, average nucleotide identity error, and phylogenetic distance.

Functional targets are often more revealing than species labels. Carbon fixation, sulfur oxidation, hydrogen metabolism, methanogenesis, metal resistance, and oxidative-stress response may transfer across sites better than taxonomic identity. Predicting sulfur oxidation in a new field is stronger evidence of ecological transfer than recognizing a familiar strain.

4. Multimodal fusion

Use separate encoders for physics, chemistry, imagery, genomes, and metadata. Early fusion is simple but brittle when a modality is missing. Cross-attention and graph models can learn richer relationships, but they require better alignment between samples and stronger controls.

Late fusion is the sensible starting point: train each modality separately, combine predictions, and add one input group at a time. Measure every improvement on held-out fields rather than on a pooled test set.

One defensible evaluation recipe

Define the prediction unit before modeling. For example, one MAG can be the unit for functional prediction, while one timestamped sensor window can be the unit for oxygen forecasting. Don’t mix reads, MAGs, and samples in the same split without an explicit hierarchy.

For each leave-one-field-out fold:

1. Hold out every sample and genome from one vent field.
2. Fit imputation, scaling, feature selection, annotation thresholds, and dimensionality reduction on training data only.
3. Cluster related genomes using training data only. Do not compute a similarity or phylogenetic clustering on the complete dataset and then split it; that allows held-out information to influence group assignment.
4. Train the model on the remaining fields.
5. Apply the frozen preprocessing and model to the held-out field.
6. Report per-field metrics and aggregate them across fields.

For confidence intervals, bootstrap the held-out fields—not individual reads or MAGs—because samples from one field are correlated. A 95% interval can come from 2,000 resamples of the 21 field-level results. Also report class support so a wide interval or apparently strong score isn’t mistaken for stable evidence.

GroupKFold handles the grouping column you provide; it does not discover biological relatedness. If related strains cross the split, the benchmark still has lineage leakage. The grouping strategy must therefore be designed before training, with clustering and thresholds fitted without access to the held-out fold.

Practical rules for adoption

Keep a versioned record of accession numbers, download dates, checksums, taxonomy databases, annotation tools, MAG filters, sensor-quality flags, and missing-data treatment. Database updates can change labels and embeddings even when the model code stays the same.

Run negative controls. Shuffle environmental labels, hold out sequencing batches, and test whether technical metadata alone predicts vent field. If it does, the biological result needs stronger safeguards.

Report per-field results, worst-field performance, confidence intervals, and the number of held-out samples. A single pooled score can conceal complete failure in one basin.

The practical division of labor is straightforward: use WOA23 for broad context, Argo and BGC-Argo for water-column validation, OOI for persistent local observations, and DSV70 for genome-resolved vent benchmarks. Bring in MGnify, NCBI, and ENA when broader cross-study comparison matters.

The strongest model won’t be the one with the highest random-split score. It’ll be the one that remains useful when the field, expedition, basin, or lineage is new—and can show which physical or chemical evidence supports its prediction.

Frequently asked questions

How should AI be tested on unseen hydrothermal vents?

Group all data by vent field, train on every field except one, and test only on the held-out field. Repeat across fields, then audit expedition, sequencing-batch, replicate, and lineage leakage.

What is DSV70?

DSV70 is a hydrothermal-vent metagenomic resource containing 70 metagenomes from 21 fields. Its reported genome-resolved output includes about 7,422 medium- to high-quality MAGs: 6,063 bacterial and 1,359 archaeal.

Can Argo resolve an individual hydrothermal vent?

Usually not. Argo and BGC-Argo are valuable for broad water-column structure and biogeochemical validation, but their spacing, depth limits, and sampling cycles generally cannot resolve meter-scale chimney or plume gradients.