Open-Omics-ESM
Open-Omics-ESM
Open-Omics-ESM is an optimized version of the Evolutionary Scale Modeling (ESM) toolkit, tailored for modern CPUs. It enhances the performance of key ESM modules—such as ESM-embeddings, LM-Design, InverseFolding, and ESMFold—by leveraging the Intel Extension for PyTorch (IPEX), the latest version of PyTorch, and lower-precision (bf16) computations.
Installation
Step 1: Run the script to create the Docker images
Execute the script using the following command:
./build_docker_images.sh
Step 2: Choose the ESM image to install
Select the ESM image you want to build based on your specific needs
Given the options, here’s how you might decide:
- esm -
Dockerfile.esmis configured to ESM-Embeddings, LM-Designs, and InverseFolding with optimizations for Python 3.11 and PyTorch 2.4.0, providing enhanced performance and speed compared to the original open-sourced ESM. - esm_fold -
Dockerfile.esmfoldis configured to ESMFold with optimizations for Python 3.7 and PyTorch 1.12.1, including Intel extensions (IPEX) to provide enhanced performance and speed. Unlike OpenFold, which also relies on PyTorch 1.12.1, ESMFold includes additional optimizations specific to Intel architectures, offering improved efficiency compared to the original open-sourced ESMFold. - Both esm and esm_fold - build both images to incorporate both ESM functions and folding capabilities, covering all use cases. This option will require additional build time and disk space
Which images do you want to build? #type task number
1 esm
2 esm_fold
3 Both esm and esm_fold
Running
Information on flags
Performance optimization with bfloat16 and Intel Extension for PyTorch
--bf16 flag accelerates performance by utilizing bfloat16 precision, which enhances computational efficiency without compromising accuracy.
Export Directories
export INPUT=$PWD/<your input folder>
export OUTPUT=$PWD/<your output folder>
export MODELS=$PWD/<your output folder>
The following is an example to do the same.
mkdir -p input output models
export INPUT=$PWD/input
export OUTPUT=$PWD/output
export MODELS=$PWD/models
chmod a+w $MODELS $OUTPUT
The input directory must contain both FASTA_FILE and PDB_FILE for the tool to process sequence and structural data
Run Commands
In this ESM setup, the input directory contains files like FASTA (protein sequences) and PDB (protein structures) for different tasks such as sequence extraction and protein folding. Each file type has a specific use. For testing, follow the provided instructions or refer to the research papers for more details.
When you run the Docker command, the models will automatically be downloaded during the first run. The user will need to wait for the download to complete, but from the second run onward, the inference will run directly without downloading the models again.
ESM-Embeddings
Compute embeddings for multiple protein sequences from a FASTA file in a single batch process using ESM.See Compute embeddings in bulk from FASTA for detailed user guide.
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest \
python scripts/extract.py esm2_t33_650M_UR50D /input/some_proteins.fasta /output --repr_layers 0 32 33 --include mean per_tok --bf16
LM-Design
LM-Design supports fixed backbone sequence generation for known structures and random sequence generation for exploratory protein design.See examples/lm-design/ for detailed user guide.
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest bash -c \
"cd examples/lm-design && python -m lm_design task=fixedbb pdb_fn=/input/2N2U.pdb bf16=True &> /output/fixed_backbone_log"
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $PWD/output:/output \
esm_image:latest bash -c \
"cd examples/lm-design && python -m lm_design task=free_generation bf16=True &> /output/free_generation_log"
Inverse_Folding
Inverse Folding involves generating sample protein sequences that are designed to match a specified structure and scoring these sequences based on their folding compatibility.See examples/inverse_folding/ for detailed user guide.
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest \
python examples/inverse_folding/sample_sequences.py /input/5YH2.pdb --chain C --temperature 1 --num-samples 3 --outpath /output/sampled_sequences.fasta --bf16
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest \
python examples/inverse_folding/score_log_likelihoods.py /input/5YH2.pdb /input/5YH2_mutated_seqs.fasta --chain C --outpath /output/scores.csv --bf16
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest \
python examples/inverse_folding/sample_sequences.py /input/5YH2.pdb --chain C --temperature 1 --num-samples 3 --outpath /output/mb_sampled_sequences.fasta --multichain-backbone --bf16
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esm_image:latest \
python examples/inverse_folding/score_log_likelihoods.py /input/5YH2.pdb /input/5YH2_mutated_seqs.fasta --chain C --outpath /output/mb_scores.csv --multichain-backbone --bf16
ESMFold Structure Prediction
ESMFold predicts protein structures from amino acid sequences using advanced machine learning techniques, facilitating rapid structural insights in protein design.See ESMFold Structure Prediction for detailed user guide.
#example
docker run -it \
-v $MODELS:/checkpoints \
-v $INPUT:/input \
-v $OUTPUT:/output \
esmfold_image:latest \
python scripts/fold.py -i /input/few_proteins.fasta -o /output --bf16
The original README content of ESM follows.
Evolutionary Scale Modeling
Update April 2023: Code for the two simultaneous preprints on protein design is now released! Code for “Language models generalize beyond natural proteins” is under examples/lm-design/. Code for “A high-level programming language for generative protein design” is under examples/protein-programming-language/.
This repository contains code and pre-trained weights for Transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR), including our state-of-the-art ESM-2 and ESMFold, as well as MSA Transformer, ESM-1v for predicting variant effects and ESM-IF1 for inverse folding. Transformer protein language models were introduced in the 2019 preprint of the paper “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences”. ESM-2 outperforms all tested single-sequence protein language models across a range of structure prediction tasks. ESMFold harnesses the ESM-2 language model to generate accurate structure predictions end to end directly from the sequence of a protein.
In November 2022, we released v0 of the ESM Metagenomic Atlas, an open atlas of 617 million predicted metagenomic protein structures.
The Atlas was updated in March 2023 in collaboration with EBI. The new v2023_02 adds another 150 million predicted structures to the Atlas, as well as pre-computed ESM2 embeddings.
Bulk download, blog post and the resources provided on the Atlas website are documented on this README.
In December 2022, we released two simultaneous preprints on protein design.
- “Language models generalize beyond natural proteins” (PAPER, CODE) uses ESM2 to design de novo proteins. The code and data associated with the preprint can be found here.
- “A high-level programming language for generative protein design” (PAPER, CODE) uses ESMFold to design proteins according to a high-level programming language.
Citation
For ESM2, ESMFold and ESM Atlas: ```bibtex @article{lin2023evolutionary, title = {Evolutionary-scale prediction of atomic-level protein structure with a language model}, author = {Zeming Lin and Halil Akin and Roshan Rao and Brian Hie and Zhongkai Zhu and Wenting Lu and Nikita Smetanin and Robert Verkuil and Ori Kabeli and Yaniv Shmueli and Allan dos Santos Costa and Maryam Fazel-Zarandi and Tom Sercu and Salvatore Candido and Alexander Rives }, journal = {Science}, volume = {379}, number = {6637}, pages = {1123-1130}, year = {2023}, doi = {10.1126/science.ade2574}, URL = {https://www.science.org/doi/abs/10.1126/science.ade2574}, note={Earlier versions as preprint: bioRxiv 2022.07.20.500902}, } ``` For transformer protein language models: ```bibtex @article{rives2021biological, title={Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences}, author={Rives, Alexander and Meier, Joshua and Sercu, Tom and Goyal, Siddharth and Lin, Zeming and Liu, Jason and Guo, Demi and Ott, Myle and Zitnick, C Lawrence and Ma, Jerry and others}, journal={Proceedings of the National Academy of Sciences}, volume={118}, number={15}, pages={e2016239118}, year={2021}, publisher={National Acad Sciences}, note={bioRxiv 10.1101/622803}, doi={10.1073/pnas.2016239118}, url={https://www.pnas.org/doi/full/10.1073/pnas.2016239118}, } ```Table of contents
- [Main models you should use](#main-models) - [Usage](#usage) - [Quick Start](#quickstart) - [Getting Started with this repository](#repostart) - [ESMFold Structure Prediction](#esmfold) - [Compute embeddings in bulk from FASTA](#bulk_fasta) - [CPU offloading for inference with large models](#fsdp) - [Zero-shot variant prediction](#zs_variant) - [Inverse folding](#invf) - [ESM Metagenomic Atlas](#atlas) - [Notebooks](#notebooks) - [Available Models and Datasets](#available) - [Pre-trained Models](#available-models) - [ESM Structural Split Dataset](#available-esmssd) - [Pre-training Dataset Split](#available-pretraining-split) - [Comparison to related works](#perf_related) - [Citations](#citations) - [License](#license)What's New
- April 2023: Code for the protein design preprints released under [examples/lm-design/](examples/lm-design/). - March 2023: We release an update to the ESM Metagenomic Atlas, `v2023_02`. See [website](https://esmatlas.com/) and [bulk download details](#atlas). - December 2022: The Meta Fundamental AI Research Protein Team (FAIR) released two simultaneous preprints on protein design: ["Language models generalize beyond natural proteins" (Verkuil, Kabeli, et al., 2022)](https://doi.org/10.1101/2022.12.21.521521), and ["A high-level programming language for generative protein design" (Hie, Candido, et al., 2022)](https://doi.org/10.1101/2022.12.21.521521). - November 2022: ESM Metagenomic Atlas, a repository of 600M+ metagenomics structures released, see [website](https://esmatlas.com/) and [bulk download details](#atlas) - November 2022: ESMFold - new end-to-end structure prediction model released (see [Lin et al. 2022](https://www.science.org/doi/abs/10.1126/science.ade2574)) - August 2022: ESM-2 - new SOTA Language Models released (see [Lin et al. 2022](https://www.science.org/doi/abs/10.1126/science.ade2574)) - April 2022: New inverse folding model ESM-IF1 released, trained on CATH and UniRef50 predicted structures. - August 2021: Added flexibility to tokenizer to allow for spaces and special tokens (like `
](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/inverse_folding/notebook_multichain.ipynb)
The ESM-IF1 inverse folding model is built for predicting protein sequences
from their backbone atom coordinates. We provide scripts here 1) to sample sequence
designs for a given structure and 2) to score sequences for a given structure.
Trained with 12M protein structures predicted by AlphaFold2, the ESM-IF1
model consists of invariant geometric input processing layers followed by a
sequence-to-sequence transformer, and achieves 51% native sequence recovery on
structurally held-out backbones with 72% recovery for buried residues.
The model is also trained with span masking to tolerate missing backbone
coordinates and therefore can predict sequences for partially masked structures.
#### Sample sequence designs for a given structure
The environment setup is described in [this subsection of examples/inverse_folding](examples/inverse_folding#recommended-environment).
To sample sequences for a given structure in PDB or mmCIF format, use the
`sample_sequences.py` script. The input file can have either `.pdb` or
`.cif` as suffix.
For example, to sample 3 sequence designs for the golgi casein kinase structure
(PDB [5YH2](https://www.rcsb.org/structure/5yh2); [PDB Molecule of the Month
from January 2022](https://pdb101.rcsb.org/motm/265)), we can run the following
command from the esm root directory:
```bash
python examples/inverse_folding/sample_sequences.py examples/inverse_folding/data/5YH2.pdb \
--chain C --temperature 1 --num-samples 3 --outpath examples/inverse_folding/output/sampled_sequences.fasta
```
The sampled sequences will be saved in a fasta format to the specified output file.
The temperature parameter controls the sharpness of the probability
distribution for sequence sampling. Higher sampling temperatures yield more
diverse sequences but likely with lower native sequence recovery.
The default sampling temperature is 1. To optimize for native sequence
recovery, we recommend sampling with low temperature such as 1e-6.
#### Scoring sequences
To score the conditional log-likelihoods for sequences conditioned on a given
structure, use the `score_log_likelihoods.py` script.
For example, to score the sequences in `examples/inverse_folding/data/5YH2_mutated_seqs.fasta`
according to the structure in `examples/inverse_folding/data/5YH2.pdb`, we can run
the following command from the esm root directory:
```
python examples/inverse_folding/score_log_likelihoods.py examples/inverse_folding/data/5YH2.pdb \
examples/inverse_folding/data/5YH2_mutated_seqs.fasta --chain C \
--outpath examples/inverse_folding/output/5YH2_mutated_seqs_scores.csv
```
The conditional log-likelihoods are saved in a csv format in the specified output path.
The output values are the average log-likelihoods averaged over all amino acids in a sequence.
For more information, see "[./examples/inverse_folding/](examples/inverse_folding/)" for detailed user guide.
## ESM Metagenomic Atlas
Please visit the [ESM Metagenomic Atlas](https://esmatlas.com/) website, and
see our [blog post](https://ai.facebook.com/blog/protein-folding-esmfold-metagenomics/) to learn more.
Bulk download instructions available at a seperate README [here](scripts/atlas/README.md).
The Atlas resources include a page to [fold a sequence using ESMFold](https://esmatlas.com/resources?action=fold),
searching a subset of the ESM Atlas by [structure](https://esmatlas.com/resources?action=search_structure) or
[sequence](https://esmatlas.com/resources?action=search_sequence),
as well as an [API](https://esmatlas.com/about#api) to access those resources programmatically.
Foldseek provides search against the Atlas without the length limitation [here](https://search.foldseek.com/search).
## Notebooks
### Inverse folding - predicting or scoring sequences based on backbone structures
[](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/inverse_folding/notebook.ipynb)
The ESM-IF1 inverse folding model predicts protein sequences from their backbone atom coordinates, trained with 12M protein structures predicted by AlphaFold2.
This notetook guide you through examples of sampling sequences, calculating conditional log-likelihoods, and extracting encoder output as structure representation.
### Supervised variant prediction - training a classifier on the embeddings
[](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/sup_variant_prediction.ipynb)
To help you get started with using the embeddings, this [jupyter notebook tutorial](examples/sup_variant_prediction.ipynb) shows how to train a supervised variant predictor using embeddings from ESM-1.
You can adopt a similar protocol to train a model for any downstream task, even with limited data.
First you can obtain the embeddings for ``examples/data/P62593.fasta`` either by [downloading the precomputed](https://dl.fbaipublicfiles.com/fair-esm/examples/P62593_reprs.tar.gz) embeddings
as instructed in the notebook or by running the following:
```bash
# Obtain the embeddings
python scripts/extract.py esm1v_t33_650M_UR90S_1 examples/data/P62593.fasta \
examples/data/P62593_emb_esm1v --repr_layers 33 --include mean
```
Then, follow the remaining instructions in the tutorial. You can also run the tutorial in a [colab notebook](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/sup_variant_prediction.ipynb).
**Note, alternatively use [the newer instructions for zero-shot variant prediction](examples/variant-prediction/),
which predicts mutational effects without any supervised training.**
### Unsupervised contact prediction
[](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/contact_prediction.ipynb)
This [jupyter notebook tutorial](examples/contact_prediction.ipynb) demonstrates contact prediction with both the ESM-2 and MSA Transformer (ESM-MSA-1) models.
Contact prediction is based on a logistic regression over the model's attention maps.
This methodology is based on our ICLR 2021 paper,
[Transformer protein language models are unsupervised structure learners. (Rao et al. 2020)](https://doi.org/10.1101/2020.12.15.422761)
The MSA Transformer (ESM-MSA-1) takes a multiple sequence alignment (MSA) as input, and uses the tied row self-attention maps in the same way.
See [MSA Transformer. (Rao et al. 2021)](https://www.biorxiv.org/content/10.1101/2021.02.12.430858v1).
To get unsupervised attention-based contacts, call `model.predict_contacts(tokens)` or `model(tokens, return_contacts=True)`.
### ESMStructuralSplitDataset and self-attention contact prediction
[](https://colab.research.google.com/github/facebookresearch/esm/blob/main/examples/esm_structural_dataset.ipynb)
And this [jupyter notebook tutorial](examples/esm_structural_dataset.ipynb) shows how to load and index the `ESMStructuralSplitDataset`,
and computes the self-attention map unsupervised contact predictions using ESM-2.
## Available Models and Datasets
### Pre-trained Models
| Shorthand | `esm.pretrained.` | #layers | #params | Dataset | Embedding Dim | Model URL (automatically downloaded to `~/.cache/torch/hub/checkpoints`) |
|-----------|---------------------|---------|-------------|---------|---------------|-----------------------------------------------------------------------|
| ESM-2 | `esm2_t48_15B_UR50D` | 48 | 15B | UR50/D 2021_04 | 5120 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t48_15B_UR50D.pt |
| | `esm2_t36_3B_UR50D` | 36 | 3B | UR50/D 2021_04 | 2560 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t36_3B_UR50D.pt |
| | `esm2_t33_650M_UR50D` | 33 | 650M | UR50/D 2021_04 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t33_650M_UR50D.pt |
| | `esm2_t30_150M_UR50D` | 30 | 150M | UR50/D 2021_04 | 640 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t30_150M_UR50D.pt |
| | `esm2_t12_35M_UR50D` | 12 | 35M | UR50/D 2021_04 | 480 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t12_35M_UR50D.pt |
| | `esm2_t6_8M_UR50D` | 6 | 8M | UR50/D 2021_04 | 320 | https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t6_8M_UR50D.pt |
| ESMFold | `esmfold_v1` | 48 (+36) | 690M (+3B) | UR50/D 2021_04 | - | https://dl.fbaipublicfiles.com/fair-esm/models/esmfold_3B_v1.pt |
| | `esmfold_v0` | 48 (+36) | 690M (+3B) | UR50/D 2021_04 | - | https://dl.fbaipublicfiles.com/fair-esm/models/esmfold_3B_v0.pt |
| | `esmfold_structure_module_only_*` | 0 (+various) | various | UR50/D 2021_04 | - | https://dl.fbaipublicfiles.com/fair-esm/models/esmfold_structure_module_only_* |
| ESM-IF1 | `esm_if1_gvp4_t16_142M_UR50` | 20 | 124M | CATH 4.3 + predicted structures for UR50 | 512 | https://dl.fbaipublicfiles.com/fair-esm/models/esm_if1_gvp4_t16_142M_UR50.pt |
| ESM-1v | `esm1v_t33_650M_UR90S_[1-5]` | 33 | 650M | UR90/S 2020_03 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1v_t33_650M_UR90S_1.pt |
| ESM-MSA-1b| `esm_msa1b_t12_100M_UR50S` | 12 | 100M | UR50/S + MSA 2018_03 | 768 | https://dl.fbaipublicfiles.com/fair-esm/models/esm_msa1b_t12_100M_UR50S.pt |
| ESM-MSA-1 | `esm_msa1_t12_100M_UR50S` | 12 | 100M | UR50/S + MSA 2018_03 | 768 | https://dl.fbaipublicfiles.com/fair-esm/models/esm_msa1_t12_100M_UR50S.pt |
| ESM-1b | `esm1b_t33_650M_UR50S` | 33 | 650M | UR50/S 2018_03 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1b_t33_650M_UR50S.pt |
| ESM-1 | `esm1_t34_670M_UR50S` | 34 | 670M | UR50/S 2018_03 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1_t34_670M_UR50S.pt |
| | `esm1_t34_670M_UR50D` | 34 | 670M | UR50/D 2018_03 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1_t34_670M_UR50D.pt |
| | `esm1_t34_670M_UR100` | 34 | 670M | UR100 2018_03 | 1280 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1_t34_670M_UR100.pt |
| | `esm1_t12_85M_UR50S` | 12 | 85M | UR50/S 2018_03 | 768 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1_t12_85M_UR50S.pt |
| | `esm1_t6_43M_UR50S` | 6 | 43M | UR50/S 2018_03 | 768 | https://dl.fbaipublicfiles.com/fair-esm/models/esm1_t6_43M_UR50S.pt |
Here is a chronological list of the released models and the paper they were introduced in:
| Shorthand | Release Notes |
|------------|---------------|
| ESM-1 | Released with Rives et al. 2019 (Aug 2020 update). |
| ESM-1b | Released with Rives et al. 2019 (Dec 2020 update). See Appendix B. |
| ESM-MSA-1 | Released with Rao et al. 2021 (Preprint v1). |
| ESM-MSA-1b | Released with Rao et al. 2021 (ICML'21 version, June 2021). |
| ESM-1v | Released with Meier et al. 2021. |
| ESM-IF1 | Released with Hsu et al. 2022. |
| ESM-2 | Released with Lin et al. 2022. |
### ESM Structural Split Dataset
This is a five-fold cross validation dataset of protein domain structures that can be used to measure generalization of representations
across different levels of structural dissimilarity.
The dataset implements structural holdouts at the family, superfamily, and fold
level. The SCOPe database is used to classify domains. Independently for each level of structural hold-out,
the domains are split into 5 equal sets, i.e. five sets of folds, superfamilies, or families. This ensures
that for each of the five partitions, structures having the same classification do not appear in both the
train and test sets. For a given classification level each structure appears in a test set once, so that
in the cross validation experiment each of the structures will be evaluated exactly once.
The dataset provides 3d coordinates, distance maps, and secondary structure labels.
For further details on the construction of the dataset
see [Rives et al. 2019](https://doi.org/10.1101/622803) Appendix A.10.
This [jupyter notebook tutorial](examples/esm_structural_dataset.ipynb) shows how to load and index the `ESMStructuralSplitDataset`.
`ESMStructuralSplitDataset`, upon initializing, will download `splits` and `pkl`.
We also provide `msas` for each of the domains. The data can be directly downloaded below.
| Name | Description | URL |
|--------|-------------------------------------------------------------------------------|-----------------------------------------------------------------------|
| splits | train/valid splits | https://dl.fbaipublicfiles.com/fair-esm/structural-data/splits.tar.gz |
| pkl | pkl objects containing sequence, SSP labels, distance map, and 3d coordinates | https://dl.fbaipublicfiles.com/fair-esm/structural-data/pkl.tar.gz |
| msas | a3m files containing MSA for each domain | https://dl.fbaipublicfiles.com/fair-esm/structural-data/msas.tar.gz |
### Pre-training Dataset Split
The split files establishing which UniRef50 clusters were used as held-out evaluation set for pre-training
in [Rives et al. 2019](https://doi.org/10.1101/622803) and [Rao et al. 2021](https://doi.org/10.1101/2021.02.12.430858) can be found here:
* [UniRef50 IDs of evaluation set](https://dl.fbaipublicfiles.com/fair-esm/pretraining-data/uniref201803_ur50_valid_headers.txt.gz): 3.016 M clusters
* [UniRef100 IDs of evaluation set](https://dl.fbaipublicfiles.com/fair-esm/pretraining-data/uniref201803_ur100_valid_headers.txt.gz): 13.745 M proteins, expanding the same UniRef50 clusters.
These files only contain only the UniRef50 IDs and UniRef100 IDs corresponding to the [UniRef database, 2018-03 release](https://ftp.uniprot.org/pub/databases/uniprot/previous_releases/release-2018_03/uniref/)
which is released by the UniProt Consortium under a [Creative Commons Attribution (CC BY 4.0) License](https://www.uniprot.org/help/license).
### Comparison to related works
| Task | Unsupervised contact prediction | Structure Prediction | |||
|---|---|---|---|---|---|
| Test set | Large valid | CASP14 | CAMEO (Apr-Jun 2022) | CASP14 | CAMEO (Apr-Jun 2022) |
| Gremlin (Potts) | 39.3 | ||||
| TAPE | 11.2 | ||||
| ProtBert-BFD | 34.1 | ||||
| Prot-T5-XL-BFD | 35.6 | 46.1 | 62.6 | ||
| Prot-T5-XL-Ur50 (3B) | 47.9 | 49.8 | 69.4 | ||
| ESM-1 | 33.7 | ||||
| ESM-1b | 41.1 | 24.4 | 39 | 41.6 | 64.5 |
| ESM-1v | 35.3 | ||||
| ESM-MSA-1b | 57.4 | ||||
| ESM-2 (8M) | 15.9 | 9.8 | 15.7 | 36.7 | 48.1 |
| ESM-2 (35M) | 28.8 | 16.4 | 28.4 | 41.4 | 56.4 |
| ESM-2 (150M) | 42.2 | 26.8 | 40.1 | 49.0 | 64.9 |
| ESM-2 (700M) | 50.1 | 32.5 | 47.6 | 51.3 | 70.1 |
| ESM-2 (3B) | 52.7 | 34.0 | 49.9 | 52.5 | 71.8 |
| ESM-2 (15B) | 54.5 | 37.0 | 51.7 | 55.4 | 72.1 |