littleworth/protgpt2-distilled-small

A compact protein language model distilled from ProtGPT2 using complementary-regularizer distillation---a method that combines uncertainty-aware position weighting with calibration-aware label smoothing to achieve 31% better perplexity than standard knowledge distillation at 3.8x compression.

A compact protein language model distilled from ProtGPT2 using complementary-regularizer distillation---a method that combines uncertainty-aware position weighting with calibration-aware label smoothing to achieve 87% better perplexity than standard knowledge distillation at 20x compression.

ESMC is a state-of-the-art protein language model that has learned the rules of protein biology from training on billions of protein sequences. ESMC provides representations of proteins enabling novel AI applications from therapeutic protein engineering to unlocking basic insights into protein…

The plant DNA large language models (LLMs) contain a series of foundation models based on different model architectures, which are pre-trained on various plant reference genomes. All the models have a comparable model size between 90 MB and 150 MB, BPE tokenizer is used for tokenization and 8000…

The ESMC scaling-study checkpoints are being released to support reproducibility of the findings in our paper, please refer to the paper and github for details. Please use the ESMC model for research work. ESMC is a state-of-the-art protein language model trained on billions of protein sequences…

The ESMC scaling-study checkpoints are being released to support reproducibility of the findings in our paper, please refer to the paper and github for details. Please use the ESMC model for research work. ESMC is a state-of-the-art protein language model trained on billions of protein sequences…