--- license: other license_name: scienta-lab-eva-model-license license_link: LICENSE language: - en tags: - biology - transcriptomics - rna-seq - gene-expression - foundation-model - single-cell - bulk-rna - immunology library_name: transformers pipeline_tag: feature-extraction --- # EVA-RNA: Foundation Model for Transcriptomics Transformer-based foundation model that produces sample-level and gene-level embeddings from RNA-seq profiles (bulk, microarray, pseudobulked single-cell) in human and mouse. **You can find a complete end-to-end tutorial notebook [here](https://colab.research.google.com/#fileId=https%3A//huggingface.co/ScientaLab/eva-rna.ipynb).** ## Installation We recommend proceeding with the [uv package manager](https://docs.astral.sh/uv/getting-started/installation/). ```bash uv venv --python 3.10 source .venv/bin/activate uv pip install transformers torch==2.6.0 scanpy anndata tqdm scipy scikit-misc pysrab scipy ``` ### Optional: Flash Attention To handle larger gene contexts, EVA-RNA automatically runs on Flash Attention if available -- only available for post-Ampere GPUs (A100 and beyond). We recommend using the following wheel. ```bash uv pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp310-cp310-linux_x86_64.whl ``` ## Quick Start A full example is provided in a notebook (see "Use this model" -> "Google colab"). ```python import scanpy as sc from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("ScientaLab/eva-rna", trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained("ScientaLab/eva-rna", trust_remote_code=True) # Load example dataset (bulk dataset, GSE) import anndata as ad import pandas as pd counts_url = "https://www.ncbi.nlm.nih.gov/geo/download/?type=rnaseq_counts&acc=GSE193677&format=file&file=GSE193677_raw_counts_GRCh38.p13_NCBI.tsv.gz" counts_df = pd.read_csv(counts_url, sep="\t", index_col=0, compression="gzip").transpose() adata = ad.AnnData(X=counts_df) # Select 2,000 genes for efficiency (kbest recommended, otherwise highly variable genes) sc.pp.highly_variable_genes(adata, n_top_genes=2000, flavor="seurat_v3") # Preprocess counts (EVA expects a target_sum of 1e6 before gene selection and log1p normalization) sc.pp.normalize_total(adata, target_sum=1e6) sc.pp.log1p(adata, base=2) adata = adata[:, adata.var.highly_variable].copy() # Encode (gene symbols auto-converted, GPU used if available) embeddings = model.encode_anndata(tokenizer, adata) adata.obsm["X_eva"] = embeddings ``` ### Options `model.encode_anndata()` accepts the following parameters: - `gene_column` — column in `adata.var` with gene identifiers (default: uses `adata.var_names`) - `species` — `"human"` or `"mouse"` for gene ID conversion (default: auto-detected) - `batch_size` — samples per inference batch (default: 32) - `device` — `"cpu"`, `"cuda"`, etc. (default: CUDA if available) - `show_progress` — show a progress bar (default: True) ## Advanced: Raw Tensor API For users who need direct control over inputs (mixed precision is applied automatically): ```python import torch from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("ScientaLab/eva-rna", trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained("ScientaLab/eva-rna", trust_remote_code=True) device = "cuda" if torch.cuda.is_available() else "cpu" model = model.to(device).eval() # Gene IDs must be NCBI GeneIDs as strings gene_ids = ["7157", "675", "672"] # TP53, BRCA2, BRCA1 # Expression values should sum to 1e6 (before gene selection) and be log1p-normalized expression_values = [5.5, 3.2, 4.1] inputs = tokenizer(gene_ids, expression_values, padding=True, return_tensors="pt") inputs = {k: v.to(device) for k, v in inputs.items()} with torch.inference_mode(): outputs = model(**inputs) sample_embedding = outputs.cls_embedding # (1, 256) gene_embeddings = outputs.gene_embeddings # (1, 3, 256) ``` ### Batch Processing ```python batch_gene_ids = [ ["7157", "675", "672"], ["7157", "1956", "5290"], ] batch_expression = [ [5.5, 3.2, 4.1], [2.1, 6.3, 1.8], ] inputs = tokenizer(batch_gene_ids, batch_expression, padding=True, return_tensors="pt") inputs = {k: v.to(device) for k, v in inputs.items()} with torch.inference_mode(): outputs = model(**inputs) sample_embeddings = outputs.cls_embedding # (2, 256) ``` ## Expression Decoder EVA-RNA includes a pre-trained deterministic expression decoder that maps gene embeddings back to predicted expression values. ```python with torch.inference_mode(): # Encode output = model.encode(**inputs) # output.cls_embedding — sample-level embedding (batch, hidden_size) # output.gene_embeddings — per-gene embeddings (batch, n_genes, hidden_size) # Decode expression values predicted_expression = model.decode(output.gene_embeddings) # predicted_expression — (batch, n_genes) ``` ## GPU and Precision EVA-RNA automatically applies mixed precision for optimal performance: - **Ampere+ GPUs** (A100, H100, RTX 30/40 series): bfloat16 - **Older CUDA GPUs** (V100, RTX 20 series): float16 - **CPU**: full precision (float32) No manual `torch.autocast()` is needed. > **Note — Flash Attention constraints:** When flash attention is installed and an > Ampere+ GPU is detected, the model uses flash attention layers. These layers > **require CUDA and half-precision inputs**. If you move the model to CPU you will > get a clear error asking you to move it back to GPU. If you pass `autocast=False`, > autocast is re-enabled automatically with a warning since flash attention cannot > run in full precision. ### Disabling Automatic Mixed Precision For advanced use cases requiring manual precision control, pass `autocast=False`. This only takes effect when flash attention is **not** active (i.e., on older GPUs or when flash attention is not installed): ```python model = model.to("cuda").eval() with torch.inference_mode(): # Disable automatic mixed precision (ignored when flash attention is active) outputs = model(**inputs, autocast=False) # Or via sample_embedding embedding = model.sample_embedding( gene_ids=gene_ids, expression_values=values, autocast=False, ) ``` ## Converting Gene Symbols to NCBI Gene IDs The tokenizer vocabulary uses NCBI GeneIDs. A built-in gene mapper is included to convert gene symbols or Ensembl IDs: ```python tokenizer = AutoTokenizer.from_pretrained("ScientaLab/eva-rna", trust_remote_code=True) # Available mappings: # "symbol_to_ncbi" – human gene symbols → NCBI GeneIDs # "ensembl_to_ncbi" – human Ensembl IDs → NCBI GeneIDs # "symbol_to_ncbi_mouse" – mouse gene symbols → NCBI GeneIDs mapper = tokenizer.gene_mapper["symbol_to_ncbi"] gene_symbols = ["TP53", "BRCA2", "BRCA1"] gene_ids = [mapper[s] for s in gene_symbols] # gene_ids = ["7157", "675", "672"] expression_values = [5.5, 3.2, 4.1] inputs = tokenizer(gene_ids, expression_values, padding=True, return_tensors="pt") ``` ## Citation ```bibtex @article{eva-rna, title={EVA: Towards a universal model of the immune system}, author={Scienta Team}, journal={arXiv}, year={2026}, } ``` ## License [Scienta Lab EVA Model License](LICENSE)