缩放自回归 Transformer 以进行单细胞生成

我们研究单细胞基因表达载体的自监督生成任务：给定一组来自​​某种细胞类型的载体，我们的目标是生成该细胞类型的其他基因表达载体。对于这项任务，我们描述了生成的基因表达向量的生物保真度和预训练损失的缩放行为。 The model is a causal Transformer paired with a learned quantized VAE tokenizer, trained with a cross-entropy loss.为了评估模型，我们将其以细胞类型的保留基因表达向量为条件并生成基因表达向量，将基因表达向量的结果分布与该细胞类型的 真值 分布进行比较。 We study the scaling properties of the proposed architecture by varying the number of trained parameters and the amount of training data. To our knowledge, we find the first jointly-fit two-exponent scaling law and compute-optimal frontier for a single-cell foundation model. Finally, we discuss how this pretrained model could be finetuned for perturbation response prediction.