---
type: source
title: "YaRN: Efficient Context Window Extension of Large Language Models"
author: "Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole"
url: https://arxiv.org/abs/2309.00071
date: 2023-08-31
domain: ai-alignment
intake_tier: research-task
rationale: "YaRN is Nous Research's context extension method adopted by Meta and DeepSeek. Demonstrates open-source research influencing frontier labs — evidence for knowledge diffusion patterns in AI development."
proposed_by: theseus
format: paper
status: unprocessed
tags: [nous-research, context-window, rotary-embeddings, yarn, meta, deepseek]
---

## YaRN: Efficient Context Window Extension of Large Language Models

arXiv:2309.00071 (August 2023, revised February 2026). First significant research publication from Nous Research.

### Problem

Transformer-based language models cannot generalize beyond their original training sequence length. This limits practical utility for tasks requiring long-context reasoning (document analysis, codebase understanding, multi-turn conversation).

### Methodology

YaRN (Yet another RoPE extensioN method) builds on Rotary Position Embeddings (RoPE). The key innovation is a compute-efficient interpolation method that extends context windows without requiring full retraining.

### Key Results

- **10x fewer tokens** required for context extension fine-tuning compared to previous methods
- **2.5x fewer training steps** than prior approaches
- Enables LLaMA models to handle 128K token contexts
- State-of-the-art performance in context window extension at time of publication
- Demonstrates ability to extrapolate beyond the fine-tuning dataset length

### Adoption

YaRN was adopted by:
- **Meta** — incorporated into Llama model family
- **DeepSeek** — used in their long-context model training

This adoption pattern is significant: a small open-source research lab (Nous Research, pre-funding) produced a technique that was adopted by two of the largest AI labs. This demonstrates that in AI research, the quality of the technique matters more than the institutional prestige of the lab — open-source research can directly influence frontier model development.

### Technical Details

The method modifies how RoPE embeddings handle positions beyond the training length. Rather than simple linear interpolation (which degrades quality) or full retraining (which is expensive), YaRN uses a frequency-based decomposition that preserves the geometric properties of RoPE while efficiently extending to longer sequences.

Code publicly available on GitHub. Licensed under CC BY 4.0.