DeepSeek drops open-source model that compresses text 10x through images, defying conventions

DeepSeek, the Chinese artificial intelligence research company that has repeatedly challenged assumptions about AI development costs, has released a new model that fundamentally reimagines how large language models process information—and the implications extend far beyond its modest branding as an optical character recognition tool.The company’s DeepSeek-OCR model, released Monday with full open-source code and weights, achieves what researchers describe as a paradigm inversion: compressing text through visual representation up to 10 times more efficiently than traditional text tokens. The finding challenges a core assumption in AI development and could pave the way for language models with dramatically expanded context windows, potentially reaching tens of millions of tokens.”We present DeepSeek-OCR as an initial investigation into the feasibility of compressing long contexts via optical 2D mapping,” the research team wrote in their technical paper. “Experiments show that when the number of text tokens is within 10 times that of vision tokens (i.e., a compression ratio < 10×), the model can achieve decoding (OCR) precision of 97%.”The implications have resonated across the AI research community. Andrej Karpathy, co-founder of OpenAI and former director of AI at Tesla, said in a post that the work raises fundamental questions about how AI systems should process information. “Maybe it makes more sense that all inputs to LLMs should only ever be images,” Karpathy wrote. “Even if you happen to have pure text input, maybe you’d prefer to render it and then feed that in.”How DeepSeek achieved 10x compression by treating text as imagesWhile DeepSeek marketed the release as an OCR model — a technology for converting images of text into digital characters — the research paper reveals more ambitious goals. The model demonstrates that visual representations can serve as a superior compression medium for textual information, inverting the conventional hierarchy where text tokens were considered more efficient than vision tokens.”Traditionally, vision LLM tokens almost seemed like an afterthought or ‘bolt on’ to the LLM paradigm,” wrote Jeffrey Emanuel, an AI researcher, in a detailed analysis of the paper. “And 10k words of English would take up far more space in a multimodal LLM when expressed as intelligible pixels than when expressed as tokens…But that gets inverted now from the ideas in this paper.”The model’s architecture consists of two primary components: DeepEncoder, a novel 380-million-parameter vision encoder, and a 3-billion-parameter mixture-of-experts language decoder with 570 million activated parameters. DeepEncoder combines Meta’s Segment Anything Model (SAM) for local visual perception with OpenAI’s CLIP model for global visual understanding, connected through a 16x compression module.To validate their compression claims, DeepSeek researchers tested the model on the Fox benchmark, a dataset of diverse document layouts. The results were striking: using just 100 vision tokens, the model achieved 97.3% accuracy on documents containing 700-800 text tokens — representing an effective compression ratio of 7.5x. Even at compression ratios approaching 20x, accuracy remained around 60%.The practical impact: Processing 200,000 pages per day on a single GPUThe efficiency gains translate directly to production capabilities. According to the company, a single Nvidia A100-40G GPU can process more than 200,000 pages per day using DeepSeek-OCR. Scaling to a cluster of 20 servers with eight GPUs each, throughput reaches 33 million pages daily — sufficient to rapidly construct training datasets for other AI models.On OmniDocBench, a comprehensive document parsing benchmark, DeepSeek-OCR outperformed GOT-OCR2.0 (which uses 256 tokens per page) while using only 100 vision tokens. More dramatically, it surpassed MinerU2.0 — which requires more than 6,000 tokens per page on average — while using fewer than 800 vision tokens.DeepSeek designed the model to support five distinct resolution modes, each optimized for different compression ratios and use cases. The “Tiny” mode operates at 512×512 resolution with just 64 vision tokens, while “Gundam” mode combines multiple resolutions dynamically for complex documents. “Gundam mode consists of n×640×640 tiles (local views) and a 1024×1024 global view,” the researchers wrote.Why this breakthrough could unlock 10 million token context windowsThe compression breakthrough has immediate implications for one of the most pressing challenges in AI development: expanding the context windows that determine how much information language models can actively consider. Current state-of-the-art models typically handle context windows measured in hundreds of thousands of tokens. DeepSeek’s approach suggests a path to windows ten times larger.”The potential of getting a frontier LLM with a 10 or 20 million token context window is pretty exciting,” Emanuel wrote. “You could basically cram all of a company’s key internal documents into a prompt preamble and cache this with OpenAI and then just add your specific query or prompt on top of that and not have to deal with search tools and still have it be fast and cost-effective.”The researchers explicitly frame their work in terms of context compression for language models. “Through DeepSeek-OCR, we demonstrate that vision-text compression can achieve significant token reduction (7-20×) for different historical context stages, offering a promising direction for addressing long-context challenges in large language models,” they wrote.The paper includes a speculative but intriguing diagram illustrating how the approach could implement memory decay mechanisms similar to human cognition. Older conversation rounds could be progressively downsampled to lower resolutions, consuming fewer tokens while maintaining key information — a form of computational forgetting that mirrors biological memory.How visual processing could eliminate the ‘ugly’ tokenizer problemBeyond compression, Karpathy highlighted how the approach challenges fundamental assumptions about how language models should process text. Traditional tokenizers—the systems that break text into units for processing—have long been criticized for their complexity and limitations.”I already ranted about how much I dislike the tokenizer,” Karpathy wrote. “Tokenizers are ugly, separate, not end-to-end stage. It ‘imports’ all the ugliness of Unicode, byte encodings, it inherits a lot of historical baggage, security/jailbreak risk (e.g. continuation bytes). It makes two characters that look identical to the eye look as two completely different tokens internally in the network.”Visual processing of text could eliminate these issues while enabling new capabilities. The approach naturally handles formatting information lost in pure text representations: bold text, colors, layout, embedded images. “Input can now be processed with bidirectional attention easily and as default, not autoregressive attention – a lot more powerful,” Karpathy noted.The implications resonate with human cognitive science. Emanuel drew a parallel to Hans Bethe, the renowned physicist who memorized vast amounts of reference data: “Having vast amounts of task-specific knowledge in your working memory is extremely useful. This seems like a very clever and additive approach to potentially expanding that memory bank by 10x or more.”The model’s training: 30 million PDF pages across 100 languagesThe model’s capabilities rest on an extensive training regimen using diverse data sources. DeepSeek collected 30 million PDF pages covering approximately 100 languages, with Chinese and English accounting for 25 million pages. The training data spans nine document types — academic papers, financial reports, textbooks, newspapers, handwritten notes, and others.Beyond document OCR, the training incorporated what the researchers call “OCR 2.0” data: 10 million synthetic charts, 5 million chemical formulas, and 1 million geometric figures. The model also received 20% general vision data for tasks like image captioning and object detection, plus 10% text-only data to maintain language capabilities.The training process employed pipeline parallelism across 160 Nvidia A100-40G GPUs (20 nodes with 8 GPUs each), with the vision encoder divided between two pipeline stages and the language model split across two others. “For multimodal data, the training speed is 70B tokens/day,” the researchers reported.Open source release accelerates research and raises competitive questionsTrue to DeepSeek’s pattern of open development, the company released the complete model weights, training code, and inference scripts on GitHub and Hugging Face. The GitHub repository gained over 4,000 stars within 24 hours of release, according to Dataconomy.The breakthrough raises questions about whether other AI labs have developed similar techniques but kept them proprietary. Emanuel speculated that Google’s Gemini models, which feature large context windows and strong OCR performance, might employ comparable approaches. “For all we know, Google could have already figured out something like this, which could explain why Gemini has such a huge context size and is so good and fast at OCR tasks,” Emanuel wrote.Google’s Gemini 2.5 Pro offers a 1-million-token context window, with plans to expand to 2 million, though the company has not publicly detailed the technical approaches enabling this capability. OpenAI’s GPT-5 supports 400,000 tokens, while Anthropic’s Claude 4.5 offers 200,000 tokens, with a 1-million-token window available in beta for eligible organizations.The unanswered question: Can AI reason over compressed visual tokens?While the compression results are impressive, researchers acknowledge important open questions. “It’s not clear how exactly this interacts with the other downstream cognitive functioning of an LLM,” Emanuel noted. “Can the model reason as intelligently over those compressed visual tokens as it can using regular text tokens? Does it make the model less articulate by forcing it into a more vision-oriented modality?”The DeepSeek paper focuses primarily on the compression-decompression capability, measured through OCR accuracy, rather than downstream reasoning performance. This leaves open whether language models could reason effectively over large contexts represented primarily as compressed visual tokens.The researchers acknowledge their work represents “an initial exploration into the boundaries of vision-text compression.” They note that “OCR alone is insufficient to fully validate true context optical compression” and plan future work including “digital-optical text interleaved pretraining, needle-in-a-haystack testing, and other evaluations.”DeepSeek has established a pattern of achieving competitive results with dramatically lower computational resources than Western AI labs. The company’s earlier DeepSeek-V3 model reportedly cost just $5.6 million to train—though this figure represents only the final training run and excludes R&D and infrastructure costs—compared to hundreds of millions for comparable models from OpenAI and Anthropic.Industry analysts have questioned the $5.6 million figure, with some estimates placing the company’s total infrastructure and operational costs closer to $1.3 billion, though still lower than American competitors’ spending.The bigger picture: Should language models process text as images?DeepSeek-OCR poses a fundamental question for AI development: should language models process text as text, or as images of text? The research demonstrates that, at least for compression purposes, visual representation offers significant advantages. Whether this translates to effective reasoning over vast contexts remains to be determined.”From another perspective, optical contexts compression still offers substantial room for research and improvement, representing a promising new direction,” the researchers concluded in their paper.For the AI industry, the work adds another dimension to the race for longer context windows — a competition that has intensified as language models are applied to increasingly complex tasks requiring vast amounts of information. The open-source release ensures the technique will be widely explored, tested, and potentially integrated into future AI systems.As Karpathy framed the deeper implication: “OCR is just one of many useful vision -> text tasks. And text -> text tasks can be made to be vision ->text tasks. Not vice versa.” In other words, the path forward for AI might not run through better tokenizers — it might bypass text tokens altogether.