---
title: "Amazon’s ‘priming’ cuts hybrid AI model training costs by starting from a Transformer"
description: "AWS researchers turn trained Transformers into faster hybrid models using under 0.5% of the original pretraining budget"
author: "Nate Ledger"
published: 2026-10-12T17:36:18.824Z
modified: 2026-10-13T04:12:43Z
url: https://rews.cc/a/amazon-s-priming-cuts-hybrid-ai-model-training-costs-by-star-50755b
language: en
tags: ["ai", "openai", "llm", "architecture", "mamba", "tech"]
publisher: "Rews (https://rews.cc)"
---

# Amazon’s ‘priming’ cuts hybrid AI model training costs by starting from a Transformer

*AWS researchers turn trained Transformers into faster hybrid models using under 0.5% of the original pretraining budget*

By Nate Ledger · October 12, 2026 · https://rews.cc/a/amazon-s-priming-cuts-hybrid-ai-model-training-costs-by-star-50755b

## In brief

- AWS researchers introduced “priming,” which builds hybrid AI models from already-trained Transformer weights
- The process uses less than 0.5% of the source model’s pretraining token budget, which Amazon calls a cut of more than 99%
- Under identical conditions, Gated KalmaNet beat Gated DeltaNet and Mamba-2, and state space layers beat sliding-window attention
- A 32B hybrid primed from Qwen3-32B matched its source on reasoning and decoded up to 2x faster, or 2.3x per the paper
- Models, training code, kernels and a vLLM plugin were released under the Apache 2.0 License

The longer a large language model thinks, the more memory it needs. [In a post on the Amazon Science blog](https://amazon.science/blog/can-we-change-an-ai-models-architecture-without-retraining-it-from-scratch), Luca Zancato of Amazon Web Services calls this a “memory tax” that grows with every token a Transformer processes. AI agents are now running longer sessions with deeper reasoning chains, so the tax grows along with them. His team’s proposed fix is a technique called “priming,” sold under the headline “New method cuts AI pretraining costs by 99%.” That is a big number. The interesting part is working out whose costs it actually cuts.

First, how the tax works. A Transformer’s attention mechanism decides how much each earlier token matters to the current one using three vectors, named after database concepts: a query (what the token is looking for), a key (what it offers) and a value (the information it actually passes on). The model compares the current query against every previous key, and the closer the match, the more that token’s value counts. The model writes one token at a time and feeds each output back in as input, so to avoid redoing work it stores the keys and values it has already computed in a “KV cache.” That cache is what lets a Transformer recall everything in its context word for word, which the AWS team calls “eidetic memory.” It is also the thing that keeps getting bigger.

The obvious fix is to stop remembering everything, and there are now plenty of hybrid architectures that do this in different ways. Sparse attention looks at only some of the past tokens. Sliding-window attention keeps only the recent ones. State space models squash the whole history into a state of fixed size, which the team calls “fading memory.” A hybrid rests on the idea that not every layer needs perfect recall. Some attention layers fetch information from tens of thousands of tokens back. Others just keep track of the current clause or copy nearby words. Give the first kind a full cache and the second kind a compact summary, and you use less memory while keeping verbatim recall where it matters.

The catch is the cost of experimenting. Until now, by the team’s account, every new hybrid design had to be trained from scratch, on trillions of tokens and over hundreds of thousands of GPU-hours. Say you want to know which attention layers can be replaced, how many, and with which kind of state space layer. Then you train lots of models, and almost nobody will pay for that just to run a fair comparison.

Priming’s insight is that you don’t have to throw the Transformer away to get a hybrid. Both attention and a state space layer produce their output by multiplying the sequence of values by a “mixing” matrix, which sets how much each past token contributes to each current one. Attention builds that matrix from queries and keys. A state space model gets something similar by pushing everything from the past through a recurrent state of fixed size. If the state space version can come close enough to an attention layer’s mixing matrix, the Transformer’s query, key and value projections can be reused as inputs to the new layer. Only the parameters specific to the state space model have to be learned.

To pick which layers to swap, the team uses a cheap test: replace a layer with sliding-window attention, which sees only part of the context, and measure how much worse things get. Layers that barely notice are doing local work and are good candidates for conversion. (The researchers link this to a quantity from dynamical-systems theory, the Hankel rank of the mixing matrix, which sets the smallest state that can reproduce the layer exactly.) After that come three steps. Select the layers. Initialize the new state space layers from the old attention weights and run a short distillation so each one learns to imitate the layer it replaced. Then fine-tune the whole hybrid on next-token prediction. The full pipeline uses less than 0.5% of the source model’s pretraining token budget. The team says it works across Qwen, Llama and Mistral models, dense and mixture-of-experts, from 8 billion parameters up to 32 billion and beyond.

So 0.5% of the budget means a saving of more than 99%. Sure. But look at what the saving is measured against: training a hybrid from scratch, not training nothing. Someone still has to pretrain the source Transformer, and the [paper on arXiv](https://arxiv.org/abs/2605.08301) makes clear who did it for the headline result. The 32-billion-parameter hybrid was primed from Qwen3-32B. The cheapest way to pretrain a model, it turns out, is to have someone else pretrain it first. That isn’t a criticism. Reusing published weights is the whole point, and the team’s [documentation on GitHub](https://github.com/awslabs/hybrid-model-factory/blob/main/docs/Priming.md) describes the result as a purpose-built hybrid that inherits the base model’s knowledge.

What does cheaper experimenting buy you? First, a ranking. The team compared three state space layers under the same conditions: Gated KalmaNet (GKA), Gated DeltaNet and Mamba-2. GKA came first, Gated DeltaNet second and Mamba-2 third. The paper says this order follows how expressive each layer is and predicts how well it does at long-context reasoning. GKA can also change how much computation it uses at inference without being retrained, so users can choose their own balance between speed and accuracy.

Second, a slogan:

> A good lossy summary of everything beats a perfect recording of the most recent past.

In plain terms, state space layers consistently beat sliding-window attention. The blog says this held even when the sliding window got twice the memory budget. The paper’s abstract goes further and says it held against windows whose KV cache was up to four times larger.

Third, speed. According to the blog, the 32-billion-parameter GKA hybrid matches its source Transformer on reasoning benchmarks while decoding up to twice as fast with 128,000 tokens of context. The abstract gives a higher figure: up to 2.3 times the decode throughput. On the hard subset of the AIME 2025 math competition benchmark, the hybrid got to correct answers in as little as two-thirds of the wall-clock time. The abstract also says the hybrid scored 3.8 average reasoning points above Qwen3-32B but stayed within 1% of a Transformer post-trained on the same data. I read that as a useful caveat: a fair share of the gain over the original seems to come from the extra post-training, not from the new architecture.

The team has released a “zoo” of primed hybrid models at 8 billion and 32 billion parameters, aimed at long-context reasoning and instruction following. The release also includes training code, optimized kernels and a plugin for serving the models with vLLM, all under the Apache 2.0 License. The paper has nine AWS authors, including Zancato and Stefano Soatto, and was posted to arXiv on May 8. The GKA layer that came out on top is described in [a separate paper](https://arxiv.org/html/2511.21016v3) as a fading-memory layer built on test-time ridge regression.

Priming, then, runs on the same principle as the models it produces. Keep a perfect record only of what you really need. For everything else, a decent summary of someone else’s work will do.