Most big companies go through the same sequence sooner or later. Senior management decides a new technology matters. Senior management wants to know whether employees are actually using it. Somebody builds a dashboard that counts how much each employee uses it. Employees notice the dashboard. Employees start using the technology a great deal.
Then comes the memo. Amazon has reached the memo. As Business Insider reported, Terk Terkowitz, Amazon’s vice president and distinguished engineer, published new guidelines for software engineers on Tuesday. The message: engineers will be judged by their impact on customers and the business, and not simply by how much AI they use. According to CNBC, the memo went around internally in September before Amazon posted it on its website. Its central warning is about something called “tokenmaxxing”:
Using these metrics could, if used as a target instead of a measure, drive negative outcomes.
Here’s how that works. AI models meter their work in tokens, which are chunks of text going in and coming out, and the bill is calculated per token. So “token consumption” measures how much AI you used, and it also measures how much of the company’s money you spent. If you rank people by token consumption, you are ranking them by how much of the company’s money they spent. Some of them will try to win.
Amazon ran this experiment. It tracked AI usage, token consumption and lines of code internally, and it used leaderboards to rank teams on some of those numbers. One of them, an informal tracker called KiroRank, ranked employees by their token use. In May, the Financial Times reported that some employees were running Amazon’s internal AI tools on trivial tasks to push their counts up. Amazon then told Business Insider that KiroRank had been “deprecated.” A spokesperson described it as a tracker built by a group of employees that “was never intended to promote the use of AI for usage’s sake,” and senior vice president Dave Treadwell told staff: “Please don’t use AI just for the sake of using AI.” The leaderboards caused friction inside the company, according to Business Insider. Some employees worried that the pressure to use AI would reward wasteful activity over productive work.
The new memo is Amazon’s clearest public admission that they had a point, and it puts the case in very Amazon terms. According to PYMNTS, Terkowitz wrote, “We are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters.” He warned that treating them as targets risked “increasing costs (tokenmaxxing is at odds with frugality), and increasing complexity (measuring LOC [lines of code] is at odds with invent and simplify).” Frugality and Invent and Simplify are two of Amazon’s leadership principles, so the charge against the token leaderboard is a sort of heresy: it got people to waste money and to write more code than they needed. If you pay engineers by the line, you get lines.
Sure. But don’t read this as Amazon going cold on AI. The memo says AI should be “at the center of software engineering practice” and argues that it is changing software development faster and more fundamentally than earlier shifts like cloud computing did. Amazon calls this the “era of the Renaissance Developer,” in which human judgment and machine intelligence together produce new products faster. “We all need to embrace this change and take the time to learn, or we risk being left behind,” Terkowitz wrote. Leaders are expected to use AI tools on production work themselves, learn where the tools help and where they don’t, and explain that judgment to their teams. He said Amazon’s most AI-forward “frontier teams” have shown stronger productivity and delivery, and that teams and individuals who aren’t adopting AI or raising their output metrics are falling behind.
So the metrics are deeply flawed, and teams whose metrics don’t go up are falling behind. Those two claims can both be true, and Terkowitz is trying to hold onto both. He said Amazon has found that token use, AI-tool adoption and code reviews can correlate with the outcomes it cares about, and that they need careful handling. More tokens can mean more cost. A focus on lines of code can mean needless complexity. A measure is fine as long as nobody is aiming at it.
His practical fix is to look at the tails. Managers are told to pay special attention to teams in the bottom 10% of AI adoption, to find out why they aren’t using it effectively, and to teams in the top 10%, to check whether all that use is producing faster delivery. I think the second half is the interesting part. Before, being in the top 10% was the goal. Now it gets you a manager asking whether you’ve shipped anything. That is an anti-tokenmaxxing audit, aimed at exactly the people the old leaderboard rewarded.
There’s also an escape hatch. Engineers whose work doesn’t yet benefit from today’s AI tools should document why and work toward a fix by sharing what they find with peers and managers. “If your work does not benefit from current AI tooling, that is a finding, not a failing,” the memo says, according to PYMNTS. And: “An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for.”
That’s a humane thing to write, and it also changes the incentives in a funny way. Under the leaderboard, the way to look good was to burn tokens. Under the memo, there are two ways to look good: ship things customers notice, or write a clear explanation of why the AI doesn’t help you yet. One of those can be gamed too, but at least it’s cheap, and it produces a document rather than a cloud bill. Cost is not a small point. Whether AI spending earns its keep outside the companies selling it is still an open question, and nobody’s return improves when engineers pay for tokens to climb a ranking. An Amazon spokesperson told CNBC that the company hasn’t capped employees’ token usage. It tracks token use to manage costs and doesn’t use it to evaluate performance, and the memo puts that distinction in writing.
Anyway, according to PYMNTS, the guidelines deliberately avoid setting a fixed definition of AI use in the job, because a definition written now could look outdated within a year. That’s sensible. The token leaderboard went from launch to “deprecated” in a few months.