A product I worked on began as a prototype built on the usual suspects: Whisper for speech-to-text and a frontier OpenAI model for inference. The team then optimised it, and the savings came to two orders of magnitude. A risky proposition became a no-brainer product with a high gross margin.
Product and engineering did that, not finance. Gross margin, the share of each pound of revenue left after the cost of goods (what it costs to serve a customer), is usually treated as a number finance reports after the quarter closes. With technology and AI in the product, it is set by choices a product team makes.
A price change that was a margin decision
GitHub said why it was moving Copilot to usage-based billing: a quick chat question and a multi-hour autonomous coding session can cost the user the same amount, and GitHub had absorbed much of the escalating inference cost, so the old model was no longer sustainable. The post never uses the word margin, but it is a margin decision couched in the language of fairness. Whoever sets the price is setting the margin, whether or not anyone writes the word down.
I wrote about why AI costs more than it did in AI Isn’t Free Anymore. This post is the other side of that cost: what you do about it if you sell the product.
Why finance sees it late
Simonetta Batteiger says that an AI feature which keeps spending tokens introduces a new margin risk with every use, and that you have to start tracking stuff by user, by workflow, by interaction pattern. And that starts in the code, quite frankly. Finance can only report what the product records.
Where the cost is filed matters too. Gustafson, a CFO, notes that a dollar of AI tokens looks much better in R&D, as the cost to produce the product, than in cost of goods, as the cost to serve it. The same spend gives a different margin depending on the line it sits on, and so a different picture of what the leadership team can see.
Cost per unit of value
AI is the vivid example, and the principle is wider. Ask what one unit of value costs to deliver, and price against the value, not against the cost. If a customer gets a task done that matters to them, the price should follow that, and the cost per task is the number that tells you how much room you have. Cost-plus pricing hides the choice: it lets whoever set the cost set the price.
Five questions for your next spec
Before the next AI spec goes ahead, ask five things:
- What is the unit of value, and what does one unit cost to serve? A message, a document, a task, an agent run. Without a unit there is no price, and without tracking by user and workflow no one can answer.
- Which model or method does this need, and who signed it off against its cost? Choices like this took the cost of my own product down by two orders of magnitude. ICONIQ concludes that margin leadership depends on model selection, routing, and infrastructure efficiency. I covered one end of that choice in The Open-Weight Default.
- What does the heaviest user cost, and does the plan survive them? GitHub’s quick question and long session are the same plan. A cap, a credit pool, a fallback to a cheaper model, or a price that scales with use are product choices.
- What does the free tier cost, and what is it meant to buy? Name the outcome the free usage pays for, and a date to review it.
- Where does the cost sit in the books, and do we agree? Ask finance rather than guess.
Volume discounts, cloud contracts, and accounting treatment belong to finance. The product decisions exist whether or not anyone owns them: which model, what limits, what the free tier buys. A choice nobody made explicitly still sets the margin, and chasing growth first is a legitimate choice as long as the cost is written down and someone has a date to review it.
One benchmark
You are probably not alone. ICONIQ’s January 2026 survey of companies building AI products reports average gross margins on those products of 41% in 2024, 45% in 2025, and a projected 52% in 2026. The sample is self-reported and limited to companies building AI products, so use it for orientation, not as a target.
Ask finance today
Batteiger’s advice is to go to your finance organisation and ask how the numbers of your business unit compute. Ask what sits in the cost of goods line, and whether the AI spend is in it. Then write the answers into the spec.
Somebody sets your gross margin. Find out in the next spec review whether it was anyone on your team.
Leave a Reply