Maybe Token Cost Does Matter?
September 28, 2026 In last week’s newsletter, I proudly announced the launch of our LoanBoss AI capabilities (seriously, check this out). This isn’t just a chat feature, this is unlimited possibilities. Run any analysis you can think of. Build every report you can think of. It’s your data, do whatever you want even if we hadn’t already built it for you.
Turns out, our customers love it…maybe a bit too much? We had more logins that day than the prior 67 work days combined.
At 11am, while we were all patting ourselves on the back, I got a message from our Chief AI Scientist, “While I was waiting on a PR review I went to see how much we have spent on the AI chat in Prod. We are at $3,912.”
In about two hours, our customers had just run up a $4k bill. On us.
Gulp.
But man, they built some cool stuff. The first five things built by various customers, all before 9am.
- 2027 budget (with all cashflows)
- Loan extension shortfall dashboard
- Rate cap audit
- IRR “what if”
- Lease approval dashboard
I love our customers, but absorbing $2.6mm in AI costs isn’t in the budget, so we dug in.
The first thing our AI guy did was build a router to help send user prompts through the appropriate model. A question like, “what loans have DSCRs below 1.25x?” doesn’t need Fable.
It turns out that bulk downloads are very very expensive - but that’s why we have download buttons! Downloads don’t need AI, so we put a limit on being able to do that.
Creating HTMLs were super expensive, and the highest models are noticeably better than the older models. But we found that if we added enough instructions to the older models, the output was the same. They just couldn’t get there on their own as easily as the newest models could.
We also learned that refreshing a skill isn’t nearly as expensive as building it the first time. We haven’t done it yet, but we are going to experiment with using a more powerful model for the initial build, then something like Sonnet for the refresh.
But you know what the biggest culprit was while we made those changes?
Internal usage.
On Day 4, we were $5,694 of the $7,112 AI charges.
- Our internal teammate tasked with being the Skill Tzar
- Head of Product
- Client Success rep
- Head of Client Success
- Me
As I noodled it, it made sense. Our customers have day jobs and the info in LoanBoss is just a tiny sliver of that, so they got what they needed and went back to work. But our internal teammates, who are already power users, were experimenting with everything possible.
Guess what that means? We’re about to assign model limits on a person by person basis. Skill Tzar? Give him what he needs. But someone just messing around? Sorry, no Fable for you.
I don’t know how that will play out for customers. Is our model router good enough to keep customers happy and costs down? Or do we start offering model access in our pricing? Or am I really just hanging on until you stand up your own AI hub and call LoanBoss data through the MCP Server? Time will tell.
But that’s why this matters to you. You will eventually stand up your own AI hub and set up a bunch of MCP connections with your partners. That’s where the magic happens…but so does the cost.
At that moment, the token expense passes from us to you. And all of your teammates, who are power users of your data/tools, will run up the bill. You’ll probably plan better than me and have all that in place before you launch, but have time set aside to turn dials for a while to create predictable costs.
I was happy with how quickly we were able to get costs down to a reasonable level, but that first day or two had me on edge.
On Friday, our costs came in at $606. That’s still $160k a year, but I think assigning model limits will further cut that in half.
With that quasi-emergency, maybe I should create a rates dashboard…5.16%?%?%?
