AI Limits and Token Efficiency
I finally hit a limit working with these cloud AIs today. I have been hearing stories of many engineers running up against limits and some even risking massive bills when the switch to token-based billing came about.
I assume this hasn't happened to me in the past because I have really fought adopting these AI tools for my personal use and spend most of my work day only narrowly interacting with AI to answer questions, summarize documentation to speed up onboarding, and for silly image generation (team logos, #shit-posting fodder).
Where do you even AI then?
I have been working mostly with local models for my coding work and it has proven quite helpful. Today's personal session was playing with the preview of Fable 5 from Anthropic. The company even warns you the drawdown using this model will happen much faster, but the outputs are quite impressive.
I will probably just wait until my usage limit resets in a few hours rather than quintuple my monthly bill (even for this month). We will need to think about how our reliance on these tools really drive the business value of the existing business models.
I personally love that I hit my usage limit today. It pushes me back to writing things myself instead of waiting for an agent to pump out the code that I will strictly review. Someday (likely very soon) we will see a proliferation of frameworks and tools that can bring the power of these frontier models even closer to us.
Once I wrap up this session for myself, I might start looking back at running an Ollama server with some custom tools to see how far I can push my own lab for generative AI work.