Lessons from Hiring and Shipping LLM Features in Production

We’ve been adding LLM features to our product over the past year, some using retrieval, others fine-tuned or few-shot, and we’ve learned a lot the hard way. If your model takes 4–6 seconds to respond, the user experience takes a hit, so we had to get creative with caching and trimming tokens. We also ran into “prompt drift”, small changes in context or user phrasing led to very different outputs, so we started testing prompts more rigorously. Monitoring was tricky too; it’s easy to track tokens and latency, but much harder to measure if the outputs are actually good, so we built tools to rate samples manually. And most importantly, we learned that users don’t care how advanced your model is, they just want it to be helpful. In some cases, we even had to hide that it was AI at all to build trust.

For those also shipping LLM features: what’s something unexpected you had to change once real users got involved?