Moved from design to build on the support desk, with five parallel AI builders each in their own workspace, while other agents summarised years of closed tickets and we chose the AI model for live tickets with a blind comparison.
- The problem
- A large, interlocking feature set, a fixed pilot date, and tens of thousands of old support conversations to clean, summarise and import, without breaking the rules about where customer data may be processed.
- What I pushed back on
- I drew a hard line on data: historic tickets are processed only within our Claude subscription, never on our cloud provider's AI service, which is reserved for new tickets. I asked for summaries to run oldest to newest so they capture how the product changed, for a screen where I could judge summary quality myself, and for cheaper non-Anthropic models to be compared. When the weekly limit ran out, I capped usage.
- What Claude pushed back on
- Claude swapped a planned grading step that would have sent more history to the cloud service for a blind local grader, and admitted the test tickets had already gone there before my rule was stated. It explained that 'local in Claude Code' still means Anthropic processes the text, put the trade-off of a strict rule to me rather than deciding quietly, and fixed its own run, which kept starting batches after hitting the usage limit.
- Learning
- Parallel AI builds work when each stream has its own workspace, branch and database, an owner for each shared file, and a fixed merge order. A blind comparison on real tasks changed our model choice: a cheaper model invented less, at about a ninth of the cost. And a data-residency rule needs restating at every new step.