Fun fact, I was hired to ship Tool Recommendation. The idea existed before I joined Arcade.dev, and so did a working alpha.
My interview process consisted largely of nerding out with Sam Partee about the recommendation systems he’d built. It was clearly going to be a challenge, but I’ve run things at scale, and we figured that between the two of us, we could probably get this thing out the door. At the time, I really thought we weren’t that far off.
Nine months later, here we are. It’s been a journey.
The premise was a game-changer
What sold me on joining Arcade was the recognition that there was so clearly a problem to be solved, and if we could be the ones to solve it first, we could tangibly impact the ecosystem.
People spend all this money on tokens because their agents need access to a full catalog of tools. Put 8,000 tools into an agent and it won’t be able to handle that full catalog, and what it does run will be expensive. That’s because there’s a ton of stuff for agents to look at that doesn’t matter. For example, if someone needed a hammer, you wouldn’t tell them to walk down every aisle in Home Depot and check every tool to see if it’s a hammer. So don’t do that for your agents.
Only give the agent the tools it needs when it’s actually doing the job. How hard could it be?
A smaller model does the fetching
One of our bigger breakthroughs was building Tool Recommendation to offload work from one LLM to another. Off the bat, you wouldn’t think that saves money. But it does once you change the kind of model. Instead of something generative, we use purpose-built embedding and ranking models that each do one job and cost far less to run than, for example, Opus.
So rather than swinging the hammer of linguistic reasoning and spitting out tokens until you land on the right thing, you fetch exactly what you need and hand it to the model doing the actual problem solving. You get a much better outcome on performance and on cost, and a lot of the excess goes away.
Give tools like this to a Kimi, for example, and you can make those cheaper models behave more like frontier models in this one instance.
A working model was only the start
A large part of what made Tool Recommendation so difficult to finalize wasn’t the solution. It was making that solution work in the real world, which in our case means for enterprise businesses.
The recommendation system itself has worked fantastically for a while. Integrating it with the Arcade actions runtime, so you get all the same built-in policy controls, was a separate job. And we’re a startup, so I was working on shipping a whole new service while more than a dozen engineers solved other problems at the same time.
Then customers wanted to run it on-prem, and the box around their environments isn’t always crystal clear. We’re committed to empowering customers to run on-prem so that was a must, but that deploy picture has been evolving over time. In addition to the basic requirements of identifying the resources required in each cloud and bringing forward the raw resources if the customer is managing their own raw hardware, you still had to answer funny questions like what happens if the region the customer is operating in runs out of GPU capacity? What then?
And then some customers buy their own hardware and don’t want to run GPUs all the time, because GPUs are expensive. Understandable. So how do you route embedding requests out of a locked-down environment that many would describe as air-gapped? Now you’re talking about punching a hole in the air gap. What does it take for a bank to trust us with those requests, and to know we’ll respect every privacy and non-training guarantee?
These were often the thornier problems that kept us up at night. Working through them one at a time until we were completely confident they worked in real production environments for our customers was what took the better part of this year.
In the weeds: the cost of version numbers and formatting
If I’m really going to dive into the details of building Tool Recommendation, two things come to mind: version numbers and formatting.
We run evaluations on Tool Recommendation every night by taking the raw model, running the search logic, then running the exact same queries against the same gateways and datasets in staging and production.
Early on, I decided to include version numbers on returned tools. I’ve been an engineer for a long time. That seemed reasonable.
Instead, it resulted in agents calling tools incorrectly because when a new tool version deployed, they would see both versions and schemas of the tool.
On the calling side we actually strip version numbers from the tool calls, so at call time, the agent could then call with the wrong schema (because it wasn’t clear which version to use) and run into schema validation errors (a confusing failure for a retrieval system that didn’t change the execution path).
We weren’t returning “incorrect” data, we could see recall was still high, but it was still causing the agent to fail.
In another case, we changed the formatting of how we passed tools to our model for re-ranking, flattening the data by dropping a newline. That alone cost us a few points of Recall@1 because our models had been trained on data that had the newline still in it. In theory you’ve got 100 points to play with, but if you’re really trying to make something impactful, you only have a couple. We’re close to 98% Recall@10, meaning the tools an agent needs show up in the top 10 results about 98% of the time. Going from 98% to 96% sounds like two points. It actually doubles your error window. Agents are resilient, so users might not notice. But every miss means another iteration, and you’re losing value on all the time, effort, and money that you put into fine-tuning a model for this.
So a lot of the work is figuring out the invariants the system has to hold. Little things like don’t add version numbers to tools, or don’t change how data is formatted. Neither is intuitive unless you’ve gone through it.
Why nobody else built it this way
Certainly others have been working on this problem, Microsoft Foundry’s Tool Search has been a leader in our space, Anthropic and OpenAI have their versions, but no one (that I’ve seen yet) has taken our approach because it’s so much easier to build with plain text search on your local machine.
Our way only makes sense if you’re thinking a few steps out and saying we can reuse a framework like this, so the value compounds. And the model makers get paid by the token, so they’re not incentivized to make it terrifically more efficient, especially when a token in ChatGPT still isn’t the same as a token in Claude.
Until you’re doing this at scale, the numbers look too small to be worthwhile, and agents are good enough to hide that anything is worse. It’s only when we started to think at enterprise scale, for hundreds of enterprises, that this became so obviously the right move.
The moments it felt real
Despite, or maybe because of, all the hard work there were a few moments that felt like celebrations.
The first was when Tool Recommendation ran in product staging. The second was when it ran on the portable deployment architecture we’re shipping to production and to single-tenant customers. That’s a brand-new deploy pipeline we don’t use for anything else yet, but it’ll end up managing everything.
It does feel like a real accomplishment to get this out into the world, and tools are only the start.
Right now, an agent looking for something in your Google Drive has to figure out how to search for it from clunky tools and base principles. Instead, we’d want the agent solving reasoning problems while a search engine does the retrieval. So our hope is that by proving this with tools, we can expand it to knowledge and context work, and then all your agents accelerate while costing less. How wild is that?
Try it now, let me know
Building Tool Recommendation has been my north star since I joined Arcade. We’ve tested and iterated until we’re confident it’s ready for enterprises to use in production, right now, and I genuinely think this will be a game-changer for anyone working with agent tools.
Try Tool Recommendation right now and let me know what you think.
