A year ago AIO (Artificial Intelligence Optimization) was a niche concept. Now it is a pillar of any modern marketing engine, and part of my job at Arcade.dev has become making sure that when someone asks an LLM how to give an agent secure access to their tools, Arcade shows up, and shows up accurately described.
Since the beginning of the year, we have partnered with Zenith to go beyond what traditional SEO techniques can offer. With their expertise applied to our domain, our AIO metrics drastically improved in what is arguably one of the most (if not the most) crowded spaces fighting for agentic attention. Since we started tracking, Arcade’s citation share is up ~90%. What we have learned so far has materialized in an internal tool I use weekly to keep track of how agents find and perceive our brand.
Today we’re publishing it.
Here I cover how I think about the problem, why the tool is built the way it is, and what six months of running it taught us about content.
A note on data. I am going to show dummy prompts and measurements here. The diagrams use a made-up company in a space we do not compete in, a todo app called Nudgelist. So don’t read too much into the numbers. They are not from actual runs, which you can have for your own brand if you clone the repo.
1. The tool
What it is
aio-tool is a weekly job that runs a fixed set of prompts against a list of targets, where a target is a provider plus a model, and stores every response in one schema.
The providers today are:
- OpenAI Responses API with web search
- Anthropic Messages API with web search
- Anthropic Agent SDK
- OpenRouter’s online models
- Perplexity Sonar
- Exa search with an LLM synthesis step
- Codex
Each response lands as one JSONL row: the prompt, the search queries the model issued, the pages that came back, the final text, the citations, timing and cost.
The second stage is an LLM-as-judge pass over each row. Did it mention the brand? How accurately does it describe us, one to five against a ground truth blurb? Did it cite a domain we own? Where do we rank against the competitors we track? Then a week-over-week comparison, keyed on prompt, provider and model, produces the deltas.
The third stage is a small dashboard and a handful of deterministic “learnings” generators. Movers, scoreboards. They turn a pile of verdicts into a readout that is easy to act on.
How I run it
It runs on a cron, same day, same hour, every week. Consistency matters more than frequency here, and our prompt corpus contains enough redundancy within each category to soften statistical noise.
The prompts live in a CSV with a category column and a branded or unbranded flag. If you clone the repo, you will find a small sample set for the dummy company.
The cost is small enough that I stopped thinking about it. The judge stage is a few cents per run with a cheap model. Generation dominates the token spend, so even at low cost I implemented resumable runs. Providers rate limit, time out, and fail in creative ways, constantly. Losing a whole week to one provider dying near the end of the run got old fast. Every stage now appends as it goes and picks up where it left off.
Onboarding a new company is one JSON file: brand name, aliases, owned domains, a ground truth description, known competitors. Unknown competitors will show up in the data, and agents are good at discovering and backfilling share of voice if you need it.
Why it is not an agent
Everything else in the agentic space right now is a ReAct loop: a model looks at the state, decides what to do next, and does it, in a loop.
In aio-tool a run is a matrix, not a conversation. Prompts × targets is fully known before the run starts. The idea is to minimize the non-determinism to study LLM behavior in isolation.
Where I did use LLM judgment is scoring, and only scoring. The judge is a separate model from the generators, with a fixed rubric, and a regex pre-filter skips the full judge call when the brand is nowhere in the text or the citations.
This also makes it cheap to audit. Every verdict carries the excerpt that justified it. Early on the judge kept confusing us with a similarly named company in an adjacent space. We could see exactly which excerpts it was scoring, so we could see exactly why. By improving the judge and retaining historical data we can tell real citations apart from confused ones.
Does it mean we should avoid measuring agents? Of course not. It means studying the behavior at the model level is very important and complementary to agent behavior.
2. Content is still king
What we have learned in the last six months is to focus on content quality, and not solely on quantity. Yes, flooding the internet with content pieces will give you some immediate signal, but simple techniques such as double checking the claims coming from search results will drive down the brand ranking if all you publish is AI slop. Here I point out some important points to consider and review for the content you intend agents to consume.
Structure
Models cite pages they can quote. A page with a clear definition near the top, a comparison table, or an explicit “X vs Y” heading gets pulled in. Long marketing prose does not. Make the content understandable and to the point.
Vendor self-published content dominates the citation pool for unbranded queries. Most unbranded citations in our data point at some vendor’s own docs or blog or clearly sponsored pieces. Publishing is the lever. PR is not.
Where quantity pays off is per-feature and per-integration pages. If a buyer asks “how do I connect X to Y” and you have no page for that specific pair, you do not exist for that prompt.
Try this: pick your ten most important pages and check whether a model could quote each one in two sentences. Rewrite the top of the ones where it could not.
Domains
Our docs get cited heavily when someone already knows our name, and close to never otherwise. The content that explains us best is invisible to people who have not heard of us.
Some search backends never cite docs organically. In our runs, Perplexity and Exa almost never do. If those providers matter to you, the same content has to live somewhere else too.
Third-party domains like dev.to, GitHub and Medium carry weight and bring their own failure modes. A dev.to post that mentions you only as a URL in a code sample counts as a citation and nothing else.
Try this: for each provider you care about, list the domains it actually cites on your unbranded prompts. Make sure you have real content on at least two of them.
The target moves
Providers swap models, change search backends, and rewrite system prompts without announcement. A metric that dropped 30 percent in a week may say nothing about you.
Mischaracterization is a bigger risk than absence. Being described as the wrong kind of product hurts more than not being mentioned. We added the accuracy score for this reason.
Brand collisions make this worse. If another company shares your name in a nearby space, the model will blend you, confidently. The ground truth description in the judge config is how we identify and quantify it.
You cannot optimize for a snapshot. Only a series tells you whether a change was yours or theirs. Look for trends and dissect the data for actionable insights.
3. What we are looking at next
With all the progress we have made, there is a long way to go, and the next big step is harness-mode AIO.
We are working on ways to run the same prompts through the agentic harnesses people actually use, Claude Code, ChatGPT and Cursor, rather than the raw APIs. The harness brings its own system prompt, its own tools, and often the user’s own context. Closer to what a real developer sees, much harder to hold constant.
Bias is a problem we will have to deal with when measuring harnesses. A logged-in user’s history, their repos and their installed tools may all shape the answer and behavior of the search with the same prompt. Does the harness store memories? If so, it is another source of bias.
4. Run it yourself
Fork the repo, put your brand in one JSON file, run it once. Then run it again next week. The first run tells you where you stand. The second run is where the value is.
The repo is at github.com/arcadeai-labs/aio. Scheduled, multi-provider, auth-heavy workflows like this one are the kind of thing we build tooling for at Arcade. That is as much of a plug as you are getting.
