Installing a .deb requires root. Okay, a coding agent that wants root access. I know almost no one is good at supply chain security unless it is their core business, 2500 companies hacked this week for using LiteLLM, etc. So I look around for source code links and come up empty.
So, I am supposed to just give a random binary on a website that may or may not be malware depending on what NPM dependencies are in play, and give it root access, and trust whatever unknown CI/CD system that builds it to be able to execute any commands on my system with root privileges, and I am not even allowed to review the code?
That is an amount of trust I would not even extend to family, and you are asking for it from every single user.
So I asked a local LLM to decompile it for me, the 198 dependencies, and license compliance.
From here I could have it write an extensive test suite, delete the code, have LLM re-write the code to pass all the tests, then open source the result myself.
I am probably not actually going to do this, but anyone could. Proprietary software is no longer a moat, and for something like this it just makes your software very hard to trust. Clean room code laundering is so easy now you might as well open source it yourself.
This is a promising direction! Unfortunately, I think the benchmark result here is essentially meaningless.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
The breakdown with which model, per-task cost, and methodology is in the "Full results and methodology" link in the post, not omitted. Definitely check it out if you haven't.
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
I did read "Full results and methodology". It doesn't seem to show which agents you are routing to. Am I missing something? And how do those agents score on SWE-Bench Verified?
The benchmark wasn't evaluated with the router, we wanted an apples-to-apples comparison of our harness against other harnesses like mini-swe-agent on the same models to see if we added speed + cost value beyond routing.
Okay, that makes more sense, but then this post is very confusing. Why mention the router at all? Or does your live system have a router and the eval doesn't? In which case why not publish an eval with the router?
Fair point, yes, the router is only in the live system (we'll try to make that more clear). We thought that comparing the router to other harnesses on the benchmark wouldn't be a fair evaluation because the speed/cost gains from our harness would mostly be from using cheaper and simpler models rather than actually having a better harness.
The same thing holds for speed. I could build a system that speeds up Fable on GPQA Diamond ~50%, while improving score, by literally randomly selecting between Fable and Gemini 3.7 Flash. (Solve time for Flash is 0.1min and 0.8min for Fable, with Flash having a better score.)
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 Flash wouldn't preserve quality.
Mentioned in my comment above. Sorry, I thought you were talking about Gemini 3.6 Flash, not their new one. Yes, that's the reason why we didn't include the router in the benchmark to have a fair comparison of the harness itself.
Good to see more harnesses coming out. I think the initial set of "features" that made into harnesses like tool calling, multi-turn chat, MCP, skills and so on can all be optimized. And then much more can be done on top.
I am trying out a two-model approach where small model has access to tools, large model does not. Small model shapes prompts from the repo graph. And repo graph is the only tool that small model has when reading. The small model is already given a set of context from git log, codebase and Markdown/text files (generally design files) depending on the user's prompt.
I do not want to use use multi-turn chat. Small model would instead create fresh prompts for the larger model feeding context and reshaping the original ask every time.
Also, reference repositories can be added for small model to help ask right questions. Once a plan is made by large model, execution is mostly task-by-task, done by small model. Lots of deterministic code doing all this orchestration.
35% faster than swe-mini-agent, nice. You say this is due to somehow parallelizing operations?
I'm using a custom harness based on swe-mini-agent (actually its little brother, their tutorial [0]) and found it way faster than codex (for small tasks) despite being "just bash" in a while loop.
The main difference is that I do the opposite of what you said, i.e. I do dump the context in the prompt. You don't need to grep for what's right in front of you :)
But my repos are small (often smaller than Claude's system prompt!), and I have a script that dumps an "outline" (e.g. function headers and top level vars).
I had an even smaller harness for surgical edits but it was a bit too fiddly and I missed the "let it poke around and figure stuff out" mode of full agentic ones.
What I do miss from that old one though is that it could edit several files simultaneously, in one LLM call. Maybe someday I'll add that back :)
That being said, within a few months everyone who cares about speed will be on Cerebras etc., which will make even the slow harnesses way faster than mine and yours! (I've tested them already and it's insane how fast it is...)
Really cool that you built your own harness off of mini-swe-agent, we took a lot of inspiration from that. We're looking forward to trying out Sol Ultra Fast ourselves too! In our own experience, testing, building, searching, and other tool work still dominate a lot of time, but yes there could be a future where harnesses don't matter at all...but we expect Cerebras will also be super expensive :/
Thanks for bringing this up and trying it out! We've disabled Cmd+Option+I in the new version and also have a message during signup about sharing chats with Bullet, let us know if you run into any other issues!
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
We're actively working on implementing a guest mode feature right now for users that don't want to make an account. For reporting bugs, we have an in-app feedback form. Bugs are not automatically shared.
I use codex cli. It doens't have a UI for linux. I use opencode for that but it doens't work that well with the new 5.6 model suite from openai. Codex is both faster and better.
I would have loved if there was a way to use a GUI that worked like codex. Gave this a try. I think this might be it.
Please consider atleast adding MCP support if you can. That'll help
Hey just some feedback. I'm giving it a try on a new project I'm spinning up. It does seem faster although I haven't timed it against Claude or anything. I like the integrated browser to immediately see the changes (or see what the agent is doing). Thanks for having Linux support.
Thanks for trying it! If you want to time it against Claude/Codex in app, we have a race lab you can use for that. Good to hear that the Linux support and integrated browser help.
harnesses are becoming the new tech stack, I created my own and will compare it to how bullet operates, I’ve also built and deployed iOS and Android apps with my tool. When you have built something with the tool, other than the tool, it’s stronger proof.
Agree, building real apps with the harness is the strongest proof. Would love to hear where Bullet fell short vs your setup, especially on speed for your builds.
I have a way different taste, my coding agent doesn't run in a shell and proposes file changes first without changing them, so its prompt -> propose changes -> accept, bullet has the typical agent workflow, prompt -> create changes -> revert (if you don't like what was changed)
My coding agent leverages vector search and grep, just using grep takes longer and seems more expensive per task.
So sorry about that! We disabled download on iOS, but not Android (we'll fix that), it's not meant to be used on a phone (except for using our remote control feature).
Congrats on the launch.
I can't find which model providers are supported?
Are you calling the claude-code CLI directly and make bullet usable with anthropic subscriptions like orca or herdr?
I guess however this is a harness and it needs to connect to the API?
I think this adds no value. I would stick to OpenCode.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models.
* Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
Thanks for the feedback, we do focus on good tool management, context efficiency, etc. As for routing, ideally it isn't the user who has to route themselves, but a layer that routes for them.
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
Can someone explain how this actually can go faster than other harnesses?
1. Model routing. Do you regret giving a task to Fable when it could have literally been done by Sonnet?
This is the one thing that I can understand.
2. Targeted code + context search. We think embedding the whole repo is dumb. We also think sticking the whole context (or compressed context) in chat is dumb. So we do faster and better greps over both.
Is this just a different system prompt? Why would other harnesses not be able to do it this way given instructions to do so?
3. Aggressive context hygiene. Tool output is bounded, stale screenshots disappear, we don’t re-read files…the garbage never floods the model.
Existing harnesses already don't re-read files that have not changed. And content that is already in the context window is cached tokens and doesn't add significant latency, what am I missing? Removing context from history invalidates KV cache.
4. Efficient turns. Batch independent investigation, make one surgical edit, then perform one focused verification. Internal measurement showed 16% fewer round trips and 27% lower cost.
Isn't this just a specific sub-agent strategy, also addressable with prompting?
I'm genuinely confused about what is mechanically different in this harness that could not be accomplished with a prompt/skill in another harness.
(Not the author but) To your last question, I was able to replicate some of the benefits of my custom harness in Claude by just adding a custom MCP, startup hooks etc.
The issue I ran into is that I was fighting Claude's system prompt, which told it to do things which negated most of the benefits of my setup...
For context my startup hook was "grep def", lol. (Now working in js I have grep function, grep class, interface etc.). Gives it a little outline for context, saves time poking around blindly. My repos are small though, this is still a big blob if you have very large repos.
Yes, the CLI can run in Docker. The desktop app is not designed to run in Docker. And yes, you can use any HTTPS OpenAI-compatible endpoint, including a custom base URL and model ID.
Does anyone think a harness is something that can generate money? I’ve got my own harness. I use it in preference to any of the others. I even turned its tools into an MCP so Claude could use it. It vastly improved Claude’s iteration speed but I still prefer my own UI. If I didn’t work at whatever we’re calling a FAANG these days it’d be on GitHub - and nobody would use it because they’d just point their own AI at it and clone it. It’s just not that hard. Learned a bunch about rust. Got a better harness. Time well spent.
Right. But Dropbox is the service provider here: the owner of the hardware. The value of DropBox is that they've got a completel solution, including the harddrives. Is there a market for a paid Dropbox client, to use your analogy?
No, we don’t support ACP today. Bullet uses its own internal harness and CLI integration. That said, we can definitely see it being something we work on in the future!
Are you freaking kidding me with YC throwing money at something like this? I guess I can fund raise just by having built https://maki.sh, and months ahead of other founders too...
I think that if there are many players in the space, it's a problem worth working on! Like I said in the post, our main problem is speed and that's what we're trying to fix. It's been a problem, and it still is a problem. We have tried tools like OpenCode, Pi, etc. but for our speed issue, it did not fix it. If we go down dying on this boat, so be it, at least we went down dying on a problem we care about!
Meanwhile, Cerebras running Sol at 750 tokens per second, the harness "speed" becomes irrelevant compared to making better use of results from insta-Sol.
Yes, we saw this and it's really exciting, can’t wait to try it in Bullet. But faster inference only speeds up generation. In our own experience, testing, building, searching, and other tool work still dominate plenty of real tasks. We’ll have to test it in practice, but it’s a great development for everyone building agents.
People were saying that about YC in 2009, when Airbnb and Stripe were funded. YC’s philosophy has always been to invest in a great many companies, accepting that you can’t know in advance which will succeed, most will fail, but the ones that succeed will be successful enough to pay for all the losses.
“Spray and pray” is a deliberately belittling term for an approach that YC largely pioneered and has delivered outsized returns over two decades. There’s no “pray” about it. YC partners work incredibly hard to attract and select the most promising founders and support them through their most nascent and fragile stages. They keep turning out huge successes, whilst also giving off the appearance that most funded companies are unimpressive. That’s always been true; it’s the easy thing to focus on for people who are motivated to be critical, but it’s a valueless observation.
I recently switched over to using primarily Bullet for my projects and the speed of it makes it very nice to get projects off the ground quickly and work properly. I also found that Adi and Alex make updates very regularly based on some of the feedback i've submitted to their feedback tab. Good work on this product guys! I'm excited to see how it develops in the future.
Installing a .deb requires root. Okay, a coding agent that wants root access. I know almost no one is good at supply chain security unless it is their core business, 2500 companies hacked this week for using LiteLLM, etc. So I look around for source code links and come up empty.
So, I am supposed to just give a random binary on a website that may or may not be malware depending on what NPM dependencies are in play, and give it root access, and trust whatever unknown CI/CD system that builds it to be able to execute any commands on my system with root privileges, and I am not even allowed to review the code?
That is an amount of trust I would not even extend to family, and you are asking for it from every single user.
So I asked a local LLM to decompile it for me, the 198 dependencies, and license compliance.
From here I could have it write an extensive test suite, delete the code, have LLM re-write the code to pass all the tests, then open source the result myself.
I am probably not actually going to do this, but anyone could. Proprietary software is no longer a moat, and for something like this it just makes your software very hard to trust. Clean room code laundering is so easy now you might as well open source it yourself.
I recently discovered this same lesson the hard way. I was trying to get a multi-agent system I was building to improve upon GPQA Diamond scores (system here: http://pellmell.ai). No matter how hard I tried, I could not get any lift. When Fable 5 dropped, it also did not improve upon Opus, and I realized my mistake. The benchmark was saturated!
Now, looking at the result here, I see a similar pattern. Fable is not better than Opus, and the score is ~95%. Notably, this post omits which subagent is being used. Why? An intellectually honest way to tell if this thing really works would be to run that agent and report its score and cost as well.
Going back to my GPQA Diamond lesson, you can see here how a saturated leaderboard behaves https://artificialanalysis.ai/evaluations/gpqa-diamond. Fable gets 92.6% for $0.22 per task while several models score higher for $0.01. I could easily publish a router that “enhances Fable on GPQA Diamond” showing improved score for lower cost, just by implementing a router that picks the model at random!
On the saturation point, we agree that a 95.8% result on a mature benchmark isn't the main proof, which is why we're currently running against harder, less saturated benchmarks like Terminal-Bench, CursorBench, and SlopCodeBench (going to publish results on these hard benchmarks shortly). Apart from current user experiences, that will show the value of our harness.
Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!
What do you mean it wouldn't preserve quality? I demonstrate in the comments here how it would on a saturated benchmark.
I am trying out a two-model approach where small model has access to tools, large model does not. Small model shapes prompts from the repo graph. And repo graph is the only tool that small model has when reading. The small model is already given a set of context from git log, codebase and Markdown/text files (generally design files) depending on the user's prompt.
I do not want to use use multi-turn chat. Small model would instead create fresh prompts for the larger model feeding context and reshaping the original ask every time.
Also, reference repositories can be added for small model to help ask right questions. Once a plan is made by large model, execution is mostly task-by-task, done by small model. Lots of deterministic code doing all this orchestration.
I'm using a custom harness based on swe-mini-agent (actually its little brother, their tutorial [0]) and found it way faster than codex (for small tasks) despite being "just bash" in a while loop.
The main difference is that I do the opposite of what you said, i.e. I do dump the context in the prompt. You don't need to grep for what's right in front of you :)
But my repos are small (often smaller than Claude's system prompt!), and I have a script that dumps an "outline" (e.g. function headers and top level vars).
I had an even smaller harness for surgical edits but it was a bit too fiddly and I missed the "let it poke around and figure stuff out" mode of full agentic ones.
What I do miss from that old one though is that it could edit several files simultaneously, in one LLM call. Maybe someday I'll add that back :)
That being said, within a few months everyone who cares about speed will be on Cerebras etc., which will make even the slow harnesses way faster than mine and yours! (I've tested them already and it's insane how fast it is...)
[0] https://minimal-agent.com/
Not sure how you monetize a harness either way though?
Is there another pivot coming? This would make me nervous.
Cmd+Option+I > Console > 'allow pasting'
const onboarding = document.querySelector('#onboarding'); const app = document.querySelector('#app'); onboarding.style.setProperty('display', 'none', 'important'); app.inert = false; app.removeAttribute('aria-hidden'); document.querySelector('#prompt')?.focus();
Also, warning:
# Share chats with Bullet — helps us improve model routing and answer quality
Enabled by default.
Bulletproof.
OP was providing a userful tips to users, and they weren't giving you a bug report to remove it. If anything, they were suggesting you remove a login-wall.
But now that you blocked the inspector tool, you'll find it harder for users to report bugs. Unless of course those are automatically shared too? /s
Please consider atleast adding MCP support if you can. That'll help
"Codex in ChatGPT desktop app for Linux is now in preview" https://community.openai.com/t/codex-in-chatgpt-desktop-app-...
My coding agent leverages vector search and grep, just using grep takes longer and seems more expensive per task.
I guess however this is a harness and it needs to connect to the API?
Is it even hidden when your AI ends up tagging it with `aria-label="Hidden secret code"`? Lol. Fun mini game.
Things I would value: high-fidelity visualization (bonus points for Figma-like visual edits), good tool use (don't force me to tell the model), token efficiency, resource efficiency. Things that are not problems:
* Routing: OpenCode predefines subagents that you can set to appropriate models. * Search: there are AST and embedding-based search MCPs. I use https://github.com/DeusData/codebase-memory-mcp
Thanks for the link to the repo! We have looked at things like https://github.com/Graphify-Labs/graphify but haven't had a chance to rigorously evaluate their speed boosts on our harness.
1. Model routing. Do you regret giving a task to Fable when it could have literally been done by Sonnet?
This is the one thing that I can understand.
2. Targeted code + context search. We think embedding the whole repo is dumb. We also think sticking the whole context (or compressed context) in chat is dumb. So we do faster and better greps over both.
Is this just a different system prompt? Why would other harnesses not be able to do it this way given instructions to do so?
3. Aggressive context hygiene. Tool output is bounded, stale screenshots disappear, we don’t re-read files…the garbage never floods the model.
Existing harnesses already don't re-read files that have not changed. And content that is already in the context window is cached tokens and doesn't add significant latency, what am I missing? Removing context from history invalidates KV cache.
4. Efficient turns. Batch independent investigation, make one surgical edit, then perform one focused verification. Internal measurement showed 16% fewer round trips and 27% lower cost.
Isn't this just a specific sub-agent strategy, also addressable with prompting?
I'm genuinely confused about what is mechanically different in this harness that could not be accomplished with a prompt/skill in another harness.
The issue I ran into is that I was fighting Claude's system prompt, which told it to do things which negated most of the benefits of my setup...
For context my startup hook was "grep def", lol. (Now working in js I have grep function, grep class, interface etc.). Gives it a little outline for context, saves time poking around blindly. My repos are small though, this is still a big blob if you have very large repos.
That should be an option, not forced.
If you’re up for giving it another shot, would love to hear other feedback on the app you have.
"The bullets" are not any different than other new agent projects. New agent projects pop up and die constantly and it's not from a lack of funding.
Dear YC, please force the startups to dedicate some of the 500k funding for standard web design.
Looks like "Techno music design of the 90s" or similar :-D