Hello Product Hunt! 👋 I am Charles, founding father of Freu AI.
Some time again, we teased that we have been running on extending our browser automation tech to all of the working gadget. Nowadays, we’re formally launching Freu AI for Mac—an AI agent that automates any desktop instrument throughout your OS the usage of herbal language.
The Drawback: Imaginative and prescient Brokers are Too Dear & RPA is Too Brittle
We hit a large wall with present GUI automation. Conventional RPA (AppleScript, inflexible X/Y coordinate clickers) breaks the instant you resize a window or an app updates its UI. At the turn facet, trendy multimodal brokers (sending screenshots to cloud LLMs) scale extraordinarily for repetitive duties.
At the moment, maximum desktop brokers function like interpreters. Each time you ask it to “Extract information from this native PDF and input it into Excel,” it takes a screenshot, sends it to the cloud, causes concerning the visible structure, and clicks.
The Conventional Value: ~10k tokens (Symbol context) × 5 steps × 10 runs an afternoon = ~500k tokens/day simply to navigate the very same desktop UI, to not point out the insufferable latency.
The Answer: AOT Compilation + Semantic UI (SUI)
Freu AI adjustments this by means of introducing Forward-of-Time (AOT) compilation for OS-level duties. As a substitute of the agent examining the display screen from scratch each and every unmarried time, you display it the cross-app workflow as soon as.
Freu AI makes use of a cloud vision-based type to “bring together” that consultation right into a deterministic, reusable DSL.
The Freu Value: You pay the cloud “AI reasoning” token value as soon as when the agent watches and learns your workflow. However for long term runs? The agent merely invokes the pre-compiled DSL command in the neighborhood. This drops your ordinary execution prices to 0 and decreases latency from mins to seconds.
The way it works below the hood:
While you file a desktop workflow, our engine does not simply save a dumb macro. It makes use of Semantic UI (SUI) to grasp the display screen:
Understand: It acknowledges buttons, textual content fields, and icons throughout any app.
Unravel: It anchors to the semantic which means of the UI, now not inflexible coordinates. If Spotify strikes their “Play” button, Freu AI nonetheless reveals it.
Execute: It binds those visible anchor issues into our DSL and executes them deterministically.
🎁 The Open-Supply Bonus:
Whilst the Mac desktop app is our core product, we’re open-sourcing freu-cli as of late—our underlying DOM-based browser automation engine. You’ll drop it into your personal brokers to present them rapid “muscle reminiscence” for internet duties. Repo right here: https://github.com/freu-ai/freu-cli
🔮 What’s Subsequent: The Native Imaginative and prescient Execution Engine
We’re relentlessly upgrading our stack. Very quickly, we can release an ability to run the execution segment the usage of a light-weight, SUI-optimized imaginative and prescient type operating fully in the neighborhood for your {hardware}. Whilst we can all the time depend on tough cloud LLMs to grasp your complicated intent all the way through the preliminary “studying” segment, this upcoming native engine approach your daily repetitive executions will value precisely 0 API tokens and stay your real-time display screen information 100% non-public.
We’d love for you to take a look at Freu AI for Mac. I’d love to listen to your comments on our AOT method or how you are these days dealing with repetitive cross-app duties. My co-founders and I can be putting out within the feedback all day to reply to your questions! 🚀
