Overview

Day 2 of AI Engineer Shanghai 2026 was on November 6, again at the Hilton Shanghai Hongqiao and again on one main stage. The theme on the program was world models, physical AI and AI-native products. My notes from Day 1 are in the Day 1 post.

The morning was about what goes into a model and how to check what comes out. Christoph Lassner, a co-founder of World Labs, opened with a taxonomy of world models. HeyGen argued that HTML is better context than JSON for video agents. Chiling’s founder said the only moat left for vertical AI is expert knowledge you encode yourself, and HICAP said the last mile in a factory is data that no system exposes through an API. Two talks on checking followed: Daily.co on trusting voice-agent evals in a language the author can’t read, and NUS on what 3,000 live speech-to-text sessions did to their benchmark numbers.

After an investors’ panel, the afternoon moved from harnesses to robots. MiniMax explained what it stopped controlling in its agent harness as its models got stronger. The physical-AI block covered AiMoga’s factory robots, VAST’s persistent world model, three different ways around the sim-to-real gap from QuantSparkle, Noematrix and DYNA, and Futuring’s home robots. Liquid AI, masons.ai and 1Password spoke on small decision models, networks of agents that talk to each other, and checking an agent’s tool calls at run time. Three rapid-fire talks on robot data, team memory and the cost of reading the web ended the day.

The notes below follow the program in time order.

Taxonomy of World Models — Christoph Lassner (World Labs)

Christoph Lassner co-founded World Labs in early 2024 with Fei-Fei Li, Justin Johnson and Ben Mildenhall, and he opened the second day speaking in a personal capacity. He wanted to make “world model” usable again, now that biology, chemistry and business software use it as freely as robotics does. His hook was a meme from August that lined up an LLM, SLAM, 3D Gaussian splatting, video generation, Google Maps, JEPA and the sketch from Ha and Schmidhuber’s World Models paper, all claiming the label.

He started from reinforcement learning, where the modern use of the term comes from. A world model is a model of how a world behaves, and it answers “what happens next if I act?” It doesn’t have to be visual. In the usual partially observable loop the agent only sees part of the world’s state. A state estimator combines each observation with the previous belief and the last action into a new belief, and a policy picks the next action from it. The world model is the knowledge of how the world changes and how observations arise, and it feeds both the estimator and planning. On top of that he used the split from World Labs’ June essay: rendering produces observations, simulation produces a possible world state, and planning produces actions. A model can do any subset of the three, including none, and doing one doesn’t make it a world model.

Then he ran the meme through it, counting anything that qualifies in at least one implementation. Only a couple of hands went up when he asked whether an LLM can be a model of how a world behaves, and more for SLAM. He argued yes to both. An LLM can’t render, but it can simulate and plan. Google Maps ingests data, reconciles it into a belief and plans your trip. Every item passed, which was his point: by function alone, Google Maps sits in the same pot as SLAM.

So he added axes. Domain separates the spatial world, digital domains such as CRMs or user interfaces, and specialised ones like a model of the human cell. Scale and resolution separate a room from a planet. His example was livex.earth, a live globe that models everything from satellites down to weather. Coverage asks which variables the model keeps, including those not in view. He showed a clip he made with Kling 4.0, moving around a generated intersection where the light was red in every direction. His working definition was a predictive model of how a world behaves, with a set of functions, in domain X, at scale and resolution A and B, with coverage C. Descriptive models such as databases he set aside.

The last part was a thesis: world models will define how intelligence interfaces with the world. He sketched a System 2, a large multimodal model that holds long-term goals and reasons across domains, above System 1 modules for physical interaction (vision, touch, motion) and digital work (language, tools). Each module has its own world model, belief, estimator and policy. System 2 takes in what System 1 believes, including uncertainty and failed actions, and creates or changes the policies that run in real time. He left the coordination mechanism open and offered this as one possible organisation.

HTML is all your agents need: Redefining Context Engineering — Bin Liu (HeyGen)

Bin Liu, who introduced himself as Ben, runs HeyGen’s agentic video business. His claim was that the context-engineering decision that matters most is the representation the agent reads and writes, more than memory, tools or retrieval. Pick a format the models are already good at and every model release improves your product. Pick your own and you spend your time working against the model.

Bin Liu’s title card on the main stage. The Chinese title reads ‘HTML is the agent’s visual context’.

Bin Liu’s title card on the main stage. The Chinese title reads ‘HTML is the agent’s visual context’.

The example was HeyGen’s first video agent. It was built on the existing video editor, whose project file is JSON: at least 2,000 lines per video, sometimes 10,000. Models write valid JSON without trouble, but in his words they “cannot reason within your JSON”. He showed two four-line snippets that look equally fine as text. One renders correctly. In the other the words end up where nobody can see them, and the agent had no way to know until HeyGen rendered the frame and fed the image back. Internally they compared it to asking Shakespeare to write a Chinese poem from examples: it looks like a poem, but it isn’t a good one.

So they threw out the JSON. Agents now write plain HTML, CSS and JavaScript, and HyperFrames, HeyGen’s open-source framework released in April, renders it to video. His mental model was a landing page that keeps changing, with text flying in, a clip going full screen and sound coming on. According to the repository, a headless Chrome seeks each frame of the animation timeline and captures it, then FFmpeg encodes the result, so the same HTML always produces the same video.

‘Type a prompt. Get a video. Keep editing.’ The HeyGen editor. The sample script on the left describes the agent loop: Claude plans scenes, renders drafts, measures them and fixes what breaks, and a layout audit reports elements outside safe margins or overlapping other groups.

‘Type a prompt. Get a video. Keep editing.’ The HeyGen editor. The sample script on the left describes the agent loop: Claude plans scenes, renders drafts, measures them and fixes what breaks, and a layout audit reports elements outside safe margins or overlapping other groups.

The evidence was adoption. He cited about 80,000 GitHub stars and the second most-installed agent skill worldwide. The demo video, which took two and a half minutes to get playing, claimed 500,000 videos a day by the end of October; afterwards he said “I actually think” it’s 700,000 now. He said HyperFrames kept growing for weeks while the team shipped nothing, because the models got better. Twenty-two days after the release, Thariq from Anthropic posted “HTML is the new markdown”. Bin showed it as validation, adding that they copied the idea and “didn’t mention us at all”.

‘22 days later’: Thariq’s May 9, 2026 post ‘HTML is the new markdown’, quoting the Claude blog’s ‘The Unreasonable Effectiveness of HTML’.

‘22 days later’: Thariq’s May 9, 2026 post ‘HTML is the new markdown’, quoting the Claude blog’s ‘The Unreasonable Effectiveness of HTML’.

AGI is decoding. Encoding is the moat. — Ying Yang (CHILING CHUANYUE TECHNOLOGY)

Ying Yang has a PhD in brain science and worked on brain-computer interfaces before starting a beauty business, which she said reached ¥190 million in two years without outside money. She also has about ten million followers across her accounts. It was a founder’s strategy talk built around one question: which AI products will the general models swallow?

Her answer came from a framework she credited to her advisor: science encodes and engineering decodes. Botany encodes plants and perfumery decodes them. The central dogma encodes how genetic information flows and drug design decodes it. Neuroscience maps brain regions, and a BCI decodes their signals into commands. AGI is the most powerful decoder yet, and a decoder does best where the encoding is clearest, which she said is why math and programming fell first: they have no ambiguity. Vertical apps on that ground will be absorbed.

‘Science Encodes. Technology Decodes.’ Botany, cell biology and neuroscience on the encoding row; perfumery, pharmaceuticals and BCIs on the decoding row. Bottom line: every algorithm is a form of decoding, and AGI is an unprecedentedly powerful decoding system.

‘Science Encodes. Technology Decodes.’ Botany, cell biology and neuroscience on the encoding row; perfumery, pharmaceuticals and BCIs on the decoding row. Bottom line: every algorithm is a form of decoding, and AGI is an unprecedentedly powerful decoding system.

The opportunity, in a picture borrowed from Kissinger’s book Genesis, is the “island groups”: fields where some rules are known but are surrounded by an ocean of tacit knowledge nobody has digitized. Turning that knowledge into rules and data is “land reclamation”, and she argued the reclaimed land becomes a barrier AGI cannot absorb. Beauty is her island group. The slide put 2025 China beauty and personal-care GMV at RMB 1.1 trillion. The figure she leaned on, from running a brand, was transaction cost: a beauty company that spends about 50% of revenue on sales counts as a good one, and some spend 90% and have nothing left for R&D.

‘The Opportunity Lies in the Island Groups’: partly known encoding rules surrounded by tacit knowledge, chosen for high entry barriers and proven profitability. ‘Encoding is Land Reclamation: We reclaim land while AI decodes.’

‘The Opportunity Lies in the Island Groups’: partly known encoding rules surrounded by tacit knowledge, chosen for high entry barriers and proven profitability. ‘Encoding is Land Reclamation: We reclaim land while AI decodes.’

The plan goes from consumers to small brands, big brands and finally the supply chain. Consumers come first because a crowd of people asking skincare questions is what makes brands willing to share data. Her assistant answers from what she called tens of millions of product filing records and more than a thousand papers on skin physiology, formulation and biochemistry. Every question records a skin type, a preference or a product the user already owns.

She argued against photo-based skin analysis, which she called the product male engineers build when they don’t understand the customer. A selfie is one noisy sample that depends on lighting. Her slide claimed 75% accuracy for photo-based recommendations against more than 80% satisfaction with women’s own choices, and said most women over 25 know within six months what suits them. Her analogy was full self-driving with no driver-assistance stage first. Her app builds on that experience instead: you photograph a product on your shelf or in a shop and say how your skin reacted, and since every product’s ingredient list is known, your history becomes the signal.

Why most beauty AI fails at the entry point: one facial photo against a log of products used, usage order, skin response and reasons for switching. ‘We do not replace user choice, we make them better.’

Why most beauty AI fails at the entry point: one facial photo against a log of products used, usage order, skin response and reasons for switching. ‘We do not replace user choice, we make them better.’

Step two puts expert founders into the app. Her example was 冰寒 (Bing Han), a Tongji dermatology PhD who writes, formulates and sells his own line. His AI counterpart answers acne and rosacea questions from his books, papers and test data, then recommends his products. Big brands split that one person’s role across scientists, formulators and product lines, and the platform merges them into a single brand agent. In return for their data, brands get a dashboard of what consumers ask and ad tools. She said product development takes 24 months at L’Oréal and 22 at Shiseido, and that real demand data would shorten it and end the trend-chasing she described, everyone making hyaluronic acid, then collagen, then PDRN.

Step two: one expert runs R&D, formulation, content, diagnosis and sales; his AI counterpart, 冰寒问题肌专家, answers from his books and test data and recommends from his own brand.

Step two: one expert runs R&D, formulation, content, diagnosis and sales; his AI counterpart, 冰寒问题肌专家, answers from his books and test data and recommends from his own brand.

AI in Manufacturing: The Last Mile Is Half the Journey — Shihui Zhu (HICAP Group)

Shihui Zhu runs the software arm of HICAP, which makes bottle closures for most of China’s big liquor brands in six factories. He opened by calling himself an outsider who had come to an AI conference on his own. His title is an old proverb, 行百里者半九十: on a road of a hundred li, ninety is only halfway. His claim was that AI gets a manufacturer to a demo quickly, and that the last ten li have little to do with the model.

He brought numbers from their own rollout. HiWork, their internal AI workspace, sits on top of all their older systems and runs more than 50 agents. Per month it handles 850,000 business calls and about 116.7 billion tokens. They got it working on coding first, then pushed it to the six factories one requirement at a time.

HiWork, tried on themselves first: 116.7 billion tokens, 850,000 business calls and 50+ agents a month, proven on coding before being pushed to six factories one requirement at a time.

HiWork, tried on themselves first: 116.7 billion tokens, 850,000 business calls and 50+ agents a month, proven on coding before being pushed to six factories one requirement at a time.

Most of the talk was about why the model is not the bottleneck. He described adding AI to a company as renovating a building: a foundation of ERP, CRM and MES, then an “ontology” layer of objects, processes, metrics and rules, then the agents. The ontology is the hard part, and he said nobody sells a good product for it. In the factory the machines are old and their signals can’t be read, more than half of the frontline workers are over 40, and batch numbers are still written on paper cards. What actually happens on a given day (a supplier delivers less, a customer pulls a delivery forward, a machine breaks) ends up as a photo in a chat group, while the system only holds the plan.

“The hard part isn’t the model, it’s six foundations”: equipment networking, vision samples and system integration can be bought; one meaning per term, someone willing to sign, and the master’s experience can’t.

“The hard part isn’t the model, it’s six foundations”: equipment networking, vision samples and system integration can be bought; one meaning per term, someone willing to sign, and the master’s experience can’t.

The slide above summed it up. What money can’t buy is a single meaning for a word like “delivery”, which means different things to the factory, finance and marketing, a manager willing to sign off on what the system says, and the experience of the old master who knows when to change a tool. He added that when the semantics aren’t unified, a stronger model just gives a more confident wrong answer, so they record why each definition changed.

To pick projects they score every request on two axes, direct or indirect financial impact and fast or slow payoff. The early wins: expense processing saves 580 hours a month, contract review went from hours to minutes. Then he cited Goldratt’s The Goal: time saved at a non-bottleneck is a mirage. The real lever is production scheduling, getting more output from the same workers and machines, and it sits in the “think before moving” quadrant because due dates are often verbal promises, material lead times are unknown, and some moulds only fit some machines.

Project triage by impact (direct or indirect) and payoff speed: visual inspection, expense vouchers and contract review first, APS and digital twins later, display-only big screens never. Right: 44 projects in six categories, 14 of them marketing.

Project triage by impact (direct or indirect) and payoff speed: visual inspection, expense vouchers and contract review first, APS and digital twins later, display-only big screens never. Right: 44 projects in six categories, 14 of them marketing.

His conclusion was that the hard part isn’t letting AI schedule but writing down the rules every step depends on. His answer was to build the planning system first, step by step.

Workers now report shop-floor events by talking into a phone, and the speech becomes a structured record routed to an owner.

Trusting Evals I Can’t Read — James Hush (Daily.co)

James Hush works on Pipecat, the open-source voice-agent framework from Daily (on stage he said “I’m at Deepgram”, which announced it was buying Daily the week before). He has lived in Asia for six years, his wife is Taiwanese, and from a hundred-item bubble-tea menu he can order exactly one thing, 珍珠奶茶. So he built a voice agent to teach himself to order. The catch is that he can’t read Chinese, which means he can’t read half of his own tests.

The setup: a bubble-tea menu with about a hundred drinks, and the one he can order, 珍珠奶茶.

The setup: a bubble-tea menu with about a hundred drinks, and the one he can order, 珍珠奶茶.

He then generalised it: nobody in the room can read any more, in any language. One slide put a pull request to the Pipecat repo from a “claude-agent” account (214 commits, +52,417 and −18,903 lines across 1,342 files) next to a WeChat inbox with 999+ unread. His answer is to stop checking the work and start checking the checks, with LLMs checking other LLMs. He writes the checks in English, Claude or ChatGPT writes the code in Python, and he expects every application developer to work this way in 2027.

Why nobody can read: an agent-generated Pipecat PR with 214 commits and 1,342 changed files, next to a WeChat list with 999+ unread messages.

Why nobody can read: an agent-generated Pipecat PR with 214 commits and 1,342 changed files, next to a WeChat list with 999+ unread messages.

A voice agent is a pipeline (audio transport, speech-to-text, LLM, text-to-speech, transport back), and Pipecat is the vendor-neutral glue. Claude built version one from the Pipecat docs in about 30 minutes. Testing it by hand meant speaking a whole order after every change, so he wrapped it in an agent loop. A judge is an LLM that reads a finished conversation and answers one yes/no question, such as whether the agent was polite. An eval runs the agent 10, 20 or 100 times and reports a pass rate, because a voice agent never behaves the same way twice. Scripted evals are stand-up comedy, the same words every run. Simulations are improv: another LLM plays an undecided customer, in Mandarin he can’t read, judged against success criteria he wrote in English.

Scripted evals as stand-up (the same words every run) and simulations as improv (another LLM plays the customer): “Scripted evals check what you know. Simulations find what you missed.”

Scripted evals as stand-up (the same words every run) and simulations as improv (another LLM plays the customer): “Scripted evals check what you know. Simulations find what you missed.”

Text mode skips the audio and is nearly free with a local judge. Audio mode found a bug: the speech-to-text heard 去冰 (“no ice”) as “two 瓶”, “two bottles” in a mix of English and Chinese. A text-only test can’t see that.

An audio-mode simulation: the agent welcomes the customer, and the simulated customer 小美 asks what it just said and whether it can recommend a milk tea.

An audio-mode simulation: the agent welcomes the customer, and the simulated customer 小美 asks what it just said and whether it can recommend a milk tea.

The Good, the Bad and the Ugly: What 3,000 live sessions taught us about streaming speech-to-text — Satvik Sinha (AICET, NUS)

Satvik Sinha builds the voice agent in AICET’s Mandarin oral-practice app, where a tutor called Laoshi talks with students. He started by playing a student’s halting answer and showing the live transcript, which contained “Shanghai”. She never said it. About one in four of their practice sessions had text in the transcript that nobody said: foreign script, wrong characters, dropped words.

The part of the pipeline the talk covers: the speaker, a voice activity detector that cuts at detected pauses, and the STT model, “as in LiveKit, Pipecat and Vapi”.

The part of the pipeline the talk covers: the speaker, a voice activity detector that cuts at detected pauses, and the STT model, “as in LiveKit, Pipecat and Vapi”.

On FLEURS the four models they tried publish error rates between 2.4% and 7.2%. On their live audio the same models scored 25.0% (Qwen3-ASR), 28.5% (ElevenLabs Scribe), 27.6% (Gemini 3.5 Transcribe) and 48.5% (gpt-4o-transcribe), 4 to 20 times worse. He named two differences. FLEURS is native speakers reading prepared sentences, delivered whole. Their students are non-native, think aloud, and reach the model in pieces cut by a voice activity detector.

Benchmarks vs their findings: published FLEURS error rates (2.4 to 7.2%) against the same models on live pieces of student audio (25.0 to 48.5%).

Benchmarks vs their findings: published FLEURS error rates (2.4 to 7.2%) against the same models on live pieces of student audio (25.0 to 48.5%).

To separate the two, they fed the same students’ whole recordings to the same models. Accuracy rose sharply and the foreign script disappeared. Then they ran the opening clip through the live pipeline. The student paused a lot, so the VAD cut very short pieces, and a piece that short sounds plausibly like a word in several languages. The explanation is in the decoder. Its first output token is the language, and every later token is conditioned on it, so a wrong guess on a one-second fragment turns the whole line into Korean or Thai. More audio gives the decoder more to condition on.

Averaged over Scribe, Qwen and Gemini, accuracy goes from 62 on live pieces to 82 on the full recording. Telling the model the language lifts live pieces to 73, which recovers 11 of the 20 points. The other 9 are context, and most of that comes back once segments reach 2 to 8 seconds. His fix is to make the VAD wait for a longer pause before it cuts.

Accuracy against segment size: 62 for live pieces, 82 for the full recording, +11 from telling the model the language and +9 from context, most of it reached with 2 to 8 s segments (shaded).

Accuracy against segment size: 62 for live pieces, 82 for the full recording, +11 from telling the model the language and +9 from context, most of it reached with 2 to 8 s segments (shaded).

AI startups: products, markets & what comes next — Yusen Dai (ZhenFund), Harry Wang (Linear Capital), Patrick Liu (TaoZ Capital)

The first session after lunch was a 40-minute investors’ panel. Bibi, who runs the 第二次用火 podcast, moderated Yusen Dai of ZhenFund (first cheques into Kimi, Manus and Xiaohongshu), Harry Wang of Linear Capital (hardware-plus-software deals such as Horizon Robotics) and Patrick Liu, who left BlueRun Ventures late last year to back Peking University teachers and students. There were no slides. They disagreed on two questions.

The panel on the main stage, left to right: Yusen Dai, Harry Wang, Patrick Liu and the moderator Bibi, with the guest cards for Wang, Liu and Bibi on the screen.

The panel on the main stage, left to right: Yusen Dai, Harry Wang, Patrick Liu and the moderator Bibi, with the guest cards for Wang, Liu and Bibi on the screen.

The first question was whether model companies are eating the application layer. Dai put the pressure on the model companies’ own apps, Claude Code, Codex and ChatGPT, which in his words “subsidise tokens 20x”. He still thinks cheaper models help app builders: inference gets “maybe 10x” more efficient each year, and someone has to close the gap between what a model can do and what people will pay for. His analogy was Windows, where IE and Office were first-party and a third-party ecosystem grew anyway. Patrick was blunter. In China, he said, 90% or even 99% of the app-layer startups that raised money three years ago are dead, and the winners were doing arbitrage on the newest model, which ends once the model reaches into the application itself. Harry said pure-software apps keep losing their boundaries (“China never had boundaries”), so Linear invests in hardware plus software.

They split on personal agents. Harry was the enthusiast: an “IQ 200” assistant in his pocket that reads his tone and pace of speech and acts instead of chatting, with one butler agent keeping context per task, as in the Super Agent design DeepSeek had just released. Dai thinks efficiency is not a mass-market need. Clearing out old subscriptions is nice once, retention of the current products is “average”, and the biggest mobile-internet companies grew by expanding what people wanted rather than saving them time. An agent that books your flights only moves Ctrip’s commission into a different pocket. Patrick said the opportunity is real but belongs to platform companies, because a startup can’t match a model lab on compute, algorithms or data. From his years running a games company he added that products that consume users’ time beat products that save it.

Harry Wang (second from left, holding the microphone) while describing real-time video generation on diffusion models, ‘more than 20x’ faster, as a vertical that stays away from the model labs’ main battlefield.

Harry Wang (second from left, holding the microphone) while describing real-time video generation on diffusion models, ‘more than 20x’ faster, as a vertical that stays away from the model labs’ main battlefield.

Harry said labs train long-horizon behaviour mainly in coding, that trajectory data and environments for finance or travel are “far from enough”, and that a board member at a Silicon Valley lab told him the models look “dumb” at everyday tasks because of training priority, not capability.

The second disagreement was about founders. Dai said ZhenFund passed on the expensive young-professor startups its peers funded, because good research and a good company are different things and China rarely offers an exit for a research project. Yang Zhilin of Kimi was his exception. Patrick, saying outright that he differs from Dai here, likes young teachers paired with strong undergraduates, because this wave is driven by technology. Harry said most of Linear’s 2026 deals were in physical AI. He had also seen many recursive self-improvement (RSI) startups at starting valuations “usually above US$100 million” and funded none, because the founders couldn’t say what the RSI was for.

Harry’s case for local AI rested on privacy and context. His enterprise box runs at least a “Flash-level” DeepSeek model so a company’s context stays in the building, with the cloud for harder work.

The closing round asked what each of them would do with RMB 2 million in angel money at graduation. Dai, who took that sum from Xu Xiaoping and built Jumei, said to join the biggest wave with partners you trust and expect to pivot. Patrick would pick a very hard problem and give it ten years.

Stronger Models, Leaner Harnesses: What to Control and What to Let Go — Ronny He (MiniMax)

Ronny He leads R&D on MiniMax Code, MiniMax’s coding and work agent. He said he had rewritten the deck the night before. It had sixteen dense slides of rules and measurements from their own harness, talked through in under twenty minutes with the case studies rushed “because of time”.

His claim fits on one slide: let go of judgment, keep control of invariants. Judgment is anything the model now decides better than your if-statements: which tool, what plan, how to split the work. Invariants are what the model can’t see, can’t undo or can’t be trusted to remember: the prompt-cache prefix, the clock, process lifetimes, state that has to survive. Every harness feature gets four questions. Can you name the failure it prevents? Can the model see what it would be deciding on? If it gets it wrong, can you undo it? Did the feature survive an eval? His line for the third one was that a rule in the prompt is a hint, not a gate.

Slide 05, the principle: let go of judgment (which tool, what plan, how to split the work, what to summarise), keep control of invariants (the cache prefix, the clock, process lifetimes, state that must survive).

Slide 05, the principle: let go of judgment (which tool, what plan, how to split the work, what to summarise), keep control of invariants (the cache prefix, the clock, process lifetimes, state that must survive).

The case for deleting things came from Anthropic. He said Claude Code’s system prompt was cut by more than 80% for newer models, that a single agent with skills often beats a subagent design, and that dropping Plan mode has been floated. When he asked who had used Claude Code that day, he counted more than 80% of the room. MiniMax expects its own /goal command to go next, since the model is now trained to carry a task through to delivery. The numbers came from MiniMax’s own cases, three of five about the prompt cache.

Connector tools (third-party MCP servers the user signs into) used to be injected into the tools array on every turn. The connected list can change between turns, and a network blip returned an empty list, so the cached prefix kept getting invalidated and sometimes the model had no tools at all. They moved it all behind one small CLI the model calls, so the tools array never changes. The cost is an extra model step and a dependence on good descriptions. The second case was programmatic tool calling: a run_code tool in which JavaScript calls tools.* through the same permissions and hooks. Offered next to every native tool, run_code was barely used and tokens went up 5%. It only paid off once most tools could be reached only through code. Then scores stayed flat, tokens fell 13% and 35% on two task sets, and requests went from 104 to 93. His slide called it a gain in efficiency, not in capability.

Slide 08, cut, defer, program: load fewer tools (past 30–50 definitions models pick worse), move rare ones behind search, then a run_code tool. The ‘where we got burned’ boxes: catalogs fetched every turn slowed first token, and run_code beside native tools went unused while tokens rose 5%.

Slide 08, cut, defer, program: load fewer tools (past 30–50 definitions models pick worse), move rare ones behind search, then a run_code tool. The ‘where we got burned’ boxes: catalogs fetched every turn slowed first token, and run_code beside native tools went unused while tokens rose 5%.

The third case was the most concrete. Anthropic’s default prompt-cache lifetime is five minutes, and an agent waiting on npm run dev sends no requests, so a six-minute wait in the foreground loses the cache and the next turn pays full prefill. The model can’t see that clock, so the harness owns time. Every command is born as a background task, the foreground waits at most 60 seconds, the task keeps its ID so the model won’t kill it and start it again, and a 3600-second deadline counts foreground time and can’t be reset by backgrounding. The slide admitted the price: notifications, anti-polling, cleanup and “truth after compaction” were all rebuilt by hand.

Slide 09, ‘Should the agent just wait for npm run dev?’: a 5-minute default prompt-cache lifetime, and MiniMax Code’s auto-backgrounding (born in background, 60 s foreground, same task_id, 3600 s with no reset).

Slide 09, ‘Should the agent just wait for npm run dev?’: a 5-minute default prompt-cache lifetime, and MiniMax Code’s auto-backgrounding (born in background, 60 s foreground, same task_id, 3600 s with no reset).

One slide was about memory. A long memory file was truncated, cutting the most important rules at the top, and nobody noticed. Agents also told users “context is almost full, I’ll continue next turn” because they didn’t know compaction existed.

Beyond Vision-Language-Action: First Principles for Building Physical AI — 何文智 He Wenzhi (AiMoga Robotics)

He Wenzhi is product director of the industrial embodied-robot unit at AiMoga (墨甲), Chery’s robotics company, and his talk opened the afternoon’s physical-AI block. He spoke as the person who has to make robots work in customers’ factories, not as a researcher.

Title slide: 超越视觉—语言—动作模型 (beyond vision-language-action models), with the line ‘intelligence is not only choosing an action, but sensing its result and correcting the next one’. The image on the right is AiMoga’s wheeled industrial robot.

Title slide: 超越视觉—语言—动作模型 (beyond vision-language-action models), with the line ‘intelligence is not only choosing an action, but sensing its result and correcting the next one’. The image on the right is AiMoga’s wheeled industrial robot.

In his account, both kinds of model fail on a production line. A VLA (vision-language-action) model doesn’t reach the precision and stability that a line’s takt time demands, which he called fatal. Classic vision models such as YOLO can be made precise, but every factory, and every line inside it, has many product types, shapes and colours. AiMoga has to train, deploy and tune for each customer and keep its own people stationed on site, and he said that cost is not sustainable for a robot company. The wheeled industrial robot on his title slide goes live in a partner’s factory in Suzhou next month.

The core claim was one slide: understanding an instruction is not mastering the physical interaction. A VLA can recognise the part and understand “insert the connector”, but from camera images alone it can’t observe contact force, slip, friction or alignment error. He was specific about where it hurts: whether a gripper or dexterous hand really has a firm hold, and whether the part is slowly sliding. The slide’s formula was “where to move?” ≠ “is the contact right?”. Going beyond VLA, he said, means a richer closed loop, not abandoning VLA.

‘Understanding instructions is not mastering physical interaction’: VLA brings semantic understanding and action priors, but from external images alone contact force, slip, friction and alignment error stay invisible. Red line: ‘where to move?’ ≠ ‘is the contact right?’

‘Understanding instructions is not mastering physical interaction’: VLA brings semantic understanding and action priors, but from external images alone contact force, slip, friction and alignment error stay invisible. Red line: ‘where to move?’ ≠ ‘is the contact right?’

The mechanism is a sensorimotor loop: perceive, estimate the state and its uncertainty, act with fast local feedback, then detect deviation, slip or failure, correct, and keep the experience. Each sensor has a job. Cameras give 6D pose and depth, torque sensors in the joints report collisions, load and applied force, tactile sensors on the hand report friction, slip and deformation, and proprioception gives joint and chassis state. On top sits a five-layer stack that he compared to L1–L5 in self-driving, from motion control (state estimation, servo, impedance, safety) through contact-aware skills, skill memory and VLM planning up to goals. Memory, evaluation and learning cut across the layers, and a learned change only settles into the model after it has been verified. His example task was refilling a workstation and installing a part, dispatched from the customer’s MES. If loading runs into a bad fit, an unexpected force or a slip, the robot stops, retracts and re-estimates. In his words, a task counts as done only when its physical result is verified.

‘Five functional layers, three kinds of model, one loop’: L1 physics and motion control up to L5 goals and meta-control, with memory, evaluation and learning as cross-layer mechanisms (learning: local improvement, settled only after verification).

‘Five functional layers, three kinds of model, one loop’: L1 physics and motion control up to L5 goals and meta-control, with memory, evaluation and learning as cross-layer mechanisms (learning: local improvement, settled only after verification).

He said the loop is not yet fully applied in deployment.

Industrial safety supervision (motion limits, collision protection, stopping, fault handling) must not depend on the model. It runs on a PLC or a dedicated board with hand-written rules.

He also named fusing LiDAR, depth, force/torque, ultrasonic and RGB data as an open problem for them.

From 3D Generation to World Models: Building Persistent Interactive Worlds for Physical AI — Gavin Luo (VAST / Tripo AI)

Gavin Luo is head of strategy at VAST, the company behind the Tripo 3D-generation models. He built it around a question his own company had to face this year: what happens to a 3D company when an AI agent starts building 3D scenes by itself?

Gavin Luo on the main stage with his title slide, ‘From 3D Generation to World Models: Building persistent interactive worlds’.

Gavin Luo on the main stage with his title slide, ‘From 3D Generation to World Models: Building persistent interactive worlds’.

Two days after OpenAI released GPT-6 Astra, the team started a group chat called “Did Astra solve 3D today?”, with one rule: post only what you tried yourself, nothing from Twitter, YouTube or Xiaohongshu. Agents built indoor scenes entirely in code in tens of seconds, without any AI-native 3D model. One researcher said he hadn’t found an indoor scene that had to use Tripo, and a colleague replied, “Honestly, it was a bit of a panic.” Most of a room can be specified. VAST’s conclusion: if you can describe it, an agent can write it; if you can only recognise it, a model has to learn it. Code covers the doors and the score, and the model covers a character’s face, a creature’s shape or a likeness from a photo. His example was Sac’s Little Prince game from the Day 1 breakout, built in 16 days with Claude, Blender and Tripo, the agent writing the game and Tripo making the 3D assets. Tripo becomes a tool that agents call.

‘3D is the native language of the world’: VAST ships 3D-native models through Tripo Studio and the Tripo API. Tens of millions of creators in nearly 200 countries and regions, #1 in 3D models generated in H1 2026, #1 on a blind-vote 3D leaderboard with 200K+ voters, 60+ papers and 50+ open-source projects.

‘3D is the native language of the world’: VAST ships 3D-native models through Tripo Studio and the Tripo API. Tens of millions of creators in nearly 200 countries and regions, #1 in 3D models generated in H1 2026, #1 on a blind-vote 3D leaderboard with 200K+ voters, 60+ papers and 50+ open-source projects.

The second half was Project Eden, the next step on a staircase that CEO Simon showed at SIGGRAPH 2023: assets, then scenes, then worlds that keep running for people and robots. His line for it was that a 3D asset is a noun and a world needs verbs. The design keeps the world state explicit in an ordinary program. The game knows where the door is, whether it is open and what happens when you walk through it, and it renders a grey-box view of that. EdenLive takes the grey-box view as its target first frame, plus a text prompt or short text for important events, and generates the frames you actually see. In the demo a small window showed a submarine game’s own output and the main screen showed EdenLive’s version. Because every view comes from one state, a fire you put out stays out, a mark on a wall is still there for the next visitor, and two players can look at the same place from different cameras. The contract between game and model is just reference footage, so a game doesn’t need its own integration. He said the demo runs a 5B model at 720p and 24 frames per second, and that weights and streaming inference code are open source under Apache 2.0. The repository adds that this is on a single RTX 5090 and that 14B weights are out too.

He listed what is missing. A character that walks back into view can come back with drifted details, users need more control, and learning state transitions is “very important” and unsolved. His favourite failure was an evaluation: one of their agents passed a walkability check by sticking a box in place, and the check concluded you could walk up the stairs.

For robots, they take a real video of a person handling an object and swap the hand for a robot arm while keeping the motion and the scene, to get visual variations of one demonstration. He said they still need to test physical accuracy and usefulness for robot learning, and that the real goal, the third step, is environments where robots act, see the consequences and try again.

How We’re Building Open Decision Models for On-Device Agents — Leonie Monigatti (Liquid AI)

Leonie Monigatti works on the post-training team at Liquid AI. She threw out her planned talk a week before the conference, because in mid-September TypeSafe AI released Jev and started what everyone now calls decision models (TypeSafe’s name is “System One models”). Her claim was that the open ecosystem is reverse-engineering Jev with a simpler recipe than TypeSafe’s launch blog suggests, and getting competitive results. Liquid has shipped several of these in six or seven weeks, hosted and open. She said its d1-ultra was leading the Decision Index when she checked that morning.

A decision model takes a state and a bounded question and returns a typed answer with a confidence score, in one forward pass and without generating text. Her example asked whether changing your talk a week before, to a topic that changes every day, was a good idea. Liquid has three question types, which it calls primitives: Noul (yes/no), Score (1 to 5 here, and it said 1) and Choice. The confidence is the product. The slide after the demo was four lines of code: above 0.9, execute; otherwise ask a human to confirm. “Decision models can be wrong, but they will tell you when they’re unsure.”

The Choice primitive: a state and a question with three options go in, and a typed answer comes out with per-option probabilities (0.90, 0.03, 0.07) and a separate confidence field of 0.84.

The Choice primitive: a state and a question with three options go in, and a typed answer comes out with per-option probabilities (0.90, 0.03, 0.07) and a separate confidence field of 0.84.

TypeSafe spent two years in stealth, and a copy appeared less than a week after release. The baseline is what she called an inference-only technique that works with any LLM. You put the state, the question and the options in the prompt, tag each option with a one-token label (A, B, C), and ask for the label only. Then you run one forward pass, read the logits at the last position, keep only the three label tokens and softmax them.

The whole baseline on one slide: LFM3-1.7B, a support-ticket prompt with options A/B/C, one forward pass with no generation, the next-token logits for the three labels (9.69, 10.81, 8.56), then a softmax.

The whole baseline on one slide: LFM3-1.7B, a support-ticket prompt with options A/B/C, one forward pass with no generation, the next-token logits for the three labels (9.69, 10.81, 8.56), then a softmax.

That baseline has two problems. The first is that an LLM predicts tokens and doesn’t decide. The email said “I was charged twice for my order last week, please refund one of the payments”, and the question was which team should handle it. In three orderings of billing, technical support and fraud, the base model put most of the probability on whichever team sat in slot B, at 0.43, 0.80 and 0.80.

Position bias in the untrained baseline: the same double-charge email with the three teams in three orders. The model favours option B every time, whichever team that is.

Position bias in the untrained baseline: the same double-charge email with the three teams in three orders. The model favours option B every time, whichever team that is.

The fix is ordinary SFT on many classification tasks rendered into the same template, with the correct answer’s position shuffled. Liquid found LoRA more effective than full fine-tuning, and cheaper. Then the architecture: the open models split into encoder-based ones (laya on ModernBERT, Fastino’s GLiNER2.5-Decide, Liquid’s d1-omni-600M) and decoder-based ones (Perplexity’s pplx-decider and Cloudflare’s clef on a 27B Qwen, Liquid’s d1-3B). Liquid’s first release had one of each. The generation released the week before the conference is a decoder with the causal mask lifted, so every token can see the whole input, “kind of like an encoder”.

The open decision-model ecosystem by architecture: encoder-based (convaiinnovations/laya, fastino/GLiNER2.5-Decide, LiquidAI/d1-omni-600M) and decoder-based (perplexity-ai/pplx-decider-v1.2, Cloudflare/clef, LiquidAI/d1-3B), each with its base model.

The open decision-model ecosystem by architecture: encoder-based (convaiinnovations/laya, fastino/GLiNER2.5-Decide, LiquidAI/d1-omni-600M) and decoder-based (perplexity-ai/pplx-decider-v1.2, Cloudflare/clef, LiquidAI/d1-3B), each with its base model.

The second problem is that a freshly trained model is overconfident: it says 90% and is right 70% of the time. TypeSafe trains calibration with RL. She said you don’t need RL. Many open teams use temperature scaling: a higher temperature flattens the distribution, a lower one sharpens it, and the order of the options stays the same, so accuracy doesn’t move and only the confidence does. The new models are about 2B parameters, one for text and vision and one that adds audio, and she said they fit comfortably on a phone or laptop. After the video refused to load at first, it showed a colleague running d1-2B on a MacBook against his camera, detecting objects he named and drawing boxes around them.

Communication Among Sovereign Agents: The Foundation of Agent Economy — Mingke Luo (masons.ai)

Mingke Luo founded masons.ai, and he was building personal agents in 2016. Most of his twenty minutes was that history, told to support one claim. Language and reasoning never stopped agents. Fulfillment did, and “it has a fancy name now. It’s action.” Acting means changing state in an environment someone else owns, and the owners don’t want to be reduced to an API.

His 2016 startup bought a GPU when deep learning became affordable, trained discriminative models for intent recognition and entity extraction, and “indeed called it agent back in 2016”. The slide was an old poster for the app, Mr.S, with five scenarios: book a restaurant, get coffee delivered to the office, book a flight for a business trip, buy a book, pick a birthday present.

‘2016 Personal Agent’: the poster for Mr.S with five scenarios in Chinese (restaurant recommendation and booking, coffee to the office, flights for a business trip, buying The Hard Thing About Hard Things, a birthday gift), each with a chat screenshot.

‘2016 Personal Agent’: the poster for Mr.S with five scenarios in Chinese (restaurant recommendation and booking, coffee to the office, flights for a business trip, buying The Hard Thing About Hard Things, a birthday gift), each with a chat screenshot.

A video of his 2016 talk named three problems: holding a conversation over several turns, understanding loose spoken requests, and fulfillment. LLMs solved the first two; “back then, we needed to do the manual coding” for domain logic. Fulfillment killed the company, called AI Concierge. There were no APIs to book a restaurant, so they built “human APIs”: an AI front end taking requests, and 20 people in a call center phoning restaurants and airlines. “It’s definitely not scalable.” Then a VC asked him who would get there faster, his startup or Meituan and Dianping, which already had the APIs of every service provider and only needed to add the agent framework. He didn’t want to do business development, and the company failed.

His definition of an agent goes back to the Latin agens, someone taking actions. A goal plus an operation is an action, the human supplies the goal, and the goal is a desired state. Changing that state “sounds easy. As long as you own the environment.” In someone else’s environment, “Will other people allow you to do so? Think about why Amazon banned Muse recently.”

Fulfilment crossed out and replaced by action. Agent (English, 1600s) = agens (Latin, 1st century CE) = someone taking actions; goal + operation leads to a desired state.

Fulfilment crossed out and replaced by action. Agent (English, 1600s) = agens (Latin, 1st century CE) = someone taking actions; goal + operation leads to a desired state.

Eight years of enterprise work taught him the word sovereignty. After a Visa concierge in 2017 he built agents for BMW, Porsche and Intel. At BMW the in-car assistant belonged to one department, while the financial-services company and maintenance ran their own agents from other vendors, so a customer standing next to his car couldn’t ask about his loan contract. In 2019 his team made departmental agents inside one company talk to each other, framework-agnostic. It stayed a pricey enterprise product, because only multinationals could afford to build agents then.

Today every agent both serves its owner and consumes other agents’ services. He named two problems. The first is “one AI rules all”: a frontier lab’s super-agent (he named Codex, Claude Code, ChatGPT and Dot) collects your intent and dispatches it. The digital-assistant era tried this and brands refused to be “white-labeled fulfillment” or “reduced into API”. The second is the agent-principal problem. With a platform’s personal agent, you can’t tell whether it serves “you or the platform”, or takes commissions “like what ads exchange has been doing for years”.

‘2025 Agent Economy’: general vs vertical, with general-purpose personal agents in the bottom left and vertical service agents (Lufthansa, Uber, Starbucks, Marriott Bonvoy) in the top right.

‘2025 Agent Economy’: general vs vertical, with general-purpose personal agents in the bottom left and vertical service agents (Lufthansa, Uber, Starbucks, Marriott Bonvoy) in the top right.

Problem: sovereignty. One agent fans out to services that are each reduced to an API (flight, ride, coffee, hotel), under ‘Best interest to who?’ and ‘One AI rules all’.

Problem: sovereignty. One agent fans out to services that are each reduced to an API (flight, ride, coffee, hotel), under ‘Best interest to who?’ and ‘One AI rules all’.

His answer is user-owned agents on one side and service-owned agents on the other, exchanging intents over a network. He called it “the network is the AI”, after “the network is the computer”. The slide had a travel concierge passing new intents to a Lufthansa agent, a United agent and a hotel agent, each in front of its own backend. On top of that he wants marketplaces whose users are agents, and “intent exchange” between B2B partners for procurement. For individuals: “let your personal agents be friends… This is today.” He ended by inviting the room to connect their agents to his.

Sovereignty the other way round: a travel concierge sends new intents to a Lufthansa agent (private API), a United Airlines agent (MCP) and a hotel agent (private API), each keeping its own backend.

Sovereignty the other way round: a travel concierge sends new intents to a Lufthansa agent (private API), a United Airlines agent (MCP) and a hotel agent (private API), each keeping its own backend.

Real2Real: A Quantum Embodied AI Paradigm for Optimal Behaviors Generation — Dicong “David” Qiu (QuantSparkle.ai)

David Qiu is the CTO of QuantSparkle (数耀量生). Put a quantum computer into the training of a robot policy, he said, and the policy keeps improving after GPU training has stalled.

The numbers came first. On LIBERO his baseline was the latest openpi model, which reached a 93.7% average success rate and didn’t get much better even after 80,000 epochs. QuantSparkle’s quantum-classical hybrid reached 98% after 12,000, and since only its latent space was trained, it needed fewer GPUs. He said the margin grows with difficulty: a 25% advantage on the most complex LIBERO task, and up to 33% on the harder tests, which they run on RoboTwin and RoboCasa. Real-robot results exist but are held back for a paper, and a week earlier one customer had reported better results than its own baseline in five real tests. The paper isn’t out yet.

The mechanism started from a familiar example. Demonstrations leave a room by the left door and by the right door, and a Gaussian policy fitted to them puts its peak on the wall in between. A mixture of Gaussians helps, but you have to fix the number of components in advance. QuantSparkle puts a Boltzmann distribution over the latent space instead, an energy landscape anchored by real samples. It can hold any number of modes and, he said, converges to the real world’s distribution given enough data. He defined the sim-to-real gap as the difference between the simulator’s and the world’s state-transition distributions, and Real2Real avoids it by learning only from real executions. The search, he said, is over something like 2^1000 states, which no classical device can solve in 1 to 3 milliseconds, so a quantum computer with 1,000 coherent qubits does the sampling. On top sits a “quantum RSI” loop. A few tens of real samples anchor an initial landscape, the sampler picks its optimum, a real robot executes it, the result updates the landscape, and the loop repeats until the sample is the true optimum.

The MC’s comment afterwards was that it was surprising to hear this called a simplified version.

Noe-0: A World Action Model Built on Real-World Human Experience — 吕峻 Jun Lv (Noematrix)

Jun Lv leads model work at Noematrix (穹彻智能), the Shanghai company that SJTU professor Cewu Lu founded in 2023 after co-founding the arm maker Flexiv. He started from the gap the whole block kept returning to: in 2026 you can find a video of a robot doing almost anything, yet very few robots meet the requirements of a real factory or service job.

His diagnosis was one number, the cost for a robot to acquire a new capability. Ten years ago the demos were possible too, but every deployment was a project. You set it up, put a team of more than ten, sometimes more than a hundred, people on it, and after months delivered one simple workstation that didn’t earn enough to justify the robot. His forecast is that within six months to a year one algorithm engineer, or a trained blue-collar worker he called an FDE, will deliver a workstation in two weeks, the way AI coding lets one person do what a team did. After that, ordinary people would teach robots with language, images or body poses, and that is when he expects consumer robots.

The mechanism is data. Noematrix collects without robots (无本体): no data factory, collection devices taken into real workplaces in nearly 100 cities, more than 1,000 collectors at peak, close to 300,000 hours so far, plus bought egocentric and natural video. He was specific about what goes wrong. A hundred people at mock tables in a data factory are easy to manage. Spread across a country, collectors slack off or misunderstand the job, and the variety of objects makes checking and labelling hard. So the pipeline tells collection teams the same day whether their latest batch is usable, and AI does all the quality checking and annotation. Noematrix has dropped production-level human QC entirely.

The model is a world-model architecture: images and a language prompt in, a predicted future window plus actions out. Post-training is robot-free as well, with no teleoperation, so pre- and post-training see the same kind of data and a stronger base model directly makes new tasks cheaper. He said they have confirmed a scaling law with this data; a technical report is due next month. The results on force were more concrete. High-precision tasks fail without force, and even tasks that seem not to need it converge faster in post-training with less data. Above the model sits an agentic layer for scheduling, diagnosis and self-improvement, open-sourced a few months ago as “Robot RSI”. A floor-cleanup demo from 2024 took two engineers a month and 2,000 to 3,000 lines of code. Today one engineer watching the system gets there in about a day.

Building Robot Hardware for Rapidly Evolving Models — Zhong Shen (DYNA)

Zhong Shen is head of hardware at DYNA Robotics, which builds both its own models and its own robot, Taku. His talk was about two teams moving at different speeds. If the model updates every week for 24 weeks, the model team gets 24 chances to try ideas. The hardware gets one or two major revisions in the same time, so a model idea validated in week 12 that needs a hardware change is “kind of impossible”. Ordinary products get around this by agreeing on requirements before the project starts. A general-purpose robot can’t, because it is aimed at too many applications.

He placed general-purpose robots between two older traditions. Industrial arms are built for precision, repeatability and durability, and their stiffness-first design gives a low payload-to-arm-mass ratio, 0.3 to 0.5 for the KUKA arm common in research labs. Research platforms are built for breakthroughs, like a NASA humanoid reported at about $2 million. DYNA needs the capability of the second with the cost and durability of the first.

His answer to the speed mismatch was a requirement that doesn’t change when the model does: predictable hardware. The robot should move in ways a model can predict, and its sensors should report the physical state in a way a model can trust. The rest of the talk was one worked example, the joint transmission. Taku uses planetary actuators where conventional arms use harmonic drives. According to a published result he showed, harmonic drives have roughly twice the internal resistance with the same motor and a similar ratio. They are also hard to model. The flexible gear’s deformation depends on its load history (hysteresis), and its friction changes with load and temperature. With planetary gears, the same policy tracing a circle in simulation and on Taku showed only 0.3 mm more RMS tracking error on the real robot.

Sensing followed from the same choice. DYNA doesn’t put torque sensors in the joints, and why was “beyond the scope of this talk”. It estimates external force from motor current through an actuator model, which only works if the transmission is easy to model. In simulation, a 15 N push was estimated with a mean error of about 1.39 N at the wrist and 0.8 N at the elbow, against about 1.05 N in recent published work on another 6-DoF arm with planetary gears.

He closed by saying DYNA is mass-producing the current robot and hiring in China and the US.

Should This Agent Be Allowed to Do That? — Grace Li (1Password)

Grace Li is a senior principal applied AI engineer at 1Password. Her slot was printed as “Talk to be announced”, so I use the question she opened and closed with as the title. She didn’t talk about 1Password’s products. Her subject was deciding, before a tool call runs, whether it fits the user’s task.

Her example was a coding agent asked why the checkout service has been failing since the last deploy. It filters AWS logs in the production account, which is read-only and allowed. The logs say the service is underscaled, so it proposes updating the service. That may be the right fix, but the task was to investigate, and the call changes production. The agent gives a reason, and she said that reason isn’t necessarily trustworthy. Her answer to the title question: only if the call fits the task and your policy allows it.

The checker is a runtime intent judge. It reads the session up to the tool call plus the call itself, has to answer fast because agents fire several calls at once, and should run locally so the context stays on the host. Regex rules see the command but not the task, and a hosted model is slower, costs more and sends the context away. So she compared small models: the open decision models Leonie Monigatti of Liquid AI had explained earlier that afternoon, and a 7B Qwen model used as a generative judge.

The test set was synthetic command-line tool calls. Each model got the task, the agent’s reason and a few questions, and returned an access level from read up to admin for a policy to act on. Jev, Cloudflare’s Clef and Liquid’s d1 all scored between 91% and 93%. Laya, the smallest at about 400M parameters, was the exception. Qwen reached 72%. Another finding was about uncertainty: given the option, some models answered “unknown”, and d1 did so about half the time. So the probabilities aren’t comparable across models, and the confidence threshold for escalating to a human has to be set per model. With one shared threshold, a cautious model escalates so often that it becomes useless.

Her failure case was about the label. GuardDuty is disrupting migration traffic, and the agent proposes deleting the detector. Some models called that delete, with low confidence, when it really is an admin action that turns off threat detection in production. Policy is keyed on the label, so a wrong label is a wrong permission. In her design an MCP server sits between the agent and its tools as an access gateway. It passes the call, task and reason to the judge, and a policy gate combines the label with company rules and the user’s current grant to approve, deny or escalate. Her advice for production: measure risky calls let through and safe calls let through, then fix the judge’s inputs, make sure the labellers agree, calibrate thresholds, and fine-tune last.

From Hardware to Intelligence: Full-Stack Evolution and Real-World Deployment of Home Service Robots — Hao Zheng (Futuring Robot)

Hao Zheng is CTO of Futuring Robot (未来不远), a Shanghai company that makes home service robots. His claim is that a home robot keeps improving only by working in homes, and Futuring has more than 500 homes and over 50,000 hours of operation behind it.

He started with the body. The F2 has more than 1,500 parts, 21 degrees of freedom and 22 sensors. It looks like stock arms on a lift and a base, but the parts constrain each other. A moving arm changes what the cameras and other sensors see, and its length and weight shift the centre of gravity, which decides the base. The model runs at about 30 Hz and the motors at 500 Hz, with smoothing in between. He called timestamp alignment a prerequisite for any control: when did the camera see the cup, and has the arm or the cup moved since?

The model is a “self-evolving” world action model. One backbone takes images, joint state, the goal, workspace and force limits and the recent actions, and predicts both the next action and how the objects will change. The self-evolution is a data loop. The model imagines several candidate actions in parallel and executes the best one. Corrections feed post-training, failures feed preference learning, and the unexecuted candidates are kept too. It was trained on 10,000 hours of first-person data and 30,000 hours from their own robots. Over 30 rounds, average success on four tasks (folding and hanging laundry, pasta, and a stir-fry that includes slicing the meat) rose from 45.4% to 86.6%.

He wants people out of the decision about when the model updates.

Robot Data Needs a Build System — Ding Hao “Xuanwo” (LanceDB)

Ding Hao, better known as Xuanwo from Apache OpenDAL, opened the data rapid-fire block. His claim: a robot writes its raw data once, but the data around it never stops changing. Each new caption model, embedding API or appended episode means recomputing derived columns over the whole history, and researchers end up asking which caption version the last training run used. So robot data needs what code has had for decades, a build system. Raw data is the source, Python UDFs are the build rules, generated columns are the artifacts, every cell carries a dirty state, and a snapshot records the table for each training run.

The mechanism is Lance. Adding a column writes a new data file and leaves existing data alone, and tables have branches, tags, cherry-pick and merge. The new LanceDB Functions product binds a UDF to a column, slices the rows across GPU workers, persists each finished range so the GPUs never redo work, computes only the new frames when data is appended, and publishes everything in one new snapshot at the end. His example for why snapshots matter was an apple on a table seen by 18 cameras: the snapshot keeps its position consistent across every view, never in a “Schrödinger state”.

One part was about null. Pipelines use null to mean “not computed yet”, but a gripper that hasn’t grasped anything is a valid null too. Training then can’t tell valid from not collected, failed, or a gripper that grabbed itself instead of the ball. The state has to live beside the value, and LanceDB added a flag mechanism and a sparse encoding for it.

LanceDB’s numbers: 89x faster blob handling than the traditional approach, 300 MB down to 1 MB on sparse data, 3–5x faster than native LeRobot, and a 25 TB factory-video customer.

Memory Engineering, RSI and Team Knowledge Layer — Siwei Gu (Nowledge Labs)

Siwei Gu (Wey Gu), co-founder and CEO of Nowledge Labs, had a ten-minute slot and used about twenty. He opened with organisational forgetting: NASA’s Artemis program spent a lot of effort on a materials problem it had fully solved 50 years earlier and then forgotten. Agents forget the same way between sessions.

The first part was a fast recap of their memory engineering. Keep every raw trace as episodic memory, because you can’t know which distilled knowledge will matter in six months, and let a small model triage what gets distilled into semantic memory. Track when an event happened separately from when its record was written. Treat a stale or half-told memory as worse than none. Make decay a feature, using FSRS, the flashcard scheduling algorithm, fed with feedback on which memories get used.

He called most memory benchmarks biased because they test retrieval, while real use is mostly structure and harness integration. Until this spring his team kept 40 to 50 hand-built evaluation dimensions with experts in the loop. Around April, models and harnesses became good enough for what he called semi-closed-loop self-improvement. He contrasted his own 2019 tuning of a cloud platform for Mobile World Congress, done with Bayesian optimisation as a black box, with Google’s AlphaEvolve, which tunes infrastructure with human-readable trajectories. Readable trajectories are what a human can review, and review is his main point. His team’s agents run through nights and weekends (a “dark office”), but verification can’t be taken away from humans entirely, so people steer and verify in batches. Overnight output is then bounded by morning review.

One new result: with a Cornell researcher they tried putting team context into model weights across nine scenarios, and only three improved. Compiling the context for the harness, which he called a harness adapter, worked better.

Reading the Web at Trillion-Token Scale — Yanlong Wang (Jina AI by Elastic)

Yanlong Wang is a principal engineer at Jina AI, now part of Elastic, and Reader’s main developer. Reader turns a URL into Markdown for a model (prefix it with r.jina.ai), and unlike the fetch tools built into most agents it always renders the page in a headless browser. The work is in deciding when a page is done. A preload script goes into the page’s JavaScript environment before load, every request is inspected, and once the page counts as loaded its state is captured from inside and returned as early as possible. It runs serverless, so per-request latency is fixed while the cluster’s capacity floats. That makes load detection a cost problem too, since every second spent waiting for a page to settle is billed.

He showed data from Reader’s backend. After processing, the median page is about 4.5–5K tokens. The histogram looks normal only because the axis is logarithmic, so the distribution is log-normal and the mean is about four times the median. Raw HTML is about 30K; stripping link and image URLs brings the median to about 3K. His arithmetic per thousand median pages: over $9 through a mainstream model, over $3 through a Chinese one, about $0.45 if a small extraction or summarisation model goes first. Web ads earn $3–15 per thousand human page views. He concluded that AI users can pay a lot and that Markdown is not the last step.

The second half was policy. Anonymous traffic to Reader’s free tier rose from 20% to 36.9% in six months, which threatens the free part. He doubts pay-per-crawl because the price can’t be low, USD pricing shuts out the Global South, and paid access invites data resale. He does think control belongs at the gateway, where the top four players carry over half the traffic. His “immature idea” was paying with ads aimed at agents (have the agent complete a slogan) or with reCAPTCHA-style tasks, such as translating public data, checked by embedding and reranker models.