Overview

AI Engineer Shanghai 2026 ran on November 5 and 6 at the Hilton Shanghai Hongqiao, the first AI Engineer event held in China. I went for Day 1, whose theme on the program was open models, agents and infrastructure.

Registration (签到处) at 09:07, with the AI Engineer Shanghai backdrop and the sponsor names above it.

Registration (签到处) at 09:07, with the AI Engineer Shanghai backdrop and the sponsor names above it.

The day had one stage in the morning, from 09:30 until lunch at 11:40. After a community panel at 13:00, parallel sessions started at 13:45, with an AI applications breakout running next to the main stage. I stayed on the main stage for the agent and infrastructure talks.

The expo hall at 09:14, with the Optiver and Exa booths along the back wall and Brave on the right.

The expo hall at 09:14, with the Optiver and Exa booths along the back wall and Brave on the right.

The Manus booth with Daytona next to it. Both companies spoke on the main stage that afternoon.

The Manus booth with Daytona next to it. Both companies spoke on the main stage that afternoon.

The notes below follow the program in time order.

Welcome and opening video — Mengying and swyx (AI Engineer)

The day started at 09:30 on the main stage. Mengying, who organised the Shanghai edition, gave the welcome. This was the first AI Engineer event in China. She thanked the sponsors and sent everyone to the expo hall, then thanked the speakers for flying in.

swyx couldn’t come because AI Engineer had a large event in San Francisco the following week, so he sent a video. His pitch was that AI Engineer is for engineers who build things, while other AI conferences book politicians and heads of state. He called Shanghai the sixth AI Engineer location, named Singapore, Miami, Sydney, Paris and Melbourne among the earlier ones, and said India and Africa come next year. The one clear request at the end was to post notes, photos and videos so he could retweet them from home, since the conference wasn’t live-streamed.

From Models to Working Enterprise Agents — Marco Casalaina (Microsoft)

Marco Casalaina is VP Products in Microsoft’s Core AI group, and his job title in the Bay Area is “AI futurist”. He said up front that he wasn’t there to sell Microsoft products and wanted to talk about what agents will look like in the next six to twelve months. He planned to demo live, but the hotel network failed almost at once, so most of the talk ran on screenshots he had taken in case that happened.

His main point was that agents are getting their own identities. He started with Meta’s Muse, which asks you to name it when you install it (he called his “Lily”). He thinks Meta does this to make you treat the assistant as a friend, because Muse will make money from engagement. Microsoft does it for security. An agent built in Microsoft Foundry gets its own Entra ID, and the ones he calls autopilots also get a mailbox, a Teams presence and a OneDrive, like a new employee. His example was a banking agent, “Marco’s Teller”, that answered a customer’s question about an ATM fee by email and copied him on the reply.

Teams with the agent chats from the demo (Payroll Funding, Hotel Finance, Banking Operations, Marco’s Teller). Marco’s Teller, marked as an AI agent, answers in Chinese about Maria Garcia’s 35 dollar ATM fee.

Teams with the agent chats from the demo (Payroll Funding, Hotel Finance, Banking Operations, Marco’s Teller). Marco’s Teller, marked as an AI agent, answers in Chinese about Maria Garcia’s 35 dollar ATM fee.

The same agent as an email user: Marco’s Teller replies to the customer from its own mailbox and CCs Marco Casalaina.

The same agent as an email user: Marco’s Teller replies to the customer from its own mailbox and CCs Marco Casalaina.

The reason for all this is impersonation. Work IQ, the headless Microsoft 365 layer that agents use for Outlook, Teams and SharePoint, can run on behalf of the user, so the agent sends mail and deletes files as you. He guessed that nearly everyone in the room had at least one agent set up that way, and that’s dangerous because the agent inherits your entire security context. Tool permissions in the harness don’t fix it: “Harness level restrictions are like traffic lights in India. They are optional.” If you turn off the file tool, the agent uses rm -rf. If you turn off the shell, it writes a Python script. Nothing stops it, because as far as the system can tell, it is you.

Microsoft 365 Copilot with Work IQ in on-behalf-of mode: “Show me all of my recent emails involving Maria Garcia” returns the user’s own mail (23 emails), not the agent’s.

Microsoft 365 Copilot with Work IQ in on-behalf-of mode: “Show me all of my recent emails involving Maria Garcia” returns the user’s own mail (23 emails), not the agent’s.

Microsoft’s fix is least privilege on the agent’s own identity. His agent is a template in Agent 365, so anyone in the organisation can create their own copy, and the admin sets the permissions for that whole class of agents. He has read-write access to his files. The agent’s identity gets Files.Read.All, so it can’t delete or even write a file, whatever route it tries.

The last eight minutes were about what comes next. Today we orchestrate multi-agent systems the way parents arrange a preschool playdate. He expects agents to start arranging things among themselves, the way teenagers arrange their own plans, and called it unorchestrated collaboration. In his example, a hotel’s finance agent asks a banking agent to move money for payroll, and the banking agent asks the human to approve. That makes testing harder. Generic scores like groundedness and fluency don’t tell you much, so he showed rubric-based evals for his banking agent: refuse cross-customer access, comply with Regulation E and P, check that a fee was really charged before refunding it. Further out, he thinks each agent will need a simulated world of fake agents, people and messages so it can be tested in isolation. He pointed to OpenEnv as one attempt at that.

Building Agents: Yesterday, Today, Tomorrow — Clemens Rawert (Langfuse)

Langfuse is the open-source tracing and evals platform that ClickHouse bought in January. Clemens introduced himself as one of the co-founders. The talk started a few minutes before its slot.

His first point was that nobody specifies an agent up front. You run it with real users, read the traces, find where it broke, ship a fix and a test, and do it again. His example came from Halo, the assistant built into Langfuse. Reading their own traces, they found the agent calling a tool with totalCost_sum while the tool expected sum_totalCost. The fix was to accept both. He said building agents is “much more artisanal than scientific”.

Agents are improved iteratively: traces, spot failure, fix, ship. The example underneath is Halo’s tool call sending totalCost_sum when the tool expected sum_totalCost, fixed by accepting both.

Agents are improved iteratively: traces, spot failure, fix, ship. The example underneath is Halo’s tool call sending totalCost_sum when the tool expected sum_totalCost, fixed by accepting both.

The history part went quickly. First came single calls and chains, which soon ran into the small context windows of early models. Then RAG, then tools called in a loop, then hand-built agent graphs that cut a problem into pieces the models of the time could handle. He said the graphs had to be designed by hand, which took time, and you had to know every path the agent might take.

The tools-and-loops slide: an LLM call decides to call a tool, reads the result and picks the next step, for a couple of turns.

The tools-and-loops slide: an LLM call decides to call a tool, reads the result and picks the next step, for a couple of turns.

What works now, in his view, is an LLM in a loop with MCP tools, skills that act as a manual the agent opens only when it needs them, and a sandbox with a file system and bash. Halo is built this way. The agent loop runs in Langfuse’s own backend workers, and the sandbox is just another tool, called only when needed, because sandbox time costs money.

How Halo is built: the agent loop runs in Langfuse’s workers, talks to tools over MCP, and reaches a sandbox (file system, bash) only through a tool. Skills tell it how to do a task, with reference details loaded if needed.

How Halo is built: the agent loop runs in Langfuse’s workers, talks to tools over MCP, and reaches a sandbox (file system, bash) only through a tool. Skills tell it how to do a task, with reference details loaded if needed.

The second half was about who runs the improvement loop. Until now it has been people: find failures, write evals, fix, backtest, release, all organised around the question “what is good?”. He said this works and has become standard practice, but it eats engineer time, and that is the one thing a company can’t scale. Agents also run longer now. Some are always on and get triggered by an email or a schedule, so their traces have grown past the point where anyone reads them.

Humans used to run the entire improvement loop: find failures, evals, fix, backtest, release, around the question of what good looks like.

Humans used to run the entire improvement loop: find failures, evals, fix, backtest, release, around the question of what good looks like.

His answer is to let coding agents work on the trace data through MCP, an API or a CLI. Millions of traces won’t fit in a context window, so you shape the data first. Cheaper models summarize each trace, guided by what you know about your domain. The summaries are embedded and clustered, an LLM labels and ranks the clusters, and only the worst failures go to the coding agent as full traces. After that come background agents that run this loop all the time. Humans keep the definition of “good” and choose which feedback signals count. He also said the best teams use the labels from this loop to train their own open models for their task.

New agent architectures made runs longer and traces bigger: from a short linear workflow to an always-on agent triggered by email or a timer, with trace size growing bar by bar.

New agent architectures made runs longer and traces bigger: from a short linear workflow to an always-on agent triggered by email or a timer, with trace size growing bar by bar.

When we can do anything, do we still know what we want? — Ryo Lu (ex-Cursor)

Ryo Lu was the founding designer at Notion and later ran design at Cursor. He left Cursor this year and moved from San Francisco to Taipei.

Ryo Lu on the main stage with the title slide: 當我們什麼都能做,還知道自己想做什麼嗎?

Ryo Lu on the main stage with the title slide: 當我們什麼都能做,還知道自己想做什麼嗎?

He started from a common feeling. Someone shows off a new model, your first reaction is “wow”, and your second is “am I falling behind again?”. The easy response is to write more tools and open more agents. He asked when that is enough. He also asked whether tools that let us do more help us know what we want.

The example that carried the talk was small. Ask an AI to “make me a tool to record my life” and two minutes later you get mood scores, a streak counter, a daily reminder and a weekly report, each with a reasonable explanation. He admitted he would probably just say “OK, fine”. His worry is not that the answer is wrong. It is that a complete answer covers over questions he never got to ask, such as whether his life should be scored at all. He then had the audience raise hands for the feature they would remove. His own pick was the streak, the reminder and the report, because he wants to keep his life, not add homework to it.

The journal mock-up in page view, with the streak counter (6 days), the 21:00 daily reminder and the weekly mood report crossed out

The journal mock-up in page view, with the streak counter (6 days), the 21:00 daily reminder and the weekly mood report crossed out

He compared a “service window” with a “workbench”. A service window hands you a finished version and leaves you three moves: accept, reject, or explain again. A workbench shows the plan first, including which features it is about to delete, lets you step in before it acts and fine-tune with a live preview after. Sometimes seeing it about to remove the reminder is how you find out you wanted to keep it. He listed what he cares about when building tools: files, plans, editing, before/after diffs, and points where a human can step in. Then he gave three rules for tool builders.

Rule one for tool builders: make important assumptions visible, keep each change small enough to try safely, and make handed-off work retrievable when needed

Rule one for tool builders: make important assumptions visible, keep each change small enough to try safely, and make handed-off work retrievable when needed

The other two rules were to avoid a confirm button on every step and instead show users they can do more than accept or reject the whole answer, and that a good tool removes obstacles but should not remove the person’s process of forming a judgement.

The free-canvas version of the same journal in a Mac-style desktop: photos and short notes placed freely, one reading 想留下的是這些碎片,不是一份作業 (what I want to keep is these fragments, not homework)

The free-canvas version of the same journal in a Mac-style desktop: photos and short notes placed freely, one reading 想留下的是這些碎片,不是一份作業 (what I want to keep is these fragments, not homework)

The second half moved away from tools. He used email as the analogy: having it doesn’t mean every matter needs a letter. Leaving San Francisco was partly practice at not justifying everything by whether it helps work. He showed the journal again, once as one page per day and once as a free canvas, and suggested keeping both and asking someone who picked the other one why. His image for working with people was a jazz band, where nobody plays the most notes. One slide summed it up: he doesn’t only want a world that responds to him better, he wants to keep a world that can change him.

His closing ask was concrete. Next time you get a very complete answer, ask which decision in it you have not thought about yourself, change one small thing, and show it to another person.

Post-Training at the Frontier: Teaching Open Models to Search and Browse the Web — David Li & Alex Triplett (You.com)

Three people from You.com shared this slot. Alex Triplett, the COO, did the company part. He started by asking everyone to cover their eyes with their hands, which he said is what an LLM experiences when it can’t reach the web. You.com now sells search to agents rather than to people. Agents query faster and read far more than humans do, and he said that in the last year agents overtook humans as the main users of web search. The products are layered by how much orchestration you want, and they’re offered as an API or over MCP. He claimed billions of queries a month, mostly from enterprises that want zero data retention, and said You.com also acts as the “search lab” for model labs during pre-training, post-training and inference. Ryan Lee, who runs APAC, then gave a one-minute pitch about a new Asia headquarters for Chinese companies that need global search.

You.com’s API line-up, from cheap to thorough: Contents (full page text for a list of URLs), Search (with a parameter for licensed data), Answer API for async jobs, and Research for multi-step agentic research.

You.com’s API line-up, from cheap to thorough: Contents (full page text for a list of URLs), Search (with a parameter for licensed data), Answer API for async jobs, and Research for multi-step agentic research.

The technical part came from David Li. With NVIDIA and CoreWeave, they post-trained Nemotron 3.5 Lightning 30B-A3B to do web research. The base model scored about 37 on BrowseComp, a set of roughly 1,200 multi-hop research questions. He described the usual failure modes of smaller models: they repeat the same search, can’t make sense of complicated pages, and keep searching after they already have the answer because they can’t tell it’s the right one.

Three companies, one stack: an NVIDIA SFT checkpoint, GRPO training and a vLLM policy on the CoreWeave RL platform, a NeMo Gym agent calling You.com Search and Contents, and an LLM judge with a binary reward that also scores the BrowseComp eval.

Three companies, one stack: an NVIDIA SFT checkpoint, GRPO training and a vLLM policy on the CoreWeave RL platform, a NeMo Gym agent calling You.com Search and Contents, and an LLM judge with a binary reward that also scores the BrowseComp eval.

He said the RL dataset was the most important part. Open datasets age quickly and depend heavily on Wikipedia, so they generated questions from You.com’s own web graph. They sample pages from the middle popularity band, skipping the head (boilerplate like search engine home pages) and the spam-heavy tail. Then they follow one-hop links, generate multi-turn QA from facts on those pages, and check that the questions are answerable with their search. Before training, they decontaminate against BrowseComp and let the base model try each question eight times. A question solved 8/8 teaches nothing and one solved 0/8 is out of reach, so RL only keeps questions in between.

Building RL data from the web graph: sample middle-popularity pages, expand one-hop links, generate grounded multi-turn QA, verify it is answerable with You.com search, then bucket by difficulty.

Building RL data from the web graph: sample middle-popularity pages, expand one-hop links, generate grounded multi-turn QA, verify it is answerable with You.com search, then bucket by difficulty.

The numbers were small on purpose: 4,932 SFT traces, 1,480 RL tasks and an RL run of about eight hours on CoreWeave. His comparison table put that next to ASearcher’s 16K H800 GPU-hours of RL and RE-TRAC’s 104K extra SFT traces. BrowseComp went from about 37 to about 45. Efficiency was never part of the reward, but tool calls still went down, because the model learned to use the tools and to verify an answer once it had one.

Small data, big gains: 4,932 SFT traces and 1,480 RL tasks (about 8 hours of RL on CoreWeave), compared with WebSailor, OpenSeeker, ASearcher, OpenResearcher and RE-TRAC.

Small data, big gains: 4,932 SFT traces and 1,480 RL tasks (about 8 hours of RL on CoreWeave), compared with WebSailor, OpenSeeker, ASearcher, OpenResearcher and RE-TRAC.

The example trace showed the change best. On one BrowseComp question about a CEO and the car he drove while selling light bulbs, the base model made 313 tool calls, burned more than 8.3M tokens and answered wrong. The post-trained model made 7 calls with 74.7k tokens and got it right.

BrowseComp task #319 before and after post-training: 313 tool calls, 8.3M+ tokens and a wrong answer (Toyota Corolla) versus 7 tool calls, 74.7k tokens and the right one (Volkswagen Jetta).

BrowseComp task #319 before and after post-training: 313 tool calls, 8.3M+ tokens and a wrong answer (Toyota Corolla) versus 7 tool calls, 74.7k tokens and the right one (Volkswagen Jetta).

Their closing blueprint was to start with strong search, build questions from the real web, filter by difficulty, do SFT and then GRPO, and measure tool calls, tokens, time and cost along with accuracy.

Full-Stack Inference Engineering: Latency, Throughput and Cost — Chen Wang (Baseten)

Chen Wang leads partner engineering at Baseten, an inference platform that runs open and custom models for other companies. He wrapped the whole talk in a story. It started with Joseph Campbell’s hero’s journey and Luke Skywalker, and the payoff came at the end, where the hero is a single token on its way through the model.

The business part covered how companies move to open models. They start with a serverless API on an open model to save money. At scale they move to dedicated deployments to hit throughput and latency targets. Teams whose customer data is their moat fine-tune their own models so they don’t hand their customer signal “on a silver platter” to a model provider. He said open-model token consumption on OpenRouter passed closed models at the start of this year and the gap is still growing. There was also a short side plot about open ecosystems as the alternative to a cyberpunk future, illustrated with a drawing he had made himself.

The journey of open model inference adoption: closed API, open source via API, dedicated deployments, post-training and fine-tuning, then many specialized models, with cost, latency, rate limits and governance as the first inflection point.

The journey of open model inference adoption: closed API, open source via API, dedicated deployments, post-training and fine-tuning, then many specialized models, with cost, latency, rate limits and governance as the first inflection point.

The technical part started with the trade-off triangle and one chart from Artificial Analysis. For DeepSeek V4.1 Flash it plots total throughput per accelerator against per-request output speed, one line per GPU, from H100 to Vera Rubin and AMD’s MI355X. Concurrency picks your point on the line. Low concurrency gives each user fast output and the GPU low total throughput. High concurrency does the reverse. His advice for heavy traffic on few GPUs was to push throughput as far as the end-to-end latency allows.

The inference trade-off triangle (latency, throughput, cost) that framed the technical half of the talk.

The inference trade-off triangle (latency, throughput, cost) that framed the technical half of the talk.

Artificial Analysis chart: total throughput per accelerator vs p25 output speed for DeepSeek V4.1 Flash, with curves for Vera Rubin, GB300, B300, GB200, B200, H200, H100 and MI355X.

Artificial Analysis chart: total throughput per accelerator vs p25 output speed for DeepSeek V4.1 Flash, with curves for Vera Rubin, GB300, B300, GB200, B200, H200, H100 and MI355X.

Then came the single-node techniques. Prefill builds the KV cache and the first token, so it sets time to first token and is compute-bound. Decode reads the KV cache on every step, so it sets inter-token latency and is limited by memory bandwidth. That difference is the argument for running prefill and decode on separate replicas. On top of that he went through chunked prefill, FlashAttention, paged KV cache, quantization, expert parallelism and speculative decoding. His summary was that all of them reduce repeated compute and data movement.

One diagram of a request through prefill and decode, labelled with where each technique applies: chunked prefill, FlashAttention, expert parallelism, P/D disaggregation, continuous batching, quantization and speculative decoding.

One diagram of a request through prefill and decode, labelled with where each technique applies: chunked prefill, FlashAttention, expert parallelism, P/D disaggregation, continuous batching, quantization and speculative decoding.

One slide showed the cluster view. A global scheduler moves capacity between clouds (one cluster at 120/128 GPUs busy, the other at 40/96). A local controller adds and removes replicas per model. Cold start copies weights GPU to GPU from a peer that is already serving, with kernels compiled in advance. A KV-aware router sends a request to the replica holding its prefix, and KV cache gets offloaded from HBM to node RAM and SSD, then to a shared cluster store.

Cluster-level serving in six parts: multi-cloud capacity under a global scheduler, autoscaling, fast cold start from a serving peer, KV-aware routing, KV cache offload (GPU HBM, node RAM/SSD, cluster store) and P/D disaggregation with KV transfer.

Cluster-level serving in six parts: multi-cloud capacity under a global scheduler, autoscaling, fast cold start from a serving peer, KV-aware routing, KV cache offload (GPU HBM, node RAM/SSD, cluster store) and P/D disaggregation with KV transfer.

Measure the frontier of AI for real world use — Chen Peng (Arena)

Chen Peng works on the data side of Arena, the company that grew out of Chatbot Arena at UC Berkeley. The first few minutes covered the usual numbers (more than 16 million monthly visitors) and the familiar side-by-side battles.

The main part was what they do for agents. Pairwise votes don’t work when a session is an hour of tool calls, so Arena built its own agent product (“agent mode”, in the same space as Manus) with file editing, bash, web search and connectors for Drive, Notion and similar. Coding is the largest share of what people do with it. The example on the slide was a financial-research RAG pipeline that ran for 84 turns and 676 tool calls.

One real Agent Arena session: a financial-research RAG pipeline built with GPT-5.5 in 84 turns and 676 tool calls, next to a chart of tool calls per task by category.

One real Agent Arena session: a financial-research RAG pipeline built with GPT-5.5 in 84 turns and 676 tool calls, next to a chart of tool calls per task by category.

The evaluation is a randomized controlled trial on live traffic. Each session samples an agent configuration at random. Today only one component is randomized, the orchestrator model, picked from the hundreds Arena hosts; tools, system prompt and harness version are next. Every trace is scored into an outcome between 0 and 1, and a model’s effect is its mean outcome minus the mean over a reference mix of configurations. Every model sees the same distribution of real tasks, so there is no fixed problem set to train against.

The core idea: each session samples a configuration (today only the orchestrator model), the trace is scored to Y in [0, 1], and a component’s effect is E[Y | T_k = t] − E[Y] against a reference mix.

The core idea: each session samples a configuration (today only the orchestrator model), the trace is scored to Y in [0, 1], and a component’s effect is E[Y | T_k = t] − E[Y] against a reference mix.

The score comes from five signals that LLMs plus human curation pull out of the traces. Confirmed success is the user clicking Yes on a “done?” pop-up. Praise vs complaint counts unprompted “nice, that’s much faster now” against “the bug is still there”. Steerability counts how often the user has to correct the agent. Bash reliability catches things like broken sed invocations, and tool hallucination counts calls to tools that don’t exist, such as a made-up deploy_to_prod().

The five signals behind the agent leaderboard: confirmed success, praise vs complaint, steerability, bash reliability and tool hallucinations, each with a pass and fail example.

The five signals behind the agent leaderboard: confirmed success, praise vs complaint, steerability, bash reliability and tool hallucinations, each with a pass and fail example.

On the resulting leaderboard (5M+ agentic sessions), Claude Opus 5.5 was first at +20.94% net improvement, Gemini 4 Carbon second at +16.04%. Every row has a ± interval and a rank range. The top model’s range was 1→1, but the models in places 3 to 8 all had ranges like 2→8 or 3→9.

Arena Agent Leaderboard: net improvement per model with confidence intervals and rank ranges, plus the per-signal columns.

Arena Agent Leaderboard: net improvement per model with confidence intervals and rank ranges, plus the per-signal columns.

He then showed cost against performance. On the Pareto frontier of net improvement versus median cost per task, the Chinese open-weight models (Tencent’s Hy4, Xiaomi’s MiMo-V2.6-Pro, DeepSeek-V4.1-Flash) sit on the curve even though they trail the US models in raw score. His own summary was that US models lead and the Chinese open-weight models are in the second tier with a real gap, so pick from the frontier based on what a task is worth.

Agent Pareto frontier: net improvement against median cost per task, with Hy4, MiMo-V2.6-Pro and DeepSeek-V4.1-Flash circled on the frontier.

Agent Pareto frontier: net improvement against median cost per task, with Hy4, MiMo-V2.6-Pro and DeepSeek-V4.1-Flash circled on the frontier.

The rest was roadmap: the Alignment Index launched a month earlier (unauthorized actions, false attribution of intent the user doesn’t have, and claiming a task is done when it isn’t), randomizing harnesses and letting labs bring their own, a user simulator trained on agent-mode sessions so an eval takes hours instead of weeks, and RL environments built from traces. It ended as a pitch to the model labs in the room.

Lunch break and the expo hall

Lunch was on your own, from 11:40 to 13:00, and I spent the end of the break at the booths. Most of them demoed agent products. One was Raytone’s AI Box, a small desktop machine for running models and coding agents locally. Its spec sheet claims up to 2070 TFLOPS of FP4 compute, 128 GB of memory shared by the CPU and GPU, Omarchy OS and Raytone’s model manager preinstalled, and one OpenAI/Anthropic-compatible endpoint that nine coding agents, including OpenCode, Claude Code and Codex, can point at. It measures 160 × 146.4 × 60 mm and weighs 1.1 kg.

The Raytone AI Box on its stand, with an Omarchy sticker on the front.

The Raytone AI Box on its stand, with an Omarchy sticker on the front.

Raytone’s spec sheet: 2070 TFLOPS FP4 with a 2560-core GPU, 128 GB unified memory, one gateway for local and public models, nine coding agents through one endpoint, and data that stays on the device.

Raytone’s spec sheet: 2070 TFLOPS FP4 with a 2560-core GPU, 128 GB unified memory, one gateway for local and public models, nine coding agents through one endpoint, and data that stays on the device.

Earlier that morning, at the Zilliz booth, I had picked up a flyer for the Vector Lakebase that Rentong Guo talked about in the afternoon. It describes one copy of the data on S3 in the Vortex columnar format, with vector indexes built directly on Lance and Iceberg tables.

Zilliz’s Vector Lakebase flyer: real-time vector search, iterative data discovery and batch analytics over one copy of data on S3.

Zilliz’s Vector Lakebase flyer: real-time vector search, iterative data discovery and batch analytics over one copy of data on S3.

Community in the AI era — AJ (WaytoAGI), Koji (Crossing), Kai Qu (42 Chapters)

The afternoon opened with a 40-minute panel of people who run Chinese AI communities. AJ founded WaytoAGI, which started as a shared document. Koji hosts the Crossing (十字路口) podcast, runs AI Hacker House in Shanghai and is a venture partner at ZhenFund. Kai Qu runs 42 Chapters (42章经), a podcast and WeChat account for founders and investors. Gao Ning of Linkloud, a community of Chinese founders going abroad, moderated.

Koji made the case for communities with two stories. Around 2010, as a Beihang student whose classmates aimed for Microsoft Research Asia or American grad schools, he interned at Fanfou in Wudaokou, with Kuxun and the early Kuaishou a few buildings away. Only there did his choice feel normal. The second story was from 2016. Xiao Hong, who later founded Manus, had spent a RMB 1 million angel cheque from ZhenFund without traction and was close to quitting. ZhenFund’s Liu Yuan told him to come to Beijing and talk to founders, Koji among them. Xiao and his two co-founders flew in from Wuhan for an afternoon and went home to keep going. Koji’s line was that “people need their own kind”.

After about ten years of trying communities of thousands, an expensive business school, dinners and events, Kai Qu concluded that what you build is decided by your business model. Media and communities make money from advertising, which follows traffic, or from transactions, which follow the community. 42 Chapters has about 500,000 followers, but most of its work is community events, financial advisory and its own investments. It gave up on advertising for high-priced, selective offerings. Communities without such an incentive, he said, rise fast and are gone within a year or two.

AJ described the other end of the scale. WaytoAGI grew from the document to WeChat and Feishu groups, then livestreamed calls, then meetups in one city, five, and now 52. The mechanism is trust built offline. Members who have met a few times know each other’s character and “delivery ability”, so one who is good at sales takes an order and passes the work to other members without hiring anyone. One member launched a short-drama tool with a community livestream and got 200 users that day. This year the focus is buildings: AJ described an eight-storey building in Beijing’s Yizhuang holding more than a hundred one-person companies (OPCs), a new space in Chengdu, and an incubator with partners such as Sequoia.

On 2026 the moods differed. AJ saw members move from playing with AI to starting businesses, mostly as OPCs who left their jobs and earn the same money with more freedom. Investors are getting cautious about agent startups, AJ said, “because anyone can do it now”. Koji pictured the foundation models as a tank coming down the middle of a road. Application founders standing in its path had the hardest year. The practical ones dig down, into forward-deployed engineering, data, infrastructure, chips, inference acceleration and API relay services, or fly up into AI for science, biology and AI EDA. Kai Qu said everyone he deals with is anxious, and that pitches now start from what the newest model can do instead of what users need.

They disagreed about how often to speak in public. Kai Qu said to protect your momentum (势能). A founder who talks every week starts repeating themselves. Many of his guests have one great episode in them after ten years of work and need three to five more years for the next. He added that most guests come to recruit, not for publicity. AJ argued the other way: the information gap has closed, AI-generated content buries everyone and big companies give tokens away, so founders should put out more substance and present better. AJ rated Chinese founders weaker on stage than those at OpenAI and YC pitch events, and cited a post by Zhang Tao of Manus: his team makes a “PRV”, a video of the feature, before the PRD. Koji’s answer was to treat PR like a product: a gray release of the narrative to a few friends, research with the target audience, and advice from someone who has done it.

From Model Intelligence to Agent Harness: Enabling Long-Horizon Agents — Ailing Teng (StepFun)

Ailing Teng is the general manager for model product, platform and ecosystem at StepFun, the Shanghai model lab. Her question was how an agent handed hours of work remembers the goal, adjusts to feedback and delivers something you can trust. Her answer had two halves. A model can hold more information, but remembering where a long task stands is the harness’s job.

She presented Step 5 Preview, released in September, as four properties to check in any model you build agents on. It activates 27B parameters per token, and that matters because a long task repeats the same step thousands of times, so per-step cost is what you pay. The 1M context is mostly useful for code, long documents and task records, and she said a longer window doesn’t help much outside those. Native vision is there so the agent can look at screenshots, design drafts and rendered output and check its own work. Tool calling is what makes feedback possible at all.

Her evidence was three long runs. An agent iterated on a kernel for a single H100 for more than 22 hours. A post-training data-pipeline experiment went from 53.3% to 60% accuracy. A Pokémon run with no game-specific tuning took more than 3,000 rounds and 6M tokens to clear about two thirds of the main story. She also talked about training. StepFun built its own Step Code Bench, with repositories, languages and tasks it chose, and developers now say the model is good at bug fixing and refactoring. Her lesson was “if you did it, you have it; if you didn’t, you don’t”, meaning not everything waits for generalization.

The harness half named four gaps: forgotten requirements and repeated failed paths, too many logs, missing acceptance criteria, and cloud work that must survive the client disconnecting. Step Code, StepFun’s open-source harness, keeps progress outside the context window in four places: a task tree for goals and steps, a plan with progress and the next step, the session history, and the filesystem with code, logs and outputs. Each step rebuilds the working context from the current goal, the relevant files and the latest tool feedback. Logs and screenshots pass through a budget stage that truncates text and compresses images, old logs stay outside until needed, and subagents hand back only conclusions. For verification she wanted acceptance criteria fixed before execution. For a web page, compiling is one step, and whether it does what the user wanted has to be defined by the user and the harness together.

Her runtime picture was a brain and hands: the cloud harness plans and checks, the goal, plan and session history sit in the middle, and the hands are the local machine or a remote sandbox. Studio, on stepfun.com, hosts that cloud harness, and a desktop app is due in November. She skipped efficiency and sent those questions to the booth.

Dark Factory to Lobsterville: Intent Engineering for Persistent Multiplayer Agents — Vincent Koc (OpenClaw Foundation)

Vincent Koc is the foundation’s Chief Architect. The abstract promised “Lobsterville”, multiplayer steering and intent handoff. Most of the twenty minutes was about OpenClaw’s CI bill.

He started from his London talk in April, where the claim was that engineers had become factory managers and that the bottleneck was the engineer’s taste. Since then his own setup has grown from 10-15 agents to 60, and he said he was now running over 300 in parallel. He said he shipped somewhere between 1,000 and 1,500 PRs the day before the talk and close to 2,000 the day before that, from a hotel room in China, half of them from his phone over Telegram. His limits were token rate limits, memory, compute, CI and “every single rate limit with every single company and provider you can possibly imagine”.

Slide 04, ‘Making changes outpaced checking them’: many lanes of agent changes feed a queue waiting for proof, which drains through one lane of tests, review and integration, and then a person who understands the change.

Slide 04, ‘Making changes outpaced checking them’: many lanes of agent changes feed a queue waiting for proof, which drains through one lane of tests, review and integration, and then a person who understands the change.

His point now is that the bottleneck has moved to proof: how you know a change works before you land it. He split the work into background loops (attempt, check, improve), which only work where the feedback is clear, and foreground work, where a human picks the outcome and inspects the evidence. He calls the new job “proof engineering” and described himself as “a proof engineer, or essentially a vibe eval guy”. One example was OpenClaw’s computer-use feature. They put Doom inside it and had an agent make the game loop faster every time it found something slow.

Slide 07, ‘A loop needs clear feedback’: background work as an attempt, check, improve loop, and foreground work as choosing the outcome, inspecting the evidence and steering the next attempt.

Slide 07, ‘A loop needs clear feedback’: background work as an attempt, check, improve loop, and foreground work as choosing the outcome, inspecting the evidence and steering the next attempt.

Then came the numbers. In mid-September OpenClaw’s CI was on pace for $1.27M a month at list price on Blacksmith runners. That is a lot for a non-profit he described as “five or six engineers”, even though most of it was sponsored compute. Agents kept adding tests, and you can’t just delete them, because the tests are the only proof the agents did anything. What they did instead was turn off CI on main entirely. PRs now get a limited check plus agent review, and the full suite runs every 30 to 60 minutes and at release.

Slide 13: projected monthly CI cost down 82.3%, from $1.27M at the September peak pace to $225k at the latest pace. The slide title adds ‘Still not done.’

Slide 13: projected monthly CI cost down 82.3%, from $1.27M at the September peak pace to $225k at the latest pace. The slide title adds ‘Still not done.’

Slide 16, ‘Cheaper did not erase the wait’: list price per 100 CI runs went from $1,176 to $297, while median CI time only went from 15m21s to 11m47s.

Slide 16, ‘Cheaper did not erase the wait’: list price per 100 CI runs went from $1,176 to $297, while median CI time only went from 15m21s to 11m47s.

Cost per 100 runs fell by three quarters, but median CI time fell by less than a quarter. He tracks agent cycle time, from PR opened to PR merged, and his line was “if the factory slows down, the factory can’t fix itself”. The next step is adaptive testing, merged a few days earlier. A classifier looks at the changed and executed code, predicts which tests are likely to regress, and only those run. He expects it to cut total CI cost much further.

Slide 17, ‘Can we run less and still catch the bug?’: the full suite versus only the affected tests, with the goal of the same regression signal for less time and cost.

Slide 17, ‘Can we run less and still catch the bug?’: the full suite versus only the affected tests, with the goal of the same regression signal for less time and cost.

He closed with a 135-task benchmark for personal agents, built with partners over two or three months, with a small public subset due within days. The tasks are deliberately ambiguous, like tax returns or two people disagreeing on an outcome.

On-Device AI: Bringing Intelligence Closer to Everyone — Alex (AutoArk)

Alex is the CTO of AutoArk (无界方舟), a company that builds models meant to run on the device itself. The first third of the talk was the business case against cloud APIs for hardware. His example was a device that sells a million units a year, with each user talking to it more than 20 times a day. At public API prices he put the token bill at about RMB 1 billion a year, so the better the product sells, the more money it loses. The other two arguments were the usual ones: no network on a plane or in a lift, and recordings and files from medical, legal or investment work ending up at the model provider.

He also listed what it costs. A device has limited memory, battery and endurance, so you can’t simply ship the biggest model, and every platform (FreeRTOS, Arm Linux, ESP32, iOS, Android) needs its own deployment code.

AutoArk splits hardware into three tiers. For glasses, earbuds and small companion robots they ship ASR and TTS models down to 0.1B parameters, and the demo played fast transcription and fairly natural emotional speech synthesis. Around the models sits the EVA SDK, which handles the audio front end and full-duplex, interruptible conversation.

EVA SDK slide: audio 3A front end (AEC, ANS, AGC), low-power VAD wake-up, on-device ASR/LLM/VLM/TTS that works offline, device-cloud switching with full-duplex interruption, on ESP32, Linux, Android, iOS and macOS, plus one-command install for coding agents.

EVA SDK slide: audio 3A front end (AEC, ANS, AGC), low-power VAD wake-up, on-device ASR/LLM/VLM/TTS that works offline, device-cloud switching with full-duplex interruption, on ESP32, Linux, Android, iOS and macOS, plus one-command install for coding agents.

The demo was a 35B MoE model on an iPhone 16 Pro using at most 2 GB of memory, in airplane mode, with a screen recording he said was not sped up. The explanation was short but concrete. In an MoE only 3 to 10% of the parameters are active for a token, so the full model stays on flash and only the active experts are loaded into RAM. The catch is that consecutive tokens can need completely different experts, and loading them from storage on demand would cost several seconds per token. Their fix is a trained “pre-router” that predicts which experts the next token, the one after it and a few more will need. A background planner then loads them asynchronously, so they are in memory when decode reaches them. He claimed an order-of-magnitude decode speed-up from this.

How the 35B model fits: the full model on SSD/flash, a RAM area with a shared core and two slots (current layer and next layer), CPU/NPU compute, and a planner that predicts and loads ahead. The three points on the left are early MoE expert routing, SSD/flash streaming and LoRA post-training to recover accuracy.

How the 35B model fits: the full model on SSD/flash, a RAM area with a shared core and two slots (current layer and next layer), CPU/NPU compute, and a planner that predicts and loads ahead. The three points on the left are early MoE expert routing, SSD/flash streaming and LoRA post-training to recover accuracy.

For laptops and desktops he previewed a 150B MoE multimodal model that will be open-sourced. He said it needs 18 to 24 GB of memory and reaches more than 1,000 tokens/s prefill and more than 100 tokens/s decode. All their models and the inference engine are open source under the Edge0 name.

The closing case was a sports earbud built with Shokz. Most of what users ask an earbud is skip track, volume, heart rate and cadence. An on-device ASR model and intent classifier handle those offline, and only open questions go to the cloud. He said this cut token cost by more than 92%.

Kubernetes Is Not Your Sandbox — Mislav Ivanda (Daytona)

Mislav Ivanda works on developer experience at Daytona, which sells sandboxes for AI agents. His argument was simple: if your team needs isolated environments for agents, don’t ask the infra team for a Kubernetes cluster. Daytona runs its own control plane on Kubernetes, so this wasn’t a rejection of Kubernetes as such. His point was that it was built for long-running, stateless services that must survive failures and roll updates, and agent sandboxes want the opposite. They are short-lived, and they have to keep their state when an agent is paused and resumed two days later.

Mislav Ivanda on the main stage: there is no settled definition of a sandbox, because the use keeps changing.

Mislav Ivanda on the main stage: there is no settled definition of a sandbox, because the use keeps changing.

He spent a few minutes on how the word sandbox drifted. First it meant a place to run model-generated Python, then a shell for the model. Then the agent itself moved inside the box. For the last half year people have been running thousands to tens of thousands of sandboxes at once for RL work: evals, benchmarks, post-training. Computer-use agents for enterprise apps without APIs are now going the same way.

Then the Kubernetes numbers, as Daytona sees them. A pod goes through a fixed sequence of steps before it runs, so you get 1 to 2 seconds at best and usually tens of seconds, around 100 pods per second, and 110 pods per node by default. State is where it gets worse. A pod has no persistent disk of its own, so you mount external volumes, and then all I/O goes over the network. He quoted about 500K IOPS on Daytona’s local disks against 50 to 60K on EBS. Attaching a volume takes tens of seconds, and a failed mount can only be retried after six minutes. A node takes 25 to 39 volumes, so the volume limit, not the 110 pods, sets how many sandboxes fit on a node.

Daytona’s three mechanisms: snapshots kept on the runner’s local disk, popular snapshots cached on more nodes, and only diffs moved between registry and runner.

Daytona’s three mechanisms: snapshots kept on the runner’s local disk, popular snapshots cached on more nodes, and only diffs moved between registry and runner.

Daytona’s answer is to keep state where the sandbox runs. A stopped sandbox stays on the runner’s disk, so resume is just a container start, and placement prefers nodes that already hold the image. Popular snapshots get copied to more runners, unused ones are garbage-collected, and the registry stores deltas so even a cold start pulls only what is missing.

Trajectory latency by action count for Daytona, Docker, Kubernetes, EC2 and Fargate, in three tiers from a 50 ms sleep action to an attrs bug-fix loop with pytest.

Trajectory latency by action count for Daytona, Docker, Kubernetes, EC2 and Fargate, in three tiers from a 50 ms sleep action to an attrs bug-fix loop with pytest.

The benchmark ran three tiers of agent trajectories: a sandbox doing nothing but 50 ms sleeps, light compile-and-test work, and a real bug-fix loop. Daytona had the lowest line in all three. Fargate sat far above the rest at 340 to 390 seconds for 150 actions, and in the sleep-only tier Daytona came in at 12.7 s, against about 25 s for Kubernetes and 34 s for EC2. He also gave a cost model: 1,000 sandboxes for 176 hours, a month of nine-to-five, came out 2.7 times cheaper than a self-managed Kubernetes cluster, mostly because the cluster sits idle between bursts. These are Daytona’s own numbers.

AI-Native Work at Manus: One Monorepo, Open Tasks — Fan Bin (Manus AI)

Fan Bin leads agent engineering at Manus and works on its agent harness. He opened with a scene from an essay by Manus co-founder Zhang Tao: a PM posts a requirement in Slack, and the engineer pastes it word for word into one of a row of Codex or Manus sessions, which does the work end to end. His claim was that the agents are already strong enough, and what stops them is information and access. The context for a task sits in private chats and people’s heads. The tools are spread over nearly 30 repositories plus test environments, logs, databases and a release system, each with its own entry point and permissions.

The first fix is manus-mono, one workspace holding 28 repositories, layered AGENTS.md rules, 39 shared skills, 43 knowledge entries and 10 runbooks, and 22 local tools, so one agent can go from writing code through testing and release to debugging and data analysis. Setting up mono also wires in Slack and Notion. Given a Slack link, the agent reads the thread, finds the spec in Notion, reads the current code, builds the feature, deploys it to the test environment and then @-mentions the PM in the same channel to try it. Logs are traced end to end with OpenTelemetry, so the agent can follow one trace ID to the repository, the change and its author, usually within 20 minutes. An express mode for urgent incidents gives a location and a suggested fix within five.

What is in mono: 28 repositories, AGENTS.md rules, 39 shared skills, 43 knowledge entries plus 10 runbooks, 22 local tools, and one entry point for the whole loop from writing code to release, debugging and data analysis.

What is in mono: 28 repositories, AGENTS.md rules, 39 shared skills, 43 knowledge entries plus 10 runbooks, 22 local tools, and one entry point for the whole loop from writing code to release, debugging and data analysis.

With that in place, people outside engineering open PRs. Since June 2026, three PMs and two designers submitted 297 PRs, of which 248 (about 84%) were merged. One PM had Manus test a dozen speech models, made a demo from the real frontend code, and shipped a voice-input feature. Quality rests on a review bot running the strongest model on every PR, plus owner sign-off for payment, billing, model and core-prompt changes. He said no production incident so far has come from code quality.

Everyone can open a PR: 297 PRs from 3 PMs and 2 designers, 248 merged, with per-person bars (PM A alone 182 submitted, 164 merged).

Everyone can open a PR: 297 PRs from 3 PMs and 2 designers, 248 merged, with per-person bars (PM A alone 182 submitted, 164 merged).

The product is split the same way: a small kernel with the agent loop, sessions and base tools, and plugins such as slides, web development and video editing that each own a repository and ship on their own schedule. His reason was the agent’s: a cohesive plugin means less cross-repository work per change and a smaller blast radius. One to three people own a module, which he said sometimes feels like maintaining an open-source project.

Plugin architecture: a lean kernel (agent loop, sessions, base tools) behind one interface for tools, skills, hooks and UI cards, with plugins released independently.

Plugin architecture: a lean kernel (agent loop, sessions, base tools) behind one interface for tools, skills, hooks and UI cards, with plugins released independently.

The second fix is that every task is public. It started because they couldn’t see how colleagues used AI, or help the ones using it badly. In practice it decouples a task from its owner. If a task is stuck on a decision, the link goes to whoever can make it, they answer inside the task, and the agent carries on. A PR links its task, so a reviewer can read how the change was prompted. His summary was that mono fixes the agent not seeing the whole context, and public tasks fix the people not seeing it.

The step after that is agents acting on their own. Colleagues have pulled agent twins into Slack. One compares pods, restarts and error rates before and after each release, and once found an incident before their monitoring did. An automatic pickup system, inspired by loop engineering, claims an issue within 10 minutes and runs it through a coordinator, a worker and a review bot to a human who merges. Completed tasks went from 13 on day one to 38 on day two. That wasn’t his point. The bot opened PRs far faster than anyone merged them, dozens sat untouched within days, and nobody knew their priority. Without a person setting priorities, he said, automated development just burns tokens.

The automatic task system (beta): issue, pickup within 10 minutes, coordinator, worker, review bot, human acceptance. 13 tasks on day 1, 38 on day 2, and after day 2 twelve PRs and three designs waiting for their owners. The slide’s last line: code is no longer scarce, acceptance is.

The automatic task system (beta): issue, pickup within 10 minutes, coordinator, worker, review bot, human acceptance. 13 tasks on day 1, 38 on day 2, and after day 2 twelve PRs and three designs waiting for their owners. The slide’s last line: code is no longer scarce, acceptance is.

The pitfalls slide listed people as the bottleneck again, approval requests nobody understood after a day of agent work, and one colleague spending $100,000 on tokens in a month. He called the last one autopilot: handing over a task and answering “you’re right, continue” to everything. The fix is design, task splitting and staged checks, because money can be earned back and wasted time can’t. His line on approvals was that tokens are renewable and human attention isn’t.

Pitfalls: people are the bottleneck again, approvals pile up that nobody understands, and one colleague burned $100k of tokens in a month.

Pitfalls: people are the bottleneck again, approvals pile up that nobody understands, and one colleague burned $100k of tokens in a month.

Powering open source personal agents with Exa search — Jakub Hojsan (Exa)

Jakub Hojsan leads forward-deployed engineering at Exa, and the talk was a single demo: a personal agent, built from open-source pieces and Exa’s API, planning one evening in Shanghai. His framing was that a personal agent adds continuity between requests and makes proactive help possible: “an agent that can pick up where we left off and help with what comes next”.

The agent found a restaurant on the Bund, pulled photos from Exa’s image results and then from the venue’s site through a browser, and narrowed the options by atmosphere, view and budget per head. He showed how they keep pages out of the prompt. Exa’s highlights feature can return only the passages relevant to the query, extracted from the page without rewriting it. If those passages are enough, the agent answers. If the user wants a photo or a detail is missing, it escalates to another tool. He also listed what Exa indexes besides web pages: news within about 15 minutes, filings, podcasts, events, places and weather. The weather got a real use in the demo. When he asked about a walk along the Bund after dinner, the agent said no because rain was forecast.

Preferences are stored per person, not per user. In the demo, his friend dislikes spicy food and he loves chili, so “my friend prefers milder food” was saved against the friend. The proactive example was an email from Jazz at Lincoln Center Shanghai offering two seats. The agent only interrupts if three conditions hold: his calendar is free, the price is within his ticket budget, and two seats are actually available. Underneath, that is either Exa’s Monitors product or plain periodic search plus browser use, which he said is how the closed-source providers do it anyway.

His advice was to start with one recurring task, then decide what the agent should remember, what it should look up, and what should survive after the chat ends.

The handover right after Jakub’s slot: Rentong Guo of Zilliz on stage with his title slide about architecture choices under 10x and 100x data growth. I didn’t get a usable photo of Exa’s slides.

The handover right after Jakub’s slot: Rentong Guo of Zilliz on stage with his title slide about architecture choices under 10x and 100x data growth. I didn’t get a usable photo of Exa’s slides.

Your Agent’s Memory Is Next Quarter’s Training Data — Rentong Guo (Zilliz)

Rentong Guo is a co-founder of Zilliz, the company behind the Milvus vector database. The abstract was about agent traces turning into training data. The talk itself was mostly about something else: why Zilliz rebuilt Milvus around object storage and now calls it a “Vector Lakebase”. Agents and training came up in the last few minutes.

Milvus intro slide: deployed at close to 10,000 enterprises, 300+ contributors, 150M+ on the Docker counter and 4.3K+ forks.

Milvus intro slide: deployed at close to 10,000 enterprises, 300+ contributors, 150M+ on the Docker counter and 4.3K+ forks.

He started from two growth curves. Token use for the same task goes up about ten times at each step from text completion to chat to reasoning to agents. The second curve is Zilliz’s own observation: the largest single business behind one of their customers held hundreds of millions of records in 2023, billions in 2024, tens of billions in 2025, and about a hundred billion this year. That is one workload, not a total. He expects 2027 and 2028 to grow faster, because more of the data will be generated or simulated (world models, autonomous driving, AI-made short dramas) and so grows with compute spend.

The 1X / 10X / 100X slide: performance tuning, then cost tuning, then re-architecture.

The 1X / 10X / 100X slide: performance tuning, then cost tuning, then re-architecture.

His framing was 1x, 10x, 100x. Performance tuning gets you through the first step and cost tuning through the second: performance versus capacity instance types, algorithm gains, DRAM versus NVMe inside one server. Past 100x, he said, there is no room left inside a server, and you have to re-architect.

To explain why, he used a beer-hall analogy. Engineers are the waiter carrying a growing stack of full beer mugs (servers) while someone keeps pouring in more beer (data), and nothing may spill. In production a spill is a severity-one incident. His slide listed OOM, scaling at the high-water mark, resharding, data rebalance storms and failure blast radius. He also cited a well-known AI coding company that, according to a talk he watched, changed databases four times in a few months during its 2024–25 growth spurt and only stabilised after moving to an object-storage design.

Finite capacity vs. exponential data growth: records per largest workload from 2023 to 2026, and the failure modes of a fixed-size cluster.

Finite capacity vs. exponential data growth: records per largest workload from 2023 to 2026, and the failure modes of a fixed-size cluster.

His definition of a Lakebase is the experience and roughly the performance of a database, at the capacity and cost of a data lake. Underneath are three tiers: a local cache on the query server, a remote cache between server and object storage, and S3. The slide priced them at about $3, $0.3 and $0.03 per GB-month. Every tier can answer queries, so the question becomes placement. In his agent-app example, all registered users live in object storage, weekly actives (8–30%) in the middle tier and daily actives (3–15%) in the local cache. When a dormant user logs in, the application tells the data layer to promote that user’s data, which takes seconds, so it is usually hot by the first query.

Case 01, an AI/agent app: daily actives on Tier 1 at about $3/GB-month, weekly actives on Tier 2 at about $0.3, all registered users on Tier 3 at about $0.03.

Case 01, an AI/agent app: daily actives on Tier 1 at about $3/GB-month, weekly actives on Tier 2 at about $0.3, all registered users on Tier 3 at about $0.03.

The training-data angle came on one late slide showing a loop. The agent reads from the store for real-time retrieval, its behaviour, feedback and notes are written back to the same bucket, and interactive discovery and batch analytics over that data feed the next training set and agent strategy.

The serving and iteration loop: the agent reads from the Vector Lakebase for real-time retrieval, its behaviour, feedback and notes go back into the same bucket, and interactive discovery and batch analytics feed new knowledge, training data and agent strategy.

The serving and iteration loop: the agent reads from the Vector Lakebase for real-time retrieval, its behaviour, feedback and notes go back into the same bucket, and interactive discovery and batch analytics feed new knowledge, training data and agent strategy.

As a next step after Lakebase he pointed at caching query results the way LLM serving caches KV state, so repeated multi-round lookups are not computed again.

Building a reliable, multi-region LLM gateway — Xinghong Fang (Optiver)

Xinghong Fang is an AI engineer at Optiver, and every model call made by Optiver’s developers, researchers and business users goes through his team’s gateway. The scale: around 3 million requests a day across four data centres, about 22 models carrying 95% of the traffic, and several providers that each speak their own protocol. Upstream providers fail on about 0.4% of requests. At this volume that is a lot of broken sessions.

Title slide, with the live caption on the side screen catching his opening line: an LLM gateway sounds simple, you take a request and forward it to an upstream provider such as Anthropic or OpenAI.

Title slide, with the live caption on the side screen catching his opening line: an LLM gateway sounds simple, you take a request and forward it to an upstream provider such as Anthropic or OpenAI.

They started with an open-source gateway (he didn’t name it) that translated between provider protocols. It worked until the Slack messages came: “is the gateway slow?”, with the +1 count climbing to 30 or 100. His point was that a gateway is a utility now, and users notice an outage before your monitoring does. The first lesson was that HTTP 200 doesn’t mean success. Streams stall halfway, and translation broke prompt caching, which matters because cached input costs about a tenth as much. The nastiest case was silent. When Anthropic models were called through the OpenAI-compatible protocol, the thinking blocks were dropped from the response. The user still got an answer, but the next turn had to redo the thinking, so latency and cost went up and nothing showed in monitoring. A user caught it.

Every important model has at least two providers. They fail over only if the user hasn’t seen any output yet, for example on a 429 or 529. Once streaming has started they surface the error and let the user retry. He knows some gateways can continue a stream on a second provider, and they decided it wasn’t worth the complexity. A circuit breaker with per-error thresholds takes a provider out for a while. Sessions are sticky, so one conversation always stays on one provider and its cache, and only features that both providers support are exposed under a shared model ID. And because “backup always breaks when you need it”, 10% of sessions always go to the secondary provider.

The multi-region story got knowing smiles from the room. They wanted one API key that works in every region, so auth sat in one central PostgreSQL. Compliance requires logging every prompt and response, and the gateway stored that in the same Postgres. The result was more than 200 GB a day crossing the Atlantic and about two seconds of auth overhead on every call. The fixes were to write content logs straight to S3 from each region and run auth from local replicas, accepting a few seconds of delay for permission changes. That cut the overhead a lot. Spend limits are eventually consistent, so going over the cap for a few minutes is acceptable.

Even after all that, users still said it was slow. Distributed tracing found the answer for one complaint: inference took 17 seconds, which was fine, and a single tool call took more than three minutes. In the end Optiver built its own gateway around the smaller feature set it actually uses. He said that isn’t for everyone, and they would go back to open source if it fit.

Nobody Planned AI Infra: How AI Builders Pushed a Database Beyond the Database — Lux Li (TiDB / PingCAP)

Lux Li runs the AI business at TiDB, and his talk was one long story built on a request that agent companies hear all the time: turn these materials into a review workbench, the team uses it tomorrow, and we will keep changing it. His point was that this one sentence assumes the work survives the agent, the sandbox and the day. Each time it didn’t, a TiDB customer asked for something new, and that is how the product line grew.

The request the talk was built on: turn these materials into a review workbench the team uses tomorrow, keep the code and materials, we will keep changing it. The flow runs from receiving the request through writing and running code to a launched workbench the team keeps using.

The request the talk was built on: turn these materials into a review workbench the team uses tomorrow, keep the code and materials, we will keep changing it. The flow runs from receiving the request through writing and running code to a launched workbench the team keeps using.

The first failure is an interrupted agent. To resume, the platform keeps two tables, an agent session snapshot (context, plan, tool-call history, 40 to 80 MB per snapshot is normal) and an agent session event log written row by row in separate transactions. Across a fleet he said this metadata reaches hundreds of TB and sometimes PB. One customer had spread it over more than 20 sharded databases and moved to TiDB to stop maintaining the shard routing.

The second failure is the sandbox. The agent writes code on an EBS volume, the task ends, and the recycling policy deletes the volume with the user’s project on it. Snapshotting the volume fixes this on paper, but his slide listed the full backup and restore chain the platform then has to own.

Slide 11: what EBS snapshots still leave the platform to build. Backup is coordinate writes, snapshot, register project and version, retain/verify/clean. Restore is pick the snapshot, create and mount the volume, rebuild the runtime, realign Meta and files.

Slide 11: what EBS snapshots still leave the platform to build. Backup is coordinate writes, snapshot, register project and version, retain/verify/clean. Restore is pick the snapshot, create and mount the volume, rebuild the runtime, realign Meta and files.

TiDB’s answer is a persistent filesystem that lives outside the sandbox. Session A writes, the sandbox goes away, session B mounts the same workspace. Each project is its own isolated space, and he claimed it scales to tens of millions of them. The same pattern covers the apps an agent generates: instead of a platform running its own instance pool and dealing with create timeouts and half-built databases, TiDB Cloud Starter hands out tens of millions of isolated databases on shared storage.

Slide 20: a trace as a queryable execution record, linked by run_id / trace_id / span_id / parent_span_id. Sample sizes: 500 MB for one trace, more than 100,000 steps for a complex task, hundreds of TB written per day by one platform.

Slide 20: a trace as a queryable execution record, linked by run_id / trace_id / span_id / parent_span_id. Sample sizes: 500 MB for one trace, more than 100,000 steps for a complex task, hundreds of TB written per day by one platform.

The tracing section had the most numbers. Agent tasks running two or three hours, sometimes more than ten, produce single traces of hundreds of MB. One trace explains one run. To decide where to optimise you need a batch, sliced by tool and task type, with unit cost that includes failed retries. He pointed out that a tool being called a lot may just mean it works badly. For an unnamed large-model team the pipeline was raw NDJSON traces, SQL cleaning, and an analytical copy on object storage (TiDB Cloud Lake) feeding two separate compute paths, one for troubleshooting and one for batch quality and cost analysis.

Slide 22: hundreds of TB to tens of PB of traces. Raw NDJSON is loaded and cleaned with SQL into events and aggregates, kept in TiDB Cloud Lake on object storage, and queried by separate troubleshooting and batch-analysis compute.

Slide 22: hundreds of TB to tens of PB of traces. Raw NDJSON is loaded and cleaned with SQL into events and aggregates, kept in TiDB Cloud Lake on object storage, and queried by separate troubleshooting and batch-analysis compute.

If you change a model or a tool and just re-run the task, the inputs, tool returns and files have moved underneath you, so you can’t say whether the change helped. A trace tells you what happened, but you can’t restart from it. Their checkpoint ties the metadata snapshot and events to a filesystem checkpoint, so you can fork two identical copies of the agent’s state, try one change on each, and judge both by the same standard before checking similar tasks and repeating runs for randomness.

Slide 24: re-running from scratch versus forking from a shared starting point (same Meta and files, independent fork candidates, the same acceptance standard), with experimental writes and external side effects isolated.

Slide 24: re-running from scratch versus forking from a shared starting point (same Meta and files, independent fork candidates, the same acceptance standard), with experimental writes and external side effects isolated.

Optimizing Organization-Wide LLM Inference Cost: Seat License, Pay-per-Token, or Provisioned, and Beyond — Do Tuan Anh (Techcombank)

Do Tuan Anh, who goes by TA, runs the data, AI and agent platforms at Techcombank, one of Vietnam’s largest commercial banks. The 17:00 break was skipped, so his talk started about ten minutes ahead of its printed slot, which is why the two rapid-fire talks after it also ran early. He took about twelve minutes.

His argument was that once AI use reaches the scale phase, the number to manage is the effective blended cost per million tokens: input and output for the whole organisation, after caching. Theirs sometimes jumps, because users want the best model, Opus, and stop using Sonnet or Haiku. There are three ways to pay: subscriptions such as Claude Code or AWS Kiro, pay-per-token APIs, and provisioned GPUs. His “efficiency point” is the monthly cost of the GPUs divided by the blended API price, which gives the volume above which you should host the model yourself.

The worked example was GLM 5.3. Through the API it costs about $1 per million input tokens and $5 per million output. With the 4:1 input-to-output ratio he gave for general use (10–15:1 for tool calling, MCP and RAG agents), that blends to about $2 per million. Two AWS nodes with eight B200s each cost about $166K a month. He gave the break-even as “around 83 million” tokens a month; with his numbers, $166K at $2 per million is 83 billion tokens. In his other example, a company at 67 billion tokens a month should not switch and loses about $32K a month if it does. Above the point he expects 30–50% savings, more with multi-year savings plans and negotiated discounts, less after paying the people who run it. For 300 billion tokens a month he put it at $500K on the API against $200K provisioned.

He gave three caveats. GPUs are hard to get, so you plan with the hyperscaler early. Peaks are real: autoscaling had to grow from 12 to 15 GPUs, so the bank keeps the API as a fallback, with provisioned GLM as the main model behind Databricks’ Unity AI Gateway. In trials about 10% of requests went to the API and users noticed nothing. And the numbers move a lot with each company’s discounts. He ended on routing. Some models tokenise Vietnamese and Chinese badly, so the same text costs more tokens on them. The bank routes by language and meaning, which he said saves another 40–50%. His NeurIPS workshop paper this year measures that tokenization tax for Vietnamese.

Reinforcement Learning at Scale on Agent Substrate: Orchestrating Millions of Sandboxes — Tinsley Shi (Google)

This was a ten-minute rapid-fire slot, and it started about ten minutes before its printed time. Tinsley Shi works on agentic infrastructure at Google, and the talk was about the sandboxes behind RL post-training for coding agents.

Her starting point was a single RL step. The model gets a task, say “write hello world in Python”. A sandbox is claimed with the task image, the LLM thinks, its untrusted code runs in the sandbox, and a grader scores the result. In practice a lab samples a group of around 16 trajectories per task over about 1,000 tasks, so one step asks for roughly 16,000 sandboxes at once, all released in lockstep. She pointed out that normal agent serving and inference almost never see that kind of concurrency, and stock container infrastructure was never built for it.

The workload slide: one GRPO rollout step is acquire (target under 1 s), generate (10-15 s idle while the LLM thinks), execute in gVisor (2-3 s burst), reward and tear down. Duty cycle about 15% CPU active, 85% waiting on the LLM.

The workload slide: one GRPO rollout step is acquire (target under 1 s), generate (10-15 s idle while the LLM thinks), execute in gVisor (2-3 s burst), reward and tear down. Duty cycle about 15% CPU active, 85% waiting on the LLM.

Two more problems came on top of the bursts. Coding tasks are packed one GitHub repo plus its dependencies per image, so each image is a few GB with almost no shared layers (the slide said about 2.2 GiB each, 55 TB in total). The other problem is about data quality: if a sandbox times out or runs out of memory, a naive harness scores the rollout as a zero reward. An infrastructure failure turns into a wrong label, and that wrong label goes into the policy gradient. Meanwhile the sandboxes sit idle about 80% of the time waiting for the model, and the accelerators wait on sandbox start. Her slide put it plainly: p99 sandbox start latency is accelerator idle time.

Headline scorecard on GKE: 993,897 concurrent gVisor pods across 36,811 images, 0.83 s median time to first instruction at 1M scale, 12.3 minutes to fill the full 1M corpus, and 0 bytes of registry traffic at fill. Six clusters, 4,567 nodes, about 36,500 vCPUs.

Headline scorecard on GKE: 993,897 concurrent gVisor pods across 36,811 images, 0.83 s median time to first instruction at 1M scale, 12.3 minutes to fill the full 1M corpus, and 0 bytes of registry traffic at fill. Six clusters, 4,567 nodes, about 36,500 vCPUs.

The numbers come from the open-source Agent Sandbox controller on GKE. They held about a million concurrent gVisor pods for eight days without losing a sandbox, with a median of 0.83 s to first instruction and a 12.3-minute fill from a warm pool. Density was 60 sandboxes per node. The fixes she described were all orchestration: placing and caching images close to the nodes, tuning the API server and controllers, and collapsing network identities so that 10,000 sandboxes need far fewer of them. They also ran it with TPUs on a 397B-parameter Qwen model. The demo video of their dashboard didn’t play.

The next step is Agent Substrate: snapshot and restore instead of standing warm pods, with under 300 ms resume on the slide and a separate data plane to get around the control-plane limits. She said a release was expected at the end of November. Both projects are open source.

Wrap-up slide: 1M sandboxes on Kubernetes works today, warm pools keep TPUs busy, and next comes snapshots instead of standing pods, with QR codes for kubernetes-sigs/agent-sandbox and the RL-on-Agent-Substrate work in progress.

Wrap-up slide: 1M sandboxes on Kubernetes works today, warm pools keep TPUs busy, and next comes snapshots instead of standing pods, with QR codes for kubernetes-sigs/agent-sandbox and the RL-on-Agent-Substrate work in progress.

Life of a Game Developer in AI era — Tang Feihu (Kimi / Moonshot AI)

Tang Feihu runs developer relations at Moonshot AI and gave the last rapid-fire slot of the day, ten minutes long. His claim was that agents can now carry long tasks on their own, so building a playable game prototype has become cheap, and the developer’s job moves to deciding what to build and accepting what comes back. He said many engineers around him no longer write code by hand, and some no longer review the AI’s code either, because there is too much of it.

The task-shift slide: Specify (player experience, constraints, success criteria) stays with people, Generate, Operate and Verify go to the agent, and Decide (game feel, architecture, release risk) comes back to people. Bottom line on the slide: displacement is strongest where outputs are repeatable and can be checked by machines.

The task-shift slide: Specify (player experience, constraints, success criteria) stays with people, Generate, Operate and Verify go to the agent, and Decide (game feel, architecture, release risk) comes back to people. Bottom line on the slide: displacement is strongest where outputs are repeatable and can be checked by machines.

The slide put the line here: work that a machine can check (scripts, shaders, scene assembly, replay and log comparison) gets automated first, and the parts nobody can write a test for stay with a person.

The rest was a demo reel, mostly built with Kimi K3.1 Pro. He showed an animation the model rendered from a goal, where the hard part was making an explosion interact with the scene around it, and a paper storyboard by a colleague with no game background turned into animation. He pointed to long-horizon benchmarks that now give the agent a sandbox and a GUI tool and score what it delivers, instead of a code snippet in a chat box. One example task asks the agent to use RPG Maker to rebuild the old Flash game Magic Tower (魔塔); others drive a game engine or Blender. At an exhibition booth Moonshot showed three games made by its own engineers with the latest model, about four days and about RMB 3,000 of tokens each.

He also went through community projects. A viral fake “grandma GTA” video on Douyin became a playable open-source game on GitHub about a week later, using a “video to game” skill. People are porting Switch games such as Diablo II to the iPhone and redesigning the controls for touch. He also showed a Little Prince game by a community ambassador who had presented in the breakout room earlier that afternoon, and an action game by a colleague with no engineering background.

A worked task brief for a Godot match-3 prototype with three rule cards: the deliverables (editable project, 3-minute playtest video, rules.md, seeded replays, test log), the agent’s build order, and acceptance checks such as ‘same seed reproduces the same board’. The footnote marks these as illustrative proposals, not reported results.

A worked task brief for a Godot match-3 prototype with three rule cards: the deliverables (editable project, 3-minute playtest video, rules.md, seeded replays, test log), the agent’s build order, and acceptance checks such as ‘same seed reproduces the same board’. The footnote marks these as illustrative proposals, not reported results.

His last example was his own. He had an earlier model reverse-engineer an old game, unpack its world map and add sailing and explorable cities. The agent watched the running game through screenshots, noticed a bug, fixed it, and wrote a report on how it found it. His point was that self-checking like this lets one iteration run longer before a human has to step in. He closed by saying mainstream studios are adding models to their pipelines and the human role becomes acceptance and design.