Day 1 digest · Oct 13

Day 1 · October 13, 2026 · Main-stage livestream

Agent systems, evaluation, and the foundations for putting AI to work.

Lessons from both days · Day 2 digest · Day 1 session summaries

Practical takeaways for AI engineers

Benchmark the system before you fix it. About 200 hand-labeled questions, built in under a week, turned a vague complaint into a metric - and five improvements moved precision at 10 from about 60% to 90-95%. (Vault: Making Unstructured Data Work for Investing)
Replay your own work history as evals. Treat each pull request as an issue-resolution pair, run agents against them, and learn which models are overkill. Vals cut spend 90% this way. Judge benchmarks on validity, reliability, and discriminability - not five or six vibe-check samples. (Beyond the Model Card: The Jagged Frontier of AI in Finance)
Authorize per tool call, not per agent. Use OAuth 2.0 token exchange so each call gets an ephemeral, task-scoped, attributable token that expires within minutes and is never stored. (Your Agent Is a Lethal Trifecta, Intent Based Auth)
Score agents by risk and route review proportionally. A ticket summarizer and a production deployment agent do not belong in one approval queue; one heavy queue breeds shadow IT. (Balancing Acceleration and Safety for AI Agents)
Turn near-certain approvals into rules. If you would say yes 99.9999999% of the time, make it a deterministic fast path instead of a human prompt - consent fatigue is the failure mode. (Balancing Acceleration and Safety for AI Agents)
Have domain experts narrate judgment, not steps, when they write skills. Test on five to ten real runs before anyone else sees it, and ship with an owner plus a reviewer who is not the author. (We Let Our Financial Planners Write the Agents)
Split translation from execution. One planner model call turns a request into a typed semantic graph; a compiler that never calls a model turns it into an immutable DAG. Make waits, approvals, and follow-ups first-class nodes, and fail closed. (Reliable and intelligent workflows for finance, wealth, and insurance)
Keep the authorization boundary outside the agent. Inject credentials the agent cannot see, control egress down to paths rather than domains, keep audit logs where the agent cannot reach them, and separate the supervising agent from the worker. (Great Agents Start with Great Infrastructure, Intent Based Auth, Your Agent Is a Lethal Trifecta)
Treat shared GPU hardware as an attack surface. The demonstration inferred model and workload information from timing. Its headline percentages and individual experiments measure different outcomes. Do not place untrusted MIG slices next to sensitive models, and watch for one kernel relaunched thousands of times against the same 40 kilobytes. (Extra lunch talk: what a GPU co-tenant can learn about your LLM from timing alone (MIG and NVLink side channels))
Tune serving before buying GPUs. With the fleet fixed at 56 B200s, Revolut moved throughput from about 3 to over 7 requests per second and cut time to first token from over 10 seconds to about 1.6 seconds. (Shipping like a startup, regulated like a bank)
Ask explicitly for product polish from coding agents. Auto-save, optimistic updates, undo, archiving, and URL state are patterns agents skip by default. End every feature request with end-to-end verification by actually running it. (Become your company’s SaaS layer — principles for building polished internal tools with agents (Railway))
Trace the factory, not just the product. Instrument the coding agents behind your agents, and evaluate the evaluators against golden truth datasets before shipping a new judge. (What does an AI engineer do?)

Five themes of the day

The diffusion gap: intelligence runs ahead of adoption. Anthropic opened with it - what a model can do on its own doubles every four months, while three quarters of AI’s economic gains go to a fifth of companies - and named finance the next threshold crossing after coding and cyber. Sid Pednekar quantified the same gap from the buyer side: only about 37% of enterprises report positive EBIT contribution from AI, and only about 6% are high performers. The MC’s return segment gave it a ladder: 90% LLM rollout, 60% trained users, 50% coding agents, 5% non-engineering agents. The Kepler fireside added the human constraint - information still has to get into your head.
Agents that run the business, not copilots beside it. Arca gives agents whole 10-day client outcomes; agents there “prompt people more than people are prompting agents” and have done “90% of the work” before a person acts. Bridgewater treats agents as a second class of reader: 100-plus queries per second against about 100 per hour from humans, ending in an always-on investor workbench. Range pushed authorship to non-coders - 24 skills company-wide, 17 in planning, all written by planners, marketers, and salespeople. Arize framed the engineer’s job as building that always-on loop.
Govern the agent: authorization, controls, and side channels. Two Sigma opened with the $900 million Citi/Revlon wire error - three humans approved a correct warning - and answered with five composable controls instead of lockdown. The Intent Based Auth panel and Keycard’s lethal-trifecta demo converged on the same fix: authorize each tool call against a mission, with ephemeral tokens that expire within minutes. Stealthium extended the threat model to hardware: a GPU co-tenant inferred model information from timing in the demonstrated setups. Revolut described sandboxing and human review in its regulated engineering environment.
Evals built on your own work history are the moat. Vals AI made the sharpest cost claim of the day: 10 to 12 engineers used a reported $1.5 million worth of free tokens in a month, and replaying GitHub pull-request history as agent tasks cut spend 90% for the same output. Vals has crossed six trillion tokens of evaluation, and warns that “agents solve the test, not the problem.” Coatue agreed from the buy side: compartmentalize into units you can evaluate, then hill climb - without a North Star “you’re just fidgeting around.” Southbridge ran a statewide Ohio entity-resolution job for $795 against a projected $10,000, and Bridgewater benchmarked search with about 200 hand-labeled examples in under a week.
Own the substrate: infrastructure economics and open foundations. Revolut self-hosts GLM on 56 B200 GPUs at 30 cents per million tokens - one-tenth of the previous Anthropic and OpenAI invoice - and reported 70% better lead time in an internal comparison. That comparison does not isolate the model change from other causes. Modal set the infrastructure bar: brains separated from hands, 10,000 sandboxes per second, and a warning not to vibe-code a sandbox platform. Perpetual designed for about 95% of model decisions never needing a frontier model, so local inference becomes a configuration change. Wells Fargo trained its own 1-billion-parameter transaction model from scratch. Nous Research closed the loop: build agents on open foundations you own, because tomorrow’s moats are network effects, reputation, and reliability.

Talks you might have missed

The published agenda listed a 12:15-2:05 PM Networking Break with no sessions. The livestream filled it with seven talks that never appeared on the schedule, plus an early Modal start. Room attendees who left for lunch missed them; stream viewers got almost two extra hours of content.

We Let Our Financial Planners Write the Agents (Range) - Matt Ragone, a certified financial planner who writes no code, on the pick-teach-pack-test-ship method for expert-written Claude skills.
Your Agent Is a Lethal Trifecta (Keycard) - Kim Maida demos an overprivileged agent leaking PII and dropping a production database, then rebuilds it with token exchange and intent-based authorization.
What a GPU co-tenant can learn about your LLM from timing alone (Stealthium) - Branislav Brzak on MIG and NVLink side channels: model and workload information inferred from timing, without reading the other tenant’s memory; results vary by setup.
Become your company’s SaaS layer (Railway) - Mahmoud Abdelwahab of Railway, recorded outdoors in Edinburgh, on replacing marketing SaaS with one agent-built internal tool, “700 million tokens later” including real-time collaboration.
No universal schema (Southbridge) - Hrishi Olickel on just-in-time ontologies, model priors as an error class, and an Ohio entity-resolution job that finished at $795 on an $800 budget.
Reliable and intelligent workflows for finance, wealth, and insurance (Perpetual) - Karthic Rao on the determinism-versus-judgment boundary, typed plans compiled by a model-free compiler, and 95% of decisions on a flash tier.
Flywire’s internal AI platform - Eduardo Simon and Javier Casal introduced the Victoria PaaS and the Orbit workflow platform before the stream audio cut mid-slide after 2 minutes 43 seconds. The recording does not establish what was presented elsewhere.

Why this block matters: two of the day’s most concrete security talks (023, 025) aired here, off-agenda, and the whole block is invisible in the published schedule. Modal’s talk then started at 1:57 PM, about 26 minutes before its 2:23 PM slot.

The most quotable moments

“And so our question for you is, does your AI know what’s going on in the world, and does it remember what your firm has learned about it?” - Sam Green, Bridgewater, 00:30:51, session 003
“Intelligence is ahead of adoption. What a model can do on its own is doubling every four months.” - Belinda Neal, Anthropic, 00:34:18, session 006
“It’s so that you can go fast and stop safely.” - John Feminella, Two Sigma, on why cars have brakes, 01:09:34, session 008
“As an AI engineer, your job is no longer to just write the feature, write an AI feature.” - Aparna Dhinakaran, Arize AI, 01:29:49, session 013
“In an AI-native company, you know you’ve built one when you can steer your business by steering your agents.” - Aman Singh, Arca, 02:32:56, session 018
“First one, long context is a scam.” - David Abitbol, Revolut, 06:10:58, session 036
“Every single thing that a human can see in pixels on the screen, an agent in your building should be able to see via API.” - Frank Long, Coatue, 07:16:17, session 043
“Skynet was, after all, a closed source company.” - Dillon Rolnick, Nous Research, 07:56:18, session 049

Watch or browse the day

Watch the Day 1 recording · Agenda and stream notes · All Day 1 summaries