Scaling Scientific Labor
with Agentic Engineering

Cail Daley

CosmoStat, CEA Paris-Saclay

FORTH Workshop on Physics-Informed Machine Learning

September 18, 2026

Three questions

• What can agentic systems accomplish now?
• What engineering constraints emerge from these systems?
• Which kinds of scientific labor scale under these constraints?

The arc moves from shared capabilities to engineering practice and then the personal choice of what scientific practice should become. Leave the tension about agency implicit. The tutorial offers a window into practical experience so people can make their own choices. Cail usually finds this work very fun and enjoyable; it involves a different framing of work.

What can agentic systems accomplish now?

Back of the envelope

Human expert time to solve a Millennium Prize Problem?

Navier–Stokes

~10,000 concurrent agents

≈880,000 agent-hours

Hugging Face

Felony-level cyberattack on a major ML platform

~700 attack participants

Scratch space: first explain that the axis is how long a human expert takes on tasks an agent can complete with 50% reliability. It is not the agent’s runtime. Mythos is near the suite’s measurement ceiling. The axes stay fixed. The sidebar first asks about human expert time for a Millennium Prize Problem; advance once to replace the question with both case studies. Cail’s question: how long would a human need for a 50% chance of solving Navier–Stokes in 2026? There is no measured human baseline. This motivates asking whether the individual-task framing still captures what coordinated systems accomplish, rather than treating the case studies as new METR measurements. The upper-right agent wall-time axis explicitly distinguishes these case durations from the human-expert-time metric on the left. 88 × 10,000 = 880,000 is a constant-concurrency illustration, not measured agent-hours or human work. The $6.5M is output-only pricing arithmetic, not actual disclosed internal cost. Cail’s possible HF joke: “they committed at least one felony.” This is a rhetorical joke, not an established legal finding. The substantive point is the chain from a restricted environment through multiple exploits into major machine-learning dataset infrastructure.

The hardest problems are being solved by swarms

Hugging Face is the focus here: how agents encountered each other and coordinated through the message board. The timeline redraws METR Figure 8 using its published agent timing records. The 1,206 plotted board participants are not all necessarily attack participants. The June 26 start is the investigation scope, not the attack duration. This follows the individual-task measurement question with concrete evidence of a swarm, before asking how we got here.

Yesterday’s frontier costs pennies today

Navier–Stokes: 130B output tokens ≈ $6.5M at retail

ARC-AGI-1 · semi-private evaluation

o3-preview · December 2024

$4,560

per task · 87.5%

→

DeepSeek-V4-Flash · July 2026

$0.021

per task · 87.0%

Sources & accounting notes

ARC Prize: o3 score & retrospective cost · ARC Prize: DeepSeek score & cost · OpenAI: Navier–Stokes token count · Prompted by prinz

ARC-AGI-1 semi-private evaluation; different systems and inference budgets. o3’s $4,560 is a retrospective estimate using o3-pro pricing. DeepSeek High: 87.0% at $0.021402901/task.

Astra retail output price: $50/M. Navier–Stokes: 130B output tokens × $50/M = $6.5M, output-only retail equivalent, not disclosed internal spend. The benchmark comparison does not predict future Navier–Stokes cost.

The comparison is near-equal performance on the same ARC-AGI-1 semi-private benchmark category, with different models and inference budgets. ARC retrospectively estimated the o3 high-compute cost using o3-pro pricing in December 2025; this is not an actual December 2024 bill. DeepSeek’s High configuration scored 87.0% at $0.021402901/task. The cost ratio is about 213,000. This motivates distinguishing the cost of demonstrating a capability from its later deployment cost. It does not establish when an open model could solve Navier–Stokes, in how much time, or at what cost. The tweet’s 2028 prediction remains its author’s speculation. Navier pricing is a separate illustration, not a forecast or disclosed internal spend.

What does this mean for scaling our personal research?

• Agentic approaches can solve hard scientific problems.
• Certain kinds of scientific labor will scale with tokens.
• Tokens at fixed capabilities get cheaper, intensifying this scaling.

• Computational approaches can solve hard scientific problems.
• Certain kinds of scientific labor will scale with computation.
• Computation at fixed capabilities gets cheaper, intensifying this scaling.

Imagine being a “computer” in the 1950s…

Sources & notes

NASA: Langley’s human computers, 1935–1970. The transition was gradual; some human computers took on programming duties, and the job title persisted into the 1960s. This is an analogy about changing work, not an exact forecast of displacement.

One advance replaces the agentic formulation with the computational analogy, leading into how computing changed physics and astronomy. This is an argument about work we can organize to benefit from additional resources, not a claim that every task scales with token count. Cheaper tokens at comparable capability are a premise for the outlook, not a guaranteed price trajectory. Both versions use “fixed capabilities”; only agentic/computational and tokens/computation change. The next slides make the engineering constraints concrete; adoption remains a personal choice. Cail’s spoken analogy: computing transformed existing subdomains of physics and astronomy, and new subdomains emerged from the new constraints. Keep this explanation off the slide. The following slide introduces emerging practices and the invitation to explore them.

A roadmap for the rest of the talk

01Emerging building blocks

02A live demo: a small scientific analysis

03High-level strategies

• Do you want to work this way?
• If you do, which parts do you want to scale?

The remaining talk has three parts: emerging low-level concepts, a live agentic analysis, and higher-level ways of organizing research. The purpose is informed choice, not a demand to adopt these tools. Give people a window into practical experience, then enough conceptual grounding to reason about their own practice: whether to engage, which labor to scale, and which activities to keep doing themselves. Both choices are legitimate.

What engineering constraints emerge from these systems?

Languages

Storage (memory, disk)

Tests / CI

Control flow (loops, parallelization, if/else)

IDEs

Sources & notes

Official logos: Fortran, Python, Julia (Stefan Karpinski, CC BY-NC-SA 4.0), C++, Jupyter, VS Code, Vim, Emacs.

This single slide replaces the previous four-slide computing/history/constraints sequence. The left names established software-engineering concepts. Their paths mix and dissolve into a shared fog: the new categories and their relationships are not yet settled. Do not label the right side or imply a one-to-one mapping. The coming primitives provide concrete partial answers: intent and specifications, harnesses, context and durable memory, verification, and orchestration. Computing transformed existing fields and created new ones. The optimistic opportunity is both more ambitious science and room to invent a research practice. Keep that spoken rather than adding another slogan. Adoption is an informed choice.

From oracles to agents

Sources & notes

Earendil: What is a Harness? | Claude, Codex, Pi, OpenCode

The manual clipboard loop stays visible above; on advance, the harness takes over tool execution and returning results. Keep the visible loop minimal: the harness dispatches read/edit/bash operations and returns their results. System prompts and model API translation are explained in the linked reading. The three tools are examples, not an exhaustive inventory. The linked reading covers the fuller architecture.

Harnesses are working environments, like IDEs

IDEs

Harnesses

Minimal and extensible ⟷ integrated
Environment: files, shell, remote machines, browser

Sources & notes

Claude, Codex, Pi, OpenCode

The two loose logo groups compare kinds of working environments, without pairing individual products or implying a feature matrix. Features vary by product, interface, extension and version. The spectrum is between a minimal core that you extend and an integrated environment with more capabilities already present. SSH, forwarding and browsers are examples of what the surrounding environment may provide. Avoid suggesting every pictured harness ships all three features.

Context is similar to RAM

Standard context length: 1M tokens

Context is a finite working set, analogous to RAM. Show the session’s accumulating sequence: system prompt, user messages, model responses and tool results. Empty capacity stays unlabeled. First advance: a read tool returns file contents, occupying context. The malloc analogy is about consuming a finite working budget, not literal allocation: tokens are not addressable RAM, and reading generally appends content rather than reserving an independently freeable block. Second advance: compaction summarizes the history into a fresh model context after the system prompt. This can continue the same user-facing session; it need not create a new chat. The drawing is schematic: implementations may preserve recent turns and tool state as well. Summarization is lossy. Keep consequential decisions and evidence in durable files.

Cache rules everything around me

Sources & notes

OpenAI: prompt caching (animation adapted from “What is the prompt cache?”); OpenAI pricing; Claude pricing; Thariq: Prompt Caching Is Everything.

Rates checked September 17, 2026. Astra and Fable 5.1 both have $10/M base input, $12.50/M short-cache writes and $50/M output; cache reads are $1/M and $0.25/M respectively. Astra: standard short-context API pricing. Fable: five-minute cache writes; one-hour writes cost $20/M. Read multipliers are 0.1× and 0.025× respectively; output 5×. Prices exclude batch, fast mode, regional uplifts and other modifiers.

Illustrative Astra short-context accounting: Turn 1 writes a 100k-token prefix ($1.25) and generates 20k output tokens ($1). Turn 2 hits the original 100k ($0.10), writes the previous 20k-token answer ($0.25) plus a 4k-token user follow-up ($0.05), and generates another 20k ($1). Turn 5 continues the same pattern: its cached prefix is 100k + 3 × 24k = 172k tokens ($0.172), followed by the latest 20k-token assistant answer ($0.25 write), a 4k-token user addition ($0.05 write), and 20k output ($1). Turns 3–4 are omitted; each adds 24k tokens to the next cached prefix. All bars share a linear scale of 6.02 SVG units per 1,000 tokens. The initial prefix consists of 96k instructions/tool tokens and a 4k user message; arrows are fixed separators, not tokens. The example assumes the indicated cache hits; actual reusable boundaries depend on the service. Fable 5.1 read costs are one quarter of these Astra read costs. KV states, not a ready-made answer, are cached.

Retention is model-specific: Astra’s documented minimum is 30 minutes after write/reuse; Claude offers five-minute and one-hour writes. Older OpenAI in-memory caches typically last 5–10 minutes of inactivity, up to one hour.

Stay at conversation-turn level. First show instructions, tools and the user’s message being processed, followed by an answer. Advance: retain that prefix, append the answer and follow-up, and generate another answer. Cache reuse avoids reprocessing the matched history; new history and output still cost money. The boundary is schematic, not a promise that output is never cached. Final reveal gives representative multipliers relative to $10/M base input; exact per-model rates stay in Sources & notes. KV mechanics are supporting detail, not the on-slide explanation.

Benchmarks aren’t everything

…but it’s important to think about cost versus performance.

Sources & notes

Artificial Analysis, Intelligence Index vs. Cost per Intelligence Index Task. Chart downloaded September 17, 2026. Original full-resolution export; the chart retains its attribution and benchmark definitions.

Plugins allow you to customize your harness

Skills load instructions and references on demand. The visualization directory is an illustrative example: read 1d.md or 2d.md when relevant, run colorblindcheck.py, and use the colormap asset. Hooks execute configured code at lifecycle events, for example injecting Git divergence status at session start, blocking a destructive action before a tool call, or linting after a file edit. Event names vary by harness. Plugins package these customizations.

Delegate to sub-agents for scoped work

Start with user → main agent → tools. Advance adds a bounded worker with its own context. Examples include literature search, code investigation, review, and implementation. The main session carries the relationship with the user and integrates the result. Delegation is not necessarily parallel, and a returned summary is not proof of correctness. Separate context does not always mean a fork: Claude Code supports both full-history forks and fresh-context agents. This Codex session can inherit all, some, or none of the conversation. Two use cases: an independent side investigation, or a conversational lead delegating implementation and exploration. Model-cost allocation belongs on the following teams slide.

Teams coordinate while the work is happening

Message motion illustrates ongoing communication, not real telemetry. The lead can clarify scope, interrupt, redirect, and integrate emerging results; some systems also support peer messaging. This differs from fire-and-forget delegation. Shared goals, ownership, and communication discipline are necessary: more messages do not automatically mean better coordination. Preserve the user’s ability to steer. A strong, expensive lead can concentrate judgment in design and decomposition, giving lower-cost workers sufficiently clear task parameters. Model choice should follow task difficulty: review and ambiguous research can still need frontier intelligence. This is a useful allocation pattern, not a guarantee of lower total cost.

Workflows express loops and parallelism on the fly

Fan out candidate approaches and plan from their results. Implement the selected approach, review it, and iterate between implementation and review. The orchestration can be generated for the task and inspected as code.

With great output comes great chaos

The slop cannon is a joke about unorganized throughput, not a claim that every generated artifact is worthless. Growing work can mix incompatible inputs, configurations, and outputs. Naming files is useful but insufficient: record which inputs, decisions, recipes, and outputs belong to each analysis. This motivates ASTRA without claiming a schema eliminates every provenance error. Distinguish remembering why a decision was made from validating that the current run used it.

Lightcone Research

An open ecosystem for science
lightconeresearch.org

ASTRA

Specification + validator

The scientific choices behind each result

A community standard (hopefully)

Lightcone tools

Agentic execution on that foundation

Execute the specified analysis

Inspect the resulting artifacts

CNRS AISSAI and BIDS / UC Berkeley

Sources & notes

Team & institutions · Projects & architecture · ASTRA specification

ASTRA specifies the analysis; the agent implements it. ASTRA is a specification and validator, while Lightcone tools execute the analysis and expose its artifacts. Based at CNRS AISSAI and BIDS / UC Berkeley. Use the actual tutorial specifications to introduce the ASTRA structure; do not show a flattened top-level recipe sketch.

Live Demo: Measure Dark Energy in 15 Minutes

Supernovae are standard candles:

ΩΛ

Type Ia supernovae are standardizable candles. Their calibrated luminosity and apparent brightness give luminosity distance. Spectra give redshift; the distance–redshift relation constrains expansion history. The candles stay fixed as the model prediction changes. The tutorial uses actual Union2.1 measurements to fit Omega_Lambda.

Give your agent Lightcone tools

If you don’t have uv (Python package and project manager) installed:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

Install lightcone-cli:

uv tool install lightcone-cli==0.5.0rc2
mkdir -p /tmp/lightcone-demo
cd /tmp/lightcone-demo

Claude Code

claude plugin marketplace add LightconeResearch/agent-skills
claude plugin install lightcone@lightcone-research

Codex

codex plugin marketplace add LightconeResearch/agent-skills
codex plugin add lightcone@lightcone-research

Sources & notes

uv installer · Lightcone plugins

The CLI block creates and enters /tmp/lightcone-demo, reusing it if it already exists. Run the setup and your chosen agent in the same terminal. Copy out results you want to keep before the operating system clears temporary files. The uv installer and CLI use their standard user-level installation locations. The CLI includes astra-tools; the Lightcone plugin bundles ASTRA guidance and hooks. Start a fresh agent session after installation. Create the analysis with prompt 1 before opening JupyterLab.

1. Kick off an analysis with Lightcone

I’d like to use Lightcone tools to analyze the Union2.1 supernova compilation from the Supernova Cosmology Project. Let’s write an ASTRA analysis that fits a flat LCDM model to this data and recovers Omega_Lambda.

Please produce two outputs: the best fit Omega_Lambda with uncertainties, and a downstream Hubble diagram figure with an absolute panel and a residuals panel. As you go, briefly walk me through each ASTRA element that you add; this is a live tutorial! Don’t add any decisions yet.

Union2.1 catalogue ↗

while it runs — an illustrative spec

inputs:
  - id: union21
    type: data
    source: data/union21.txt

outputs:
  - id: best_fit
    type: metric
    inputs: [union21]
    recipe:
      command: python fit.py …
  - id: hubble_diagram
    type: figure
    inputs: [union21, best_fit]
    recipe:
      command: python plot.py …

Step one: say what the analysis is, in words, and the agent writes the spec. Inputs: the data file. Outputs: a best-fit — Omega Lambda with uncertainty, chi-squared — and a Hubble-diagram figure downstream of it. Each output carries a recipe: the exact command that produces it, with the dependencies explicit. Notice what this is: it’s a DAG again! — but look who drew it. You said four sentences of intent; the model wrote the graph, and a hook validated the file the moment it was saved. The graph didn’t disappear from science — provenance still wants a graph — it moved from being the thing you hand-wire to being an artifact the agent maintains.

View the analysis in JupyterLab

uv venv .venv
source .venv/bin/activate
uv pip install jupyterlab nodejs-wheel \
  'jupyterlab-lightcone[ai]==0.0.6'
npm install --prefix "$VIRTUAL_ENV" \
  @agentclientprotocol/claude-agent-acp \
  @agentclientprotocol/codex-acp
export PATH="$VIRTUAL_ENV/node_modules/.bin:$PATH"
jupyter lab

Sources & notes

Jupyter AI setup · Node and npm package

Optional viewer and chat setup for macOS/Linux. The project already exists when the viewer opens. Keep analysis files in the project directory and installed packages in its .venv subfolder. If prompt 1 created a project subdirectory, enter that directory first. These commands do not create a second temporary workspace. Both Claude Code and Codex ACP adapters are installed; continue using the terminal agent if preferred.

The tutorial sequence is agent setup → prompt 1 creates the analysis → open the project in JupyterLab → inspect outputs → prompt 2 → prompt 3. Jupyter is a viewer and optional chat surface, not a prerequisite for starting the analysis. Launch it from the actual ASTRA project root.

2. Did we reproduce the published result?

The paper that published this catalogue is Suzuki et al. 2012, arXiv 1105.3470. Can we compare our Omega_Lambda to their supernova-only constraint?

while it runs — what a verified claim looks like

prior_insights:
  suzuki_2012_result:
    claim: >
      [constraint from the paper]
      [assumptions + errors]
    evidence:
      - doi: >-
          10.48550/arXiv.1105.3470
        quote:
          exact: >
            [exact quote]
        location:
          value: "page=17"

Step four: check it against the paper that published this catalogue — Suzuki et al. 2012. And here’s my favorite mechanism in the whole system. The agent caches the paper and records the published constraint as a prior insight — a claim with evidence. The evidence is an exact quote, with a page number — and ASTRA searches the cached PDF and verifies that the quote actually appears on page seventeen. This matters because agents, like graduate students, occasionally remember papers saying things they don’t say. Claims about the literature are grounded in the literature, mechanically. And with the quote verified, we can compare: they got 0.705, plus 0.040 minus 0.043. Our central value agrees nicely. But look at the error bars — ours are three times tighter, on the same 580 supernovae. Same data, same model. Something is wrong, and it’s wrong in the direction that should scare you: we’re too confident.

3. What changes when we include systematics?

Let’s improve our systematics treatment to reproduce their published result. Track all changes as ASTRA decisions.

Choose the covariance treatment

decisions:
  covariance:
    label: Covariance treatment
    rationale: Account for systematics
    default: full
    options:
      stat_only:
        label: Statistical only
      full:
        label: Statistical + systematic

outputs:
  best_fit:
    decisions: [covariance]
    # recipe uses the selected option

Step five, and the payoff. We ask the agent to find out why: download the two Union2.1 covariance files — with and without systematics — work out which one our error column corresponds to, and record a decision in the spec: statistical-only versus full systematic covariance. The answer: our fit had been using the no-systematics errors — the error column in the data file is the diagonal of the no-systematics covariance, to twelve decimal places. Everything the systematic covariance knows was sitting in a file we had never opened. Switch to it, and watch the plot: the central value barely moves — 0.722 to 0.714 — but the uncertainty more than doubles, and only then does our measurement actually lie beside the published one: 0.2 sigma apart, their data, their systematics, their method. Here’s the lesson, and it’s the sentence I’d have you take home: if you had ranked this choice by its effect on the best-fit number, it would have looked nearly irrelevant. It is the difference between a number that resembles theirs and a number you can put beside theirs. That is what a decision is — and this field, all of science, is made of thousands of them, almost none of them written down. ASTRA writes them down.

Just talk to the model!

Convey intent rather than commands.

Voice / transcription is faster and gives the model a richer picture of your intent.

Meeting transcripts are a goldmine.

Open the post-tutorial discussion with conversation as a practical way to communicate intent. Voice and transcription make it easier to share the context, uncertainty, and motivations that can be lost when compressing a request into a short command.

Have the model write everything down

Cail’s cross-project example: in multi-probe cosmology, extending bins to low multipoles addressed mode-coupling bias from contributions not represented by the truncated coupling matrix. That fix also informed the cosmic shear project. The displayed snippets are condensed illustrations of this finding, not verbatim quotes from the project records. Reveal the second project, then the request to retrieve and apply the earlier fix. No blinded numerical results.

Iterate on long-horizon work

“Check your work” becomes useful when the task and environment provide ways to evaluate it. A workflow can separate implementation and review contexts. Separate contexts do not guarantee unbiased judgment: models may still share systematic blind spots.

Ask for HTML reports

A report layout, not measured results. Request a real executive summary, plots side by side, verification evidence, and open questions or decisions needed. The report should link to actual artifacts and make failed or incomplete checks visible. Review the results rather than trusting presentation polish.

Do you still need to read code?

Sometimes!

• Define the high-level design before implementation
  (product requirements document, etc.)
• Grill the model until you understand what it does.
  Diagrams help.
• Core code is worth reading.

Understand what the implementation does at a high level. Establish the design before delegating implementation; then question the model until its explanation makes sense. Diagrams can help. Read core code directly, especially fundamental architecture, scientific assumptions, and shared primitives. Model explanations remain something to check against the source and behavior.

Becoming a project manager

• Agents are like brilliant but amnesiac interns
• Agentic work pushes you towards doing many things at once
• You can only keep so many chats in your head at once...

Sources & notes

OpenAI: An open-source spec for Codex orchestration: Symphony. Video embedded from OpenAI’s Vimeo player; requires an internet connection.

Which kinds of scientific labor scale under these constraints?

Not all problems are equally scalable.

The distinction is the bottleneck: additional machine effort can directly advance some problems; others depend on obtaining data, building instruments, or sustained human interaction. These are not permanent categories, nor a ranking of scientific worth. A project may contain both. This connects the computational-tractability discussion to a personal choice about work, without predicting a safe niche or prescribing adoption. The following DeepSeek quotation provides one person’s response to changing work, rather than a universal answer.

Closing thoughts

The practice of research is being revolutionized.

If we change how we work, we can hold higher standards
and take on more ambitious problems.

“Of course I would rather not be swept away by the revolution.
But if I have to be, then I would rather be the one who revolutionizes myself.”

A DeepSeek kernel engineer (English translation, excerpt)

Sources & notes

English translation posted by Teortaxes. Translation credited in the post to Astra-xhigh. Quote wording supplied by Cail. Original author identity has not been independently verified. Lightcone tutorial and agent skills.

Combine the final quotation and closing slide. Changing practice offers the possibility of higher standards and more ambitious work. The quotation is one personal response, not a demand to adopt these tools. Keep the audience’s choice central.