Does Scaffolding Substitute for Scale?

An empirical study of how much of an AI model's measured capability on real occupational tasks comes from model scale versus the scaffolding around the model, and what that means for estimates of AI's labor-market exposure.

The question

Most estimates of "AI exposure" by occupation test a model zero-shot: one prompt, one response. But deployed AI systems are scaffolded (structured prompts, planning, tool use, self-critique). This study measures how much measured task capability changes as scaffolding increases, comparing a small, cheap open-weights model against a large, costly one, and asks a sharper question than the usual frontier-vs-open comparison:

Can cheap scaffolding on a small model substitute for a bigger, costlier model run plainly?

If a well-scaffolded 7B model matches a zero-shot 70B model on real occupational tasks, that has direct implications for who can deploy AI against labor tasks, and at what cost. This is the deploy-time economic choice a real organization faces: pay more per call for a bigger model, or spend cheap scaffolding on a smaller one.

Design

• Tasks: real occupational task statements from O*NET, converted into work-product prompts, across 6 occupations spanning wage and skill levels.
• Two model sizes so findings are not a single-model artifact: a small cheap model (Qwen2.5 7B) and a large costly one (Llama 3.3 70B), both served serverless via a hosted open-weights API. They are different families; this is disclosed as a limitation (the small-vs-large gap blends scale with training recipe) and makes the comparison a realistic cheap-vs-costly deploy choice rather than a pure same-family scale sweep.
• Four harness conditions:
  • H0 zero-shot single call
  • H1 structured prompt (role, context, explicit output spec)
  • H2 plan-then-execute (decomposition / chain-of-thought)
  • H3 tool-augmented multi-step with a self-critique revision loop
• 3 runs per cell at fixed temperature; report means with confidence intervals.
• Grading: a 1-5 rubric per task, scored by an LLM judge, with a hand-graded 15-20% subsample and reported judge-human agreement.
• Economic join: weight task-level capability by BLS OEWS wages and employment to produce a harness-adjusted exposure estimate per occupation.

Honest scope

This uses open-weights models, not the current frontier. Absolute exposure levels here are therefore a floor, not an estimate of frontier capability. The contribution is the shape of the relationship between scaffolding, scale, and measured exposure, not a headline exposure number.

Reproduce

pip install -r requirements.txt
python -m src.pipeline --stage all      # see src/ for individual stages

Set TOGETHER_API_KEY (or GROQ_API_KEY) in your environment first. All model calls are cached, so reruns are free and deterministic.

Status

Work in progress. See docs/ for the pre-registered hypotheses (written before the full run) and the limitations section.