LLM Structured Control Research

This repository contains the code, configuration, experiments, reports, tables, figures, and paper sources for a study on where large language models are useful in structured and constrained control problems.

The project evaluates LLMs in a simulated greenhouse grow-box control task. The main comparison is between using an LLM as a direct low-level actuator controller and using it as a higher-level component for supervision, explanation, and goal interpretation.

• Repository: https://github.com/tahsinkoc/llm-structured-control
• Paper record: Structured Outputs Are Not Controllers: Evaluating LLM Roles in Constrained Greenhouse Control
• Reproducibility guide: REPRODUCIBILITY.md

Research Question

In structured, low-entropy, constrained control problems, should LLMs be used as direct controllers, or are they more valuable as high-level components that supervise classical controllers, explain anomalies, and interpret human goals?

Main Claim

LLMs should not be the default choice for low-level actuator decisions in this kind of control problem. Speed, determinism, latency, cost, parseability, and safety constraints make classical controllers strong baselines.

The more rational role for an LLM is around the control loop: goal interpretation, anomaly explanation, parameter suggestions, operator reporting, and supervisor or meta-controller behavior. Even in those roles, LLM outputs must be validated, gated, and audited before they affect actuation.

Hypotheses

H1. For low-level deterministic control decisions, rule-based and classical ML methods will be faster, cheaper, more stable, and safer than a direct LLM controller.

H2. The most useful LLM role will be supervision or meta-control, not direct control.

H3. Structured output techniques will reduce invalid JSON and schema failures, but validators and safety shields will still be required for semantic decision quality and safety.

H4. Under sensor noise, missing data, threshold-adjacent states, and OOD weather, direct LLM failures will differ from classical baseline failures. These differences are measured with a failure taxonomy.

H5. As human goals become more ambiguous, the added value of an LLM supervisor may increase, but this value must be measured separately from low-level actuator control.

Current Findings

The experiments so far support the central caution: valid structured output is not the same thing as correct control.

• Direct Mistral control did not outperform classical baselines.
• Structured output reduced JSON and schema failures in later prompts.
• Semantic control quality remained limited even when output formatting was valid.
• Supervisor and goal-interpretation results were more promising than direct actuation, but did not beat calibrated or keyword baselines in the current benchmark.
• Hybrid improvements came mainly from calibrating the classical controller, not from the LLM supervisor signal.

Minimum Success Criteria

• Working grow-box simulator.
• At least five controllers: rule-based, rule+hysteresis, random forest, direct LLM, and LLM+validator+retry.
• At least five test sets: normal, boundary, noisy, missing-sensor, and OOD weather.
• Static decision metrics and closed-loop metrics.
• LLM reliability metrics: invalid output, schema failure, semantic failure, fallback use, retry rate, latency, and cost.
• Reproducible configuration, seeds, logs, and report outputs.

Repository Layout

configs/          Default experiment configuration.
data/             Raw and processed datasets.
docs/             Day-by-day planning notes and experiment notes.
experiments/      LLM logs and experiment traces.
paper/            Venue-neutral LaTeX paper draft.
reports/          Research reports, generated tables, and figures.
scripts/          Dataset generation, evaluation, and figure scripts.
src/              Simulator, controllers, supervisor, safety, and metrics code.
tests/            Unit tests for simulator, controllers, safety, and evaluation.

Key Files

• benchmark_spec.en.md: frozen benchmark decisions, controller interface, test sets, and metrics.
• REPRODUCIBILITY.md: setup and reproduction commands.
• references.en.md: literature-reading guide.
• references.bib: BibTeX database used by the paper draft.
• paper/main.tex: venue-neutral LaTeX paper source.
• reports/final_research_synthesis.en.md: final hypothesis-level synthesis.
• reports/paper_draft.md: Markdown paper draft.
• reports/paper_appendix.md: appendix material for the H5 rubric, risk-flag examples, and action-impact analysis.

Reproducing the Results

Create a virtual environment and install dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run the tests:

pytest

Rebuild non-LLM artifacts:

python3 scripts/generate_expert_datasets.py
python3 scripts/evaluate_baselines.py
python3 scripts/generate_paper_figures.py

Paid LLM experiment commands are documented in REPRODUCIBILITY.md. API keys should be passed through environment variables and never committed.

Reports

• reports/baseline_results.en.md: classical controller baseline results.
• reports/llm_protocol_results.en.md: direct LLM, validator, and prompt-ablation results.
• reports/supervisor_results.en.md: supervisor, hybrid, risk-audit, and calibrated-gate results.
• reports/h5_goal_interpretation_results.en.md: ambiguous human-goal benchmark and H5 sanity results.
• reports/tables/hypothesis_evidence_summary.csv: hypothesis decision table.
• reports/tables/final_key_metrics.csv: main metric summary.

Figures

• reports/figures/paper_architecture.png: paper architecture diagram.
• reports/figures/controller_static_exact_match_comparison_with_ablation.png: static exact-match comparison.
• reports/figures/supervisor_risk_flag_audit.png: supervisor v2 risk-flag audit.
• reports/figures/h5_goal_interpretation_scores.png: H5 strict, rubric, and clarification scores.