diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..cde8704 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,29 @@ +name: CI + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + test: + runs-on: ubuntu-latest + + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install test dependencies + run: | + python -m pip install --upgrade pip + python -m pip install -r requirements-test.txt + + - name: Run repository tests + run: pytest -q diff --git a/README.md b/README.md index 8cedff2..0197609 100644 --- a/README.md +++ b/README.md @@ -1,97 +1,194 @@ # Indexical Circuits -## Mechanistic Sociolinguistics of LLM Epistemic Deference +## A replication-first study of sociolinguistic cues and epistemic judgment in language models -**Indexical Circuits** investigates whether socially meaningful linguistic -constructions systematically alter an LLM's epistemic judgments — and, if a -robust behavioral effect can first be established, whether the effect can be -traced to causal internal model mechanisms. +**Model:** `google/gemma-2-2b-it` -The project combines sociolinguistic analysis with controlled behavioral -experimentation and a pre-specified evidence gate for mechanistic -interpretability. +**Study status: behavioral replication sequence complete; mechanistic gate closed.** -**Model:** `google/gemma-2-2b-it` +Indexical Circuits asks whether socially meaningful linguistic +constructions systematically alter a language model's epistemic judgments +when factual content is held constant. + +The project was designed with a pre-specified evidence gate: + +**mechanistic localization would begin only if the behavioral effect first +replicated robustly.** + +That condition was not met. + +The project therefore stops before making claims about an "authority +circuit", an "indexical circuit", or another causal internal mechanism. + +--- + +## Executive summary + +An exploratory pilot suggested that formal linguistic framing strongly +increased Gemma-2-2B-IT's agreement with false claims. + +That apparent effect survived an important measurement-interface +correction, motivating stronger tests. + +But the effect did not survive increasingly controlled independent +replication. + +The research sequence was: + +**Pilot** + +→ **Measurement audit** + +→ **Broad confirmatory null** + +→ **Exploratory linguistic phenotyping** + +→ **Held-out role-relational null** + +→ **Exploratory `functions as` candidate** -**Current stage:** behavioral replication sequence completed. The final -preregistered primary hypothesis was not supported, so the pre-specified -mechanistic gate remains closed. +→ **Preregistered focused replication** + +→ **Primary null / mechanistic gate closed** + +The strongest methodological conclusion is: + +> Before explaining an apparent model behavior mechanistically, first +> establish that the behavior itself is reproducible. + +The project preserves the complete path from apparently strong signal to +falsification rather than selecting only the positive stages. --- -## Research Question +## Key results -Can the linguistic form of a proposition change an LLM's willingness to -endorse it even when the underlying factual claim is held constant? +| Stage | Primary result | +| --- | --- | +| Initial exploratory pilot | False-claim formal-minus-plain effect: `+2.4574` | +| Measurement audit | A/B probability mass >99% after correcting the interface | +| Broad confirmatory experiment | `t(59) = 0.7996`, `p = .4272` | +| Held-out role-relational replication | `t(59) = 0.3734`, `p = .7102` | +| Focused preregistered `functions as` replication | `t(29) = -0.9101`, `p = .3703` | +| Mechanistic evidence gate | **Closed** | +| Secondary truth-dependent interaction | `t(29) = 3.3542`, `p = .00223`; requires independent replication | -The project originally tested whether **formal register** generally increases -epistemic deference. +The original hypothesis that linguistic formality reliably increases +agreement with false claims was not supported. -A stronger confirmatory experiment did not support that broad hypothesis. +The later role-relational hypothesis also failed held-out replication. -Subsequent exploratory analyses identified progressively narrower candidates, -first **role-relational framing** and later the specific construction -**`functions as`**. Both were subjected to new held-out testing. +The focused `functions as` hypothesis likewise failed its preregistered +primary test. -Neither produced a replicated positive effect on agreement with false claims. +A secondary truth-dependent interaction appeared in the final focused +study, but it was driven primarily by **reduced agreement with true +claims**, not by reliably increased agreement with false claims. -The final focused replication instead revealed an unexpected truth-dependent -pattern: `functions as` reliably reduced agreement with true claims while -showing no reliable increase in agreement with false claims. +That secondary finding is reported as hypothesis-generating rather than +as a rescue of the original theory. --- -## Current Empirical Status +## Effect trajectory + +The apparent false-claim effect became progressively weaker as the tests +became more independent and more tightly controlled. + +![Effect trajectory](assets/figures/figure_1_effect_trajectory_PORTFOLIO.png) + +This trajectory is central to the project. + +A strong-looking pilot effect was not sufficient evidence for a stable +behavioral phenomenon. + +--- + +## Research question + +Can the linguistic form of a proposition alter a language model's +willingness to endorse it even when the underlying factual claim is held +constant? + +The project began with a broad hypothesis: + +**formal register may increase epistemic deference.** + +After that hypothesis failed confirmatory testing, exploratory linguistic +phenotyping asked whether the effect might instead depend on more specific +sociolinguistic structures such as: + +- technical lexicality; +- role and status framing; +- relational reframing; +- taxonomic framing; +- syntactic restructuring; +- nominalization; +- semantic specificity. + +This produced narrower candidate hypotheses, each of which was then +tested using new held-out material rather than being treated as confirmed +from the exploratory data that generated it. + +--- -### 1. Initial pilot +## 1. Exploratory pilot -A 20-item exploratory pilot produced a substantial apparent register effect for -false claims. +A 20-item exploratory pilot produced a substantial apparent register +effect on false claims. False-claim formal-minus-plain effect: `+2.4574` -This initial signal motivated a stronger experimental design. +The size of the apparent signal justified a more careful measurement +audit before scaling the experiment. --- -### 2. Measurement audit +## 2. Measurement-interface audit + +The initial experiment used raw A/B next-token scoring. + +A diagnostic audit showed that this was not an appropriate measurement +interface for Gemma-2-2B-IT: bare A/B tokens received negligible +probability mass without the conversational formatting expected by the +instruction-tuned model. -A diagnostic analysis showed that raw A/B next-token scoring was inappropriate -for Gemma because bare A/B tokens received negligible probability mass without -the model's expected conversational formatting. +The experiment was therefore rebuilt using: -The experiment was therefore rebuilt using Gemma's official chat template and -AB/BA label-order counterbalancing. +- Gemma's official chat template; +- AB/BA response-label counterbalancing. -Under the corrected interface, A/B jointly captured more than 99% of -next-token probability. +Under the corrected interface, the A/B alternatives jointly captured +more than 99% of next-token probability. -The original pilot signal survived this interface correction, justifying a -larger confirmatory experiment. +Importantly, the original pilot signal survived this correction. + +The project therefore did not dismiss the pilot as a measurement artifact; +it proceeded to an independent confirmatory experiment. --- -### 3. Broad confirmatory experiment +## 3. Broad confirmatory experiment -A new experiment used: +The confirmatory experiment used: -- 60 factual families -- 240 stimuli -- true and false claims -- plain and formal realizations -- AB/BA counterbalanced scoring +- 60 factual families; +- 240 unique stimuli; +- true and false claims; +- plain and formal realizations; +- AB/BA counterbalanced scoring. -The broad formal-register hypothesis did **not** replicate. +The broad formal-register hypothesis did not replicate. -Condition means showed: +Observed effects: -- false formal-minus-plain effect: `+0.3376` -- true formal-minus-plain effect: `+0.0179` -- false-minus-true interaction: `+0.3197` +- false formal-minus-plain: `+0.3376`; +- true formal-minus-plain: `+0.0179`; +- false-minus-true interaction: `+0.3197`. -Primary statistical test: +Primary test: `t(59) = 0.7996, p = .4272` @@ -104,447 +201,461 @@ deference-inducing feature. --- -### 4. Exploratory linguistic phenotyping - -The heterogeneous register transformations were subsequently coded for -linguistic properties including: +## 4. Exploratory linguistic phenotyping -- technical lexicality -- role/status framing -- relation reframing -- taxonomic framing -- syntactic restructuring -- nominalization -- semantic specificity +The heterogeneous register transformations were coded for more specific +linguistic properties. -Technical vocabulary showed essentially no association with the false-claim -effect: +Technical lexicality showed essentially no association with the +false-claim effect: `technical lexicon present - absent = -0.070` -The strongest descriptive pattern instead occurred when **role/status framing -and relation reframing occurred together**. +The strongest descriptive pattern instead appeared when role/status +framing and relation reframing occurred together. -Examples included constructions such as: +Examples included: -- `serves as` -- `functions as` -- `qualifies as` -- `ranks as` -- `is the author of` +- `serves as`; +- `functions as`; +- `qualifies as`; +- `ranks as`; +- `is the author of`. -For the six exploratory role-relational cases: +For six exploratory role-relational cases: -- mean effect: `+3.145` -- median effect: `+1.412` -- positive cases: `6/6` +- mean effect: `+3.145`; +- median effect: `+1.412`; +- positive cases: `6/6`. After removing the two largest effects: -- mean effect: `+0.840` -- positive cases: `4/4` +- mean effect: `+0.840`; +- positive cases: `4/4`. -This pattern was treated explicitly as **exploratory**, not as a confirmed -effect. +This pattern was explicitly treated as exploratory. + +It generated a new hypothesis rather than being presented as evidence +that the hypothesis was already true. --- -### 5. Held-out role-relational replication +## 5. Held-out role-relational replication -A completely new 60-family held-out bank was constructed to test the -role-relational hypothesis. +A completely new 60-family stimulus bank was constructed. Each factual family contained six conditions: -1. true plain direct -2. true formal control -3. true role-relational -4. false plain direct -5. false formal control -6. false role-relational +1. true plain direct; +2. true formal control; +3. true role-relational; +4. false plain direct; +5. false formal control; +6. false role-relational. This produced: -- 60 factual families -- 360 unique stimuli -- 720 model evaluations after AB/BA counterbalancing +- 60 factual families; +- 360 unique stimuli; +- 720 model evaluations after AB/BA counterbalancing. + +Before model testing, the frozen bank was audited against previous +experimental material. -The frozen bank was audited against previous experimental materials before -model testing: +The audit found: -- exact prior overlaps: `0` -- prior similarities >= .90: `0` -- formal-control role contamination: `0` -- missing role markers: `0` +- exact previous overlaps: `0`; +- previous similarities >= `.90`: `0`; +- formal-control role contamination: `0`; +- missing role markers: `0`. The preregistered primary comparison was: `false_role_relational - false_formal_control` -The result was: +Observed result: -- mean effect: `+0.0904` -- median: `+0.1836` -- positive rate: `58.3%` -- Cohen's dz: `0.048` -- `t(59) = 0.3734` -- `p = .7102` -- bootstrap 95% CI: `[-0.3928, 0.5546]` +- mean effect: `+0.0904`; +- median: `+0.1836`; +- positive rate: `58.3%`; +- Cohen's dz: `0.048`; +- `t(59) = 0.3734`; +- `p = .7102`; +- bootstrap 95% CI: `[-0.3928, 0.5546]`. -The broad role-relational hypothesis was therefore **not supported**. +The broad role-relational hypothesis was not supported. --- -### 6. Construction-level heterogeneity +## 6. Construction-level heterogeneity Exploratory decomposition of the held-out results showed substantial construction-specific heterogeneity. -The most promising apparent pattern occurred for the `functions as` -construction. +![Construction-level heterogeneity](assets/figures/figure_2_construction_heterogeneity_PORTFOLIO.png) + +The most promising apparent pattern occurred for the specific +construction: + +`functions as` For the 10 exploratory `functions as` cases: -- false role-minus-formal effect: `+1.080` -- false positive rate: `8/10` -- true role-minus-formal effect: `-0.655` -- true positive rate: `2/10` -- false-minus-true interaction: `+1.735` +- false role-minus-formal effect: `+1.080`; +- false positive rate: `8/10`; +- true role-minus-formal effect: `-0.655`; +- true positive rate: `2/10`; +- false-minus-true interaction: `+1.735`. -This appeared to fit the hypothesized pattern particularly well: the -construction seemed to increase agreement with false propositions while -moving true propositions in the opposite direction. +Because this pattern was discovered after inspecting construction-level +results, it was treated as a new exploratory finding. -Because this pattern was identified **after inspecting construction-level -results**, it was treated as a new exploratory finding requiring an entirely -new confirmatory replication. +A new confirmatory dataset was required. --- -### 7. Focused preregistered `functions as` replication +## 7. Focused preregistered `functions as` replication -A final focused replication was preregistered before model testing. +The final study used a new preregistered focused replication. -The design used: +Design: -- 30 completely new factual families -- 4 conditions per family -- 120 unique stimuli -- AB/BA counterbalanced scoring -- 240 total model evaluations +- 30 completely new factual families; +- 4 conditions per family; +- 120 unique stimuli; +- AB/BA counterbalanced scoring; +- 240 total model evaluations. -Conditions were: +Conditions: -1. `true_formal_control` -2. `true_functions_as` -3. `false_formal_control` -4. `false_functions_as` +1. `true_formal_control`; +2. `true_functions_as`; +3. `false_formal_control`; +4. `false_functions_as`. The primary hypothesis was: `false_functions_as - false_formal_control > 0` -The crucial truth-dependent contrast was: +The truth-dependent contrast was: `(false_functions_as - false_formal_control) - (true_functions_as - true_formal_control)` ### Stimulus provenance Before model testing, the focused bank was checked against all previous -experimental materials. +experimental material. The final audit found: -- previous stimulus texts checked: `640` -- exact overlaps: `0` -- similarities >= .90: `0` -- reused subject families: `0` +- previous stimulus texts checked: `640`; +- exact overlaps: `0`; +- similarities >= `.90`: `0`; +- reused subject families: `0`. -The 30 families were source-checked before model testing and the final bank was -frozen. +The 30 families were also source-checked before model testing. -Its SHA-256 fingerprint is: +The frozen dataset fingerprint is: `64b3fd0cc9d0cba4f650789fc0a6e7b35787352fa17b15a5592cf885db48a14e` -The fingerprint allows the exact model input to be verified independently. - -### Primary focused result +### Primary result -The preregistered primary hypothesis was **not supported**. +The preregistered primary hypothesis was not supported. False `functions as` minus formal-control effect: -- mean: `-0.0666` -- median: `-0.0996` -- positive rate: `36.7%` -- Cohen's dz: `-0.166` -- `t(29) = -0.9101` -- `p = .3703` -- bootstrap 95% CI: `[-0.2109, 0.0717]` +- mean: `-0.0666`; +- median: `-0.0996`; +- positive rate: `36.7%`; +- Cohen's dz: `-0.166`; +- `t(29) = -0.9101`; +- `p = .3703`; +- bootstrap 95% CI: `[-0.2109, 0.0717]`. -The effect was therefore neither positive nor statistically distinguishable -from zero. +Leave-one-item-out analyses also failed the preregistered robustness +criterion. -Leave-one-item-out analyses also failed the preregistered robustness criterion. +Domain-level estimates changed direction: -Domain-level results were heterogeneous: +- biology: `-0.097`; +- computing: `+0.271`; +- engineering: `-0.364`; +- physics: `+0.172` (`n = 1`). -- biology: `-0.097` -- computing: `+0.271` -- engineering: `-0.364` -- physics: `+0.172` (`n = 1`) +The apparent exploratory `functions as` false-claim effect therefore did +not replicate. -Removing different semantic domains changed the direction of the overall -effect. - -Accordingly, the apparent `functions as` false-claim effect observed in the -earlier exploratory subset did **not replicate**. +![Focused replication](assets/figures/figure_3_focused_replication_PORTFOLIO.png) --- -## Unexpected Truth-Dependent Effect +## Secondary truth-dependent effect -Although the primary hypothesis failed, the focused replication revealed an -unexpected truth-dependent pattern. +Although the primary hypothesis failed, the focused replication produced +an unexpected truth-dependent pattern. -For **true claims**, `functions as` reliably reduced agreement relative to the -matched formal control: +For true claims, `functions as` reduced agreement relative to the matched +formal control: -- mean effect: `-0.3240` -- positive rate: `13.3%` -- `p = 1.59e-06` -- bootstrap 95% CI: `[-0.4298, -0.2255]` +- mean effect: `-0.3240`; +- positive rate: `13.3%`; +- `p = 1.59e-06`; +- bootstrap 95% CI: `[-0.4298, -0.2255]`. -This produced a positive false-minus-true interaction: +This yielded a positive false-minus-true interaction: -- mean interaction: `+0.2574` -- median interaction: `+0.2451` -- `t(29) = 3.3542` -- `p = .00223` -- bootstrap 95% CI: `[0.1083, 0.4073]` +- mean interaction: `+0.2574`; +- median interaction: `+0.2451`; +- `t(29) = 3.3542`; +- `p = .00223`; +- bootstrap 95% CI: `[0.1083, 0.4073]`. -This interaction does **not** rescue the original hypothesis. +This finding does **not** rescue the original epistemic-deference +hypothesis. -It arose primarily because functional-role wording reduced agreement with -true claims, not because it reliably increased agreement with false claims. +The interaction arose primarily because functional-role wording reduced +agreement with true claims, not because it reliably increased agreement +with false claims. -The truth-dependent effect is therefore reported as a **secondary finding -requiring independent replication**, rather than as evidence for the original -epistemic-deference hypothesis. +It is therefore retained as a secondary finding requiring independent +replication. --- -## Evidence Gate - -The project deliberately separates behavioral discovery from mechanistic -interpretation. +## Robustness across semantic domains -The completed research sequence is: +The focused primary effect was not stable across semantic domains. -`Pilot` +![Domain robustness](assets/figures/figure_4_domain_robustness_PORTFOLIO.png) -→ `Measurement Audit` +Removing different domains changed the sign of the aggregate effect. -→ `Broad Confirmatory Null` +This was another reason the preregistered robustness criterion was not +met. -→ `Linguistic Phenotyping` +--- -→ `Exploratory Role-Relational Candidate` +## Evidence gate -→ `Held-Out Role-Relational Null` +The project deliberately separates: -→ `Exploratory functions-as Candidate` +**behavioral discovery** -→ `Preregistered Focused Replication` +from: -→ **`Primary Null / Mechanistic Gate Failed`** +**mechanistic interpretation** Mechanistic localization was pre-specified to begin only if a robust -behavioral effect replicated. - -The focused replication failed that criterion. - -Mechanistic localization and causal intervention were therefore -**intentionally not pursued**. - -No claim about an "authority circuit", "indexical circuit", or other causal -internal mechanism is made. - ---- +behavioral phenomenon survived replication. -## What the Project Shows +The final focused primary hypothesis failed that criterion. -The central result of Indexical Circuits is methodological as much as -behavioral. +Therefore: -A linguistically plausible LLM effect can appear substantial in a pilot or -small exploratory subset and yet disappear when subjected to stronger, -independent testing. +**Mechanistic gate: closed** -The project documents the complete trajectory from apparent signal to -falsification rather than selecting only positive findings. +**Mechanistic localization: intentionally not pursued** -It demonstrates: +**Causal mechanism: not established** -- explicit measurement-interface validation -- AB/BA label-order counterbalancing -- separation of exploratory and confirmatory analyses -- increasingly controlled held-out stimulus construction -- prior-data contamination audits -- factual source verification before model testing -- frozen stimulus banks -- cryptographic stimulus fingerprints -- preregistered hypotheses -- pre-specified success criteria -- bootstrap uncertainty estimation -- item-level robustness testing -- semantic-domain robustness testing -- explicit stopping rules -- refusal to infer a mechanism from an effect that did not replicate +No claim is made that the model contains an: -The project therefore illustrates an important methodological principle for -LLM interpretability: +- authority circuit; +- indexical circuit; +- epistemic-deference circuit; +- causal sociolinguistic feature corresponding to the exploratory effects. -> **Before explaining an apparent model behavior mechanistically, first -> establish that the behavior itself is reproducible.** +This stopping decision is part of the study design rather than an +unfinished stage of the project. --- +## Why the negative result matters -## Visual Summary +A large exploratory effect can be tempting to explain mechanistically. -### Effect trajectory +But if the effect is unstable, stimulus-specific, or measurement-dependent, +a mechanistic analysis may produce an explanation for a pattern that does +not generalize. -![Effect trajectory](assets/figures/figure_1_effect_trajectory_PORTFOLIO.png) +Indexical Circuits therefore treats behavioral replication as a +precondition for causal interpretation. -*The apparent false-claim effect shrinks across progressively stronger tests: from an exploratory pilot to confirmatory, held-out, and preregistered focused replication.* +The project demonstrates several safeguards: -### Construction-level heterogeneity +- measurement-interface validation; +- AB/BA label-order counterbalancing; +- separation of exploratory and confirmatory analyses; +- newly constructed held-out stimulus banks; +- prior-data contamination audits; +- factual source verification; +- frozen stimulus sets; +- cryptographic dataset fingerprints; +- preregistered hypotheses; +- pre-specified success criteria; +- bootstrap uncertainty estimation; +- item-level robustness analysis; +- semantic-domain robustness analysis; +- explicit stopping rules; +- retention of null results. -![Construction heterogeneity](assets/figures/figure_2_construction_heterogeneity_PORTFOLIO.png) +--- -*The broad role-relational category concealed substantial construction-level heterogeneity. The exploratory `functions as` subset looked promising, but other constructions moved in different directions.* +## What this project supports -### Focused replication +The project supports a methodological conclusion: -![Focused replication](assets/figures/figure_3_focused_replication_PORTFOLIO.png) +**linguistically plausible LLM effects that look strong in pilots or +exploratory subsets may disappear under independent replication.** -*The preregistered primary `functions as` effect on false claims did not replicate. A secondary truth-dependent contrast remained, driven mainly by reduced agreement with true claims.* +It also identifies an unresolved empirical question: -### Domain robustness +**why did functional-role wording reduce agreement with true claims in +the final focused dataset?** -![Domain robustness](assets/figures/figure_4_domain_robustness_PORTFOLIO.png) +That truth-dependent effect is a candidate for a future independent study. -*The focused false-claim effect changed direction across semantic domains, failing the preregistered robustness criterion.* +It is not treated as confirmed here. --- -## Why This Matters +## What this project does not claim -Most work on LLM behavior treats linguistic variation primarily as changes in -wording or style. +This repository does not claim that: -This project asks whether socially meaningful linguistic constructions can -systematically change model epistemic behavior — but it also asks a prior -methodological question: +- formal register reliably induces epistemic deference; +- technical vocabulary explains the apparent pilot effect; +- role-relational framing reliably increases false-claim agreement; +- `functions as` reliably increases false-claim agreement; +- the secondary truth-dependent interaction is independently replicated; +- a causal internal mechanism has been localized. -> **Which apparent linguistic effects survive strong replication?** +Those claims are not supported by the completed evidence sequence. -The results show why this distinction matters. +--- -Effects that appeared substantial in exploratory subsets weakened or -disappeared under newly constructed, counterbalanced, source-verified, -held-out tests. +## Repository guide -This has implications for both sociolinguistic research on language models and -mechanistic interpretability. +### Research documentation -Without rigorous behavioral replication, researchers risk mechanistically -explaining patterns that are unstable, stimulus-specific, or artifacts of -experimental design. +Key documents in `docs/` include: -The broader contribution of Indexical Circuits is therefore to connect -sociolinguistic concepts such as register, relational construal, stance, and -indexical meaning with rigorous experimental practices for distinguishing -robust LLM behavior from unstable apparent effects. +- `hypotheses.md` — hypothesis development; +- `dataset_design.md` — experimental-data design; +- `behavioral_phenotyping.md` — exploratory linguistic coding; +- `heldout_role_relational_replication_preregistration.md` — held-out replication protocol; +- `functions_as_focused_replication_preregistration.md` — focused final preregistration; +- `register_realization_coding_manual.md` — linguistic annotation definitions; +- `experiment_log.md` — chronological research record; +- `results_and_lessons.md` — consolidated interpretation; +- `project_architecture.md` — project structure and workflow. ---- +### Data -## Repository Structure +`data/` contains stimulus banks, validation material, linguistic coding, +and frozen held-out materials. -`notebooks/` — behavioral experiments and analyses +`data/functions_as_focused/` contains the final 30-family focused +replication bank together with its provenance and audit material. -`data/` — stimuli, validation materials, linguistic coding, and frozen -held-out banks +### Results -`data/functions_as_focused/` — frozen 30-family focused replication bank, -prior-data audit, and source-verification log +`results/confirmatory/` contains broad confirmatory outputs. -`results/confirmatory/` — confirmatory behavioral outputs +`results/phenotyping/` contains exploratory linguistic phenotyping +outputs. -`results/phenotyping/` — exploratory linguistic phenotyping outputs +`results/functions_as_focused/` contains the final preregistered focused +replication outputs. -`results/functions_as_focused/` — complete outputs from the preregistered -focused replication +### Computational notebooks -`docs/` — hypotheses, preregistrations, experiment log, coding manual, -validation plans, and project architecture +The computational workflow is preserved in numbered notebooks: ---- +- `00_environment_and_sanity_check.ipynb` +- `01_behavioral_pilot.ipynb` +- `02_stimulus_validation_materials.ipynb` +- `03_stimulus_validation_analysis.ipynb` +- `04_confirmatory_stimulus_bank.ipynb` +- `05_confirmatory_validation_materials.ipynb` +- `06_confirmatory_behavior.ipynb` -## Scientific Status +The notebooks are retained in research order rather than being rewritten +after the final result. -The behavioral replication sequence is complete. +--- -**Initial pilot signal:** observed +## Study history + +| Stage | Status | +| --- | --- | +| Exploratory pilot | Apparent positive effect | +| Measurement audit | Interface corrected; pilot signal survived | +| Broad generic-formality replication | Not supported | +| Technical-lexicality explanation | Not supported | +| Exploratory role-relational candidate | Identified | +| Held-out role-relational replication | Not supported | +| Exploratory `functions as` candidate | Identified | +| Focused preregistered replication | Primary hypothesis not supported | +| Secondary truth-dependent interaction | Observed; requires independent replication | +| Mechanistic localization | Intentionally not pursued | -**Measurement-interface correction:** completed; pilot signal survived +--- -**Broad generic-formality effect:** not supported +## Future work -**Technical-lexicality explanation:** not supported +A clean follow-up should not reinterpret the failed primary hypotheses as +successful. -**Exploratory role-relational candidate:** identified +The most defensible next study would independently test the unexpected +truth-dependent effect using: -**Held-out role-relational replication:** primary hypothesis not supported +- new factual families; +- a preregistered directional hypothesis; +- multiple functional-role constructions; +- multiple model families; +- matched controls; +- the same measurement-interface safeguards. -**Exploratory `functions as` candidate:** identified +Only if such a behavioral effect replicates robustly should mechanistic +localization become the primary next step. -**Preregistered focused `functions as` replication:** primary hypothesis not -supported +--- -**False-claim `functions as` effect:** not replicated +## Research areas -**Truth-dependent `functions as` interaction:** observed as a secondary finding -requiring independent replication +- mechanistic interpretability; +- behavioral evaluation; +- sociolinguistics; +- linguistic indexicality; +- register and relational construal; +- epistemic judgment; +- large language models; +- AI safety; +- reproducibility. -**Mechanistic gate:** failed +--- -**Mechanistic localization:** intentionally not pursued under the pre-specified -stopping rule +## Researcher -**Causal mechanism:** not established +**Irene Theodoropoulou** ---- +Linguist working at the intersection of sociolinguistics, language +variation, discourse, and artificial intelligence. -## Research Areas +--- -- Mechanistic interpretability -- Sociolinguistics -- Linguistic indexicality -- Register and relational construal -- Large language models -- AI safety -- Epistemic deference -- Behavioral replication -- Reproducibility +## Bottom line ---- +The project began with an apparently large sociolinguistic effect on model +epistemic judgment. -## Researcher +The effect became weaker and ultimately failed as the experimental +evidence became more independent and more tightly controlled. -**Irene Theodoropoulou** +Rather than localizing a mechanism anyway, the study obeyed its +pre-specified evidence gate and stopped. -Linguist working at the intersection of sociolinguistics, language variation, -discourse, and artificial intelligence. +For mechanistic interpretability, knowing when **not** to explain a +behavior is part of the research. diff --git a/docs/reproducibility.md b/docs/reproducibility.md new file mode 100644 index 0000000..d0a228d --- /dev/null +++ b/docs/reproducibility.md @@ -0,0 +1,145 @@ +# Reproducibility + +## Scope + +Indexical Circuits preserves the complete research sequence from the +initial exploratory pilot through the final preregistered focused +replication. + +The repository distinguishes between: + +1. lightweight repository-integrity checks that can run automatically + in GitHub Actions; and +2. GPU/model experiments preserved in the research notebooks. + +The CI workflow does not attempt to redownload or reevaluate +Gemma-2-2B-IT on every repository change. + +## Primary model + +The behavioral experiments use: + +google/gemma-2-2b-it + +Access to the model may require authentication through Hugging Face and +acceptance of the model's applicable access conditions. + +## Original computational environment + +The original experimental notebooks were run in Google Colab with GPU +acceleration. + +The environment sanity-check notebook records use of a Tesla T4 GPU. + +Core libraries used in the notebook environment include: + +- PyTorch +- transformers +- accelerate +- huggingface_hub +- pandas + +The environment notebook explicitly installed: + +- transformers +- accelerate +- huggingface_hub +- pandas==2.2.3 + +PyTorch was available in the Colab runtime. + +Because hosted notebook environments and model-serving libraries change +over time, the repository preserves the executed notebooks and outputs +as part of the computational record rather than claiming that future +library versions will reproduce floating-point values bit-for-bit. + +## Authentication + +Model loading requires appropriate Hugging Face access. + +The notebooks use Hugging Face authentication before loading +google/gemma-2-2b-it. + +Authentication credentials are not stored in this repository. + +## Research workflow + +The numbered notebooks in `notebooks/` preserve the chronological +experimental workflow. + +They should be read in order. + +The project intentionally preserves exploratory and confirmatory stages +rather than rewriting the notebook history after the final result. + +## Lightweight automated checks + +The repository includes a GitHub Actions workflow that runs: + +pytest -q + +The test environment is intentionally minimal and is specified in: + +requirements-test.txt + +These checks validate repository-level invariants without requiring a +GPU or model download. + +## Data and result provenance + +The repository retains: + +- exploratory stimulus material; +- confirmatory stimulus banks; +- held-out replication material; +- prior-data overlap audits; +- source-verification logs; +- preregistration documents; +- frozen focused-replication data; +- behavioral outputs; +- analysis results; +- portfolio figures. + +The final focused `functions as` stimulus bank was constructed from 30 +new factual families and checked against prior experimental material +before model testing. + +Its frozen dataset fingerprint is: + +64b3fd0cc9d0cba4f650789fc0a6e7b35787352fa17b15a5592cf885db48a14e + +## Interpretation boundary + +Reproducing an already completed analysis does not turn an exposed result +into a new confirmatory test. + +The final primary `functions as` hypothesis did not replicate, and the +pre-specified mechanistic gate remained closed. + +Any future test of the secondary truth-dependent interaction should use +new held-out material and should be treated as a separate study. + +## Recommended reproduction path + +For repository-integrity checks: + +1. Clone the repository. +2. Create a Python environment. +3. Install: + + python -m pip install -r requirements-test.txt + +4. Run: + + pytest -q + +For the original model experiments: + +1. Use a GPU-capable Python environment. +2. Obtain access to google/gemma-2-2b-it through Hugging Face. +3. Authenticate without storing credentials in the repository. +4. Follow the numbered notebooks in `notebooks/`. +5. Compare regenerated outputs with the preserved files in `results/`. + +The repository's primary purpose is to preserve the experimental design, +evidence sequence, and decision logic transparently. diff --git a/requirements-test.txt b/requirements-test.txt new file mode 100644 index 0000000..e96a830 --- /dev/null +++ b/requirements-test.txt @@ -0,0 +1 @@ +pytest>=8,<10