Kendalls W

[juliuspfadt]

adds Kendalls coefficient of concordance with the irr::kendall function.
also restructures the agreement analysis a bit
fixes jasp-stats/jasp-issues#2151

[Copilot]

Pull request overview

Adds Kendall’s coefficient of concordance (W) to the Reliability module’s Rater Agreement analysis (using irr::kendall()), and refactors the Rater Agreement UI/options around a shared bootstrap/CI + seed “Advanced Options” section. Also adds extensive info tooltips across multiple analyses and updates tests + versioning.

Changes:

• Add Kendall’s W computation + bootstrap CI support to raterAgreement (R + QML) and extend unit tests with a published reference dataset.
• Restructure Rater Agreement options (new bootstrapSamples, coefficient checkboxes default off) + add upgrade mapping for renamed option.
• Add info metadata to many tables/plots/controls; bump version + update renv.lock.

Reviewed changes

Copilot reviewed 27 out of 64 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
R/raterAgreement.R	Adds Kendall’s W + placeholder table; renames bootstrap option usage; adds info strings.
inst/qml/RaterAgreement.qml	Adds Kendall’s W UI + “Advanced Options” section for CI/bootstrap/seed; expands allowed variable types.
tests/testthat/test-raterAgreement.R	Updates options for new defaults; adds Kendall’s W tests + reference check.
tests/testthat/anxietyRatings.csv	New reference dataset for Kendall’s W verification test.
inst/Upgrades.qml	Adds rename upgrade: krippendorffsAlphaBootstrapSamplesForCI → bootstrapSamples.
R/standardErrorOfMeasurement.R	Adds info metadata; adjusts $dependOn() wiring for CI tables/plots.
inst/qml/StandardErrorOfMeasurement.qml	Adds info tooltips across SEM UI controls.
R/unidimensionalReliabilityFrequentist.R	Adds info metadata to tables.
inst/qml/UnidimensionalReliabilityFrequentist.qml	Adds info tooltips and minor UI restructuring (e.g., Samples section).
R/unidimensionalReliabilityBayesian.R	Adds info metadata to tables/containers/plots.
inst/qml/UnidimensionalReliabilityBayesian.qml	Adds info tooltips across Bayesian UI controls.
R/intraclassCorrelation.R	Adds info metadata to main ICC table.
inst/qml/IntraclassCorrelation.qml	Adds info tooltips to ICC controls.
R/blandAltman.R	Adds info metadata to Bland–Altman containers/plots/tables.
inst/qml/BlandAltman.qml	Adds info tooltips to Bland–Altman controls.
NEWS.md	Adds release notes for 0.97.1 (Kendall’s W + UI restructuring).
DESCRIPTION	Bumps package version/date.
renv.lock	Updates pinned dependency metadata (incl. jaspTools entry) + updates JASP GitHub SHAs.
.Rbuildignore	Ignores additional dev directories (.positai, .claude).
.gitignore	Ignores .positai.
.vscode/settings.json	Adds workspace R LSP setting.
inst/Description.qml	Minor formatting change (blank line).
inst/help/unidimensionalReliabilityFrequentist.md	Deleted help page.
inst/help/unidimensionalReliabilityBayesian.md	Deleted help page.
inst/help/standardErrorOfMeasurement.md	Deleted help page.
inst/help/raterAgreement.md	Deleted help page.
inst/help/intraclassCorrelation.md	Deleted help page.
inst/help/toolTip/sampleSavingFreq.md	Deleted tooltip help page (replaced by info).
inst/help/toolTip/sampleSavingBayes.md	Deleted tooltip help page (replaced by info).

[FBartos]

is it jaspResults[["kendallWBootstrapSamples"]]$object or jaspResults[["kendallWBootstrapSamples"]] for the null call?

[juliuspfadt]

It's jaspResults[["kendallWBootstrapSamples"]] — guarding on the state element itself, per the module's standard idempotency convention. Fixed here and in the sibling .kripAlphaBoot for consistency. (The old $object form didn't actually error — NULL$object is NULL in R — but this is cleaner.)

[FBartos]

@juliuspfadt maybe good points?

[juliuspfadt]

Implemented (cbb4262): added id: ciOpt to the CI checkbox; the bootstrap-samples field, CI level, and SetSeed now bind enabled: ciOpt.checked, so the advanced controls are disabled when CI is unchecked.

[FBartos]

@juliuspfadt maybe good points?

[juliuspfadt]

Implemented (cbb4262): SetSeed now enabled: ciOpt.checked, disabled when the CI box is unchecked (same for the bootstrap-samples field and CI level).

[Copilot]

Pull request overview

Copilot reviewed 27 out of 64 changed files in this pull request and generated 10 comments.

[FBartos]

build builder bot!

[FBartos]

@juliuspfadt maybe good points?

[FBartos]

@juliuspfadt maybe good points?

[FBartos]

build builder bot!

[Copilot]

Pull request overview

Copilot reviewed 29 out of 66 changed files in this pull request and generated 3 comments.

[juliuspfadt]

Good catch — fixed in 6f21c58. fromVersion lowered to 0.95.5 (the released version that still carries krippendorffsAlphaBootstrapSamplesForCI); 0.97.0 was never released so the migration was being skipped for real .jasp files.

[juliuspfadt]

Correction to the above after checking the Desktop upgrader source (upgrades.cpp findClosestVersion): a step applies to files saved with fromVersion or lower (Desktop picks the closest step at or above the file's version), not fromVersion-or-higher as assumed here. So the original 0.97.0 was actually fine, and lowering to 0.95.5 (6f21c58) introduced a gap for 0.96.x files (released v0.96.0, master's 0.96.4) — and 0.95.5 was never a release anyway (tags: v0.95.0, v0.95.3, v0.96.0). Fixed in 9b58f31: fromVersion = 0.96.4, the last version carrying the old option name, with a QML comment documenting the or-lower rule. Manually verified: a file saved in JASP 0.96 with a non-default bootstrap count opens in the current build with the value intact.

[juliuspfadt]

Intentional. dcurver is a transitive dependency via mirt (used for the IRT SEM analysis), and forcing a source build avoids an OpenMP linker failure on CI (35af3c2). Leaving as-is.

[juliuspfadt]

This is auto-generated renv boilerplate (regenerated by renv when its version changes), not hand-authored logic, so not modifying it here.

[FBartos]

I did a deeper pass over the current branch, including targeted checks against irr under R 4.5.2. I think several issues should be resolved before merge.

Correctness issues

1. Ordered factor levels are discarded for Kendall's W and Krippendorff's alpha (R/raterAgreement.R:314, 391-392, 436).

   as.matrix(data.frame) converts ordered factors to character. irr::kendall() then ranks the labels lexicographically rather than using the declared factor-level order. In a targeted ordered-factor example, the current path gave W = 0.1349206; preserving the factor codes gave W = 0.3634921. Ordinal alpha likewise changed from -0.3031556 to 0.01939338, while interval alpha on character labels returned NA. This becomes especially relevant now that the QML variable list admits ordinal variables.

2. Tie correction defaults off, and the warning returned by irr is ignored (inst/qml/RaterAgreement.qml:128-133; R/raterAgreement.R:392, 436).

   On the included anxiety data, the default uncorrected result is W = 0.5019215, p = .0723804; correcting ties gives W = 0.5396569, p = .0428835. irr explicitly returns "Coefficient may be incorrect due to ties" in the first case, but the backend drops it. Bootstrap resampling also creates duplicate rows and hence ties even when the source data are continuous. I would strongly consider making correction the default or mandatory, and at minimum surface the warning.

3. Kendall bootstrap runs before general validation and can terminate the analysis on incomplete or small data (R/raterAgreement.R:40-43, 370-393, 428-434).

   The nominal-type guard fixed only part of the earlier review concern. The bootstrap still runs before the observation/infinity checks and resamples the original data before listwise deletion. A replicate can therefore contain only one complete subject. Under R 4.5.2, a four-row example with one incomplete row failed at replicate 12 with incorrect number of dimensions. A one-subject input similarly exposes a raw irr/apply() error instead of a JASP validation message. Validation and complete-case handling need to precede bootstrapping, with an explicit policy for invalid replicates.

4. The Kendall validation appears incompatible with ratersInRows (R/raterAgreement.R:66-78, 428-433).

   After the dataset is transposed, .hasErrors() still receives the original selected variable names through all.target. Those names generally no longer identify columns in the transposed data, so a valid row-oriented input can be reported as having fewer than two observations. All new Kendall tests use ratersInColumns, so this path is untested.

5. Fleiss' kappa can now crash when a category exists only in a listwise-deleted row (R/raterAgreement.R:239-254).

   irr::kappam.fleiss() deletes incomplete rows before producing detail, but categories is derived from the original dataset. With rows A A A, B B B, and C NA C, detail contains only A/B; indexing the matrix with A/B/C raises subscript out of bounds. Displayed categories should come from the actual analyzed data/detail rows, while preserving the intended ordering.

6. Fleiss SEs/CIs are reconstructed from rounded display values (R/raterAgreement.R:233-237, 252-269).

   irr::kappam.fleiss(detail = TRUE) rounds category kappa and z to three decimals. Dividing kappa by z cannot recover an accurate SE, and zero kappa gives 0/0. For ratings AA, AB, BA, BB, the current overall/category SEs are NaN, although the analytic category SE is 0.5. The SE should be computed directly from the underlying formula rather than reconstructed from rounded output.

7. Krippendorff bootstrap state can be stale after changing the measurement method (R/raterAgreement.R:294-302, 343-366).

   Both the table and bootstrap calculation read krippendorffsAlphaMethod, but neither dependency list includes it. Switching Nominal/Ordinal/Interval/Ratio in a live analysis can therefore reuse samples generated under the previous method and pair them with the new estimate/label.

8. The widened variable box lacks coefficient-specific validation (inst/qml/RaterAgreement.qml:30-35; Cohen/Fleiss code in R/raterAgreement.R:81-282).

   The shared list now accepts nominal, ordinal, and scale columns so Kendall can use ordinal/scale variables. Cohen and Fleiss have no corresponding backend checks, however. Continuous scale variables can be treated as one category per distinct value and produce plausible-looking but meaningless kappa results. The UI text says weighted Cohen requires ordinal ratings and Fleiss requires nominal ratings, but this is not enforced.

Documentation and integration concerns

9. All long-form analysis help is removed without equivalent replacement.

   The deleted ICC, Rater Agreement, SEM, Bayesian reliability, frequentist reliability, and sample-saving help pages contained assumptions, equations, interpretation guidance, references/DOIs, and package provenance. The new one-sentence info strings do not retain most of that content. Please confirm that wholesale removal of module help is an explicit product decision; otherwise the pages should remain alongside the inline information.

10. The Bayesian samples tooltip is incorrect (inst/qml/UnidimensionalReliabilityBayesian.qml:282-290; R/unidimensionalReliabilityBayesian.R:169-188).

    It describes samples as posterior samples per chain after burn-in, but the backend passes it as total n.iter; retained samples are (samples - burnin) / thinning.

11. The committed macOS workaround is ineffective (renv/settings.json:2).

    renv 1.2.3 does not recognize the "build" setting and silently omits it. The current macOS job still installs the dcurver binary and then fails loading dcurver.so with missing ___kmpc_for_static_fini. Current master has the same platform failure, so it is not caused by Kendall W, but this PR's attempted workaround does not solve it and should not remain as a claimed fix.

12. The new snapshot RDS files imply an undeclared minimum R version.

    They use serialization version 3. R CMD build warns and injects R >= 3.5.0, while DESCRIPTION declares no R minimum. The package should state its actual supported minimum R version.

Tests and scope to examine

The new tests cover point estimates, an explicitly tie-corrected case, and one published dataset, but not:

• ordered factors with nonalphabetic level order;
• default behavior when ties are present;
• sparse incomplete data and bootstrap failure handling;
• ratersInRows;
• invalid coefficient/type combinations;
• a Fleiss category present only in a deleted row;
• zero/near-zero Fleiss kappa CIs;
• live Krippendorff method invalidation;
• migration of a non-default bootstrap count from a real 0.95.5/0.96.4 .jasp file.

The originating issue also contains a request for the alternative F-test alongside the large-sample chi-square test. This PR implements only chi-square, so whether the F-test is in or out of scope should be decided explicitly.

Minor user-facing mismatches also remain: the Bland-Altman info says the bias line is solid although all three lines are drawn dashed; the Rater Agreement variable tooltip always describes columns as raters even in ratersInRows mode; and bootstrap/seed controls remain enabled when only Cohen/Fleiss (asymptotic CIs) or no coefficient is selected.

Verification context: all R sources parse, Windows and both Ubuntu CI jobs pass, and the branch is currently mergeable without code conflicts. macOS and coverage fail before feature tests because of shared dependency/workflow problems also present on master. The targeted statistical reproductions above used the lockfile's R 4.5.2 and irr 0.84.1.

[juliuspfadt]

Thanks for the thorough pass — all reproducible. Addressed in a9479f5 (R backend + tests), 3a997e0 (QML), f570a36 (packaging):

Correctness

1. Fixed. Kendall's W converts factors to their level codes (ranks are within-rater, so codes suffice). Krippendorff's alpha compares values across raters, so plain per-column codes would collapse distinct labels from different columns onto the same code — it instead maps all columns onto the union of their levels, preserving declared level order, so identical labels keep identical codes. Regression test with non-alphabetic ordered levels validates against irr on the codes.
2. Fixed. correctForTies now defaults to TRUE (option is new in this PR, no upgrade needed). When correction is off and irr flags ties, the warning is surfaced as a table footnote. Test: anxiety data uncorrected reproduces your W = .50192, p = .0724 and asserts the footnote.
3. Fixed. The bootstrap now listwise-deletes and validates (≥ 2 complete rows, finite) before resampling, resamples complete cases only, and wraps each replicate in try() — failed replicates become NA, are excluded from the CI, and a footnote reports the count. Your 4-row/one-incomplete scenario is now a test (completes; "16 of 100 bootstrap samples could not be computed").
4. Fixed. Validation no longer goes through .hasErrors(all.target = ...); it checks the (transposed) ratings matrix directly, so ratersInRows works. Test added against the transposed irr reference.
5. Fixed. Categories now come from the listwise-complete data, so a category that only occurs in a deleted row no longer crashes. Your A/B/C example is a test.
6. Fixed. Per-category kappas and all SEs are now computed directly from the Fleiss–Nee–Landis formulas instead of irr's 3-decimal detail table (category SE = sqrt(2/(ns·nr·(nr−1)))). Your AA/AB/BA/BB case is a test asserting kappa = 0, SE = 0.5. Existing Fleiss expected values regenerated (unrounded).
7. Fixed. krippendorffsAlphaMethod added to both the bootstrap state and table dependencies. Invalidation can't be unit-tested via jaspTools (one state per run) — verified manually in the GUI: switching Nominal → Ordinal now recomputes the bootstrap, SE/CI update together with the estimate.
8. Fixed. Cohen's and Fleiss' kappa now error on scale variables; weighted Cohen additionally errors on nominal. I kept Fleiss accepting ordinal (it treats categories as unordered, which is standard practice) rather than nominal-only. Tests added.

Docs / integration

9. The help-page removal is intentional — the module moved to inline info-based help. Agreed some depth (references, interpretation guidance) got lost; I'll extend the info strings where it matters rather than resurrect the separate pages.
10. Fixed. Tooltip now says total iterations per chain incl. burn-in, retained = (samples − burn-in) / thinning.
11. Removed — confirmed renv 1.2.3 ignores the "build" setting. The macOS dcurver/OpenMP failure is shared with master and needs a separate fix.
12. Fixed. Depends: R (>= 3.5.0) declared (serialization-v3 floor).

Tests: ordered factors, ties default/warning, sparse bootstrap, ratersInRows, invalid type combos, deleted-category Fleiss, and zero-kappa Fleiss are all covered now (raterAgreement suite: 27 passing; full suite green locally). Not unit-testable via jaspTools: live Krippendorff method invalidation (see 7) and the .jasp migration (Desktop applies Upgrades.qml at file load, jaspTools has no upgrade machinery) — both now verified manually: method switching recomputes the bootstrap, and a JASP 0.96 file with a non-default bootstrap count keeps its value in the current build (after the fromVersion correction in 9b58f31, see the inline thread).

F-test: implemented after all (5921860) — shown alongside the chi-square test (F = (m−1)·W/(1−W), df₁ = n−1−2/m, df₂ = df₁·(m−1); Kendall & Babington Smith 1939, Legendre 2005), cross-validated against vegan::kendall.global. That fully covers jasp-stats/jasp-issues#2151.

Minor mismatches (Bland-Altman line description, variables tooltip, bootstrap/seed enablement when only asymptotic-CI coefficients are selected) are all fixed in 3a997e0.

[FBartos]

Remaining correctness blockers

1. Weighted Cohen's kappa ignores declared ordinal order, including in the normal columns layout (R/raterAgreement.R:159).

   Ordered factors are passed directly to psych::cohen.kappa(), which derives an alphabetical order from their labels. With declared levels low < medium < high, a concrete two-rater example returns weighted quadratic kappa 0; converting the same ratings with their declared level codes gives 0.75. The estimator path predates this PR, but the PR now explicitly admits ordinal variables and advertises weighted ordinal kappa, making this a supported workflow. A validated common category order must be established before calling psych.

2. Factor metadata is still lost in ratersInRows (R/raterAgreement.R:73, 83-107).

   The dataset is transposed before factors are encoded. t() converts ordered factors to character and discards their levels:

   • Kendall parses low/medium/high as numbers, obtains zero complete rows, and rejects valid data.
   • Krippendorff reconstructs an arbitrary first-appearance order; one reproduction gave ordinal alpha 0.08333 instead of 0.77083 using the declared order.
   • Weighted Cohen can likewise use alphabetical rather than declared order.

   Encoding has to happen before transposition. The added row-layout test uses numeric data and misses this interaction.

3. Effective sample-size checks still occur before missing-data removal (R/raterAgreement.R:154-159, 262-277).

   Cohen checks raw rows, then can terminate fatally inside psych when a pair has no overlapping observations. Fleiss also checks raw rows before irr performs listwise deletion: zero complete rows fatal with subscript out of bounds, while only two complete rows are analyzed despite the stated three-subject minimum. Validate the actual pairwise/listwise analyzed data first.

4. Krippendorff can report perfect reliability with no usable data (R/raterAgreement.R:369-373).

   A 3-by-2 all-NA dataset completes with nominal alpha 1, including a bootstrap CI of [1,1]. There is no check for at least two raters, pairable observations, finite metric data, or estimable expected disagreement.

5. One-rater row-layout data remain fatal or invalid for three coefficients (R/raterAgreement.R:21, 156, 373, 508-541).

   ready checks selected subject columns rather than the number of resulting raters. With one row/rater, Cohen fatals in combn(), Krippendorff fatals inside irr, and Kendall reports W = 1 followed by invalid F output. Fleiss already has the necessary rater check; the other coefficients need equivalent validation.

6. The F test is emitted outside its mathematical domain (R/raterAgreement.R:529-541).

   With two subjects and two raters, df1 = n - 1 - 2/m = 0 and df2 = 0. The current table reports either F = Inf or F = 0, with pF = NaN. Suppress the F columns or show a clear error/footnote whenever either degree of freedom is nonpositive.

Additional behavioral issues

7. The new Fleiss calculation creates malformed rows for a single category (R/raterAgreement.R:303).

   apply() simplifies the one-category result to a vector and t() produces the wrong dimensions. Constant "A" ratings produce Overall, A, and extra NULL-labelled rows. Since kappa is not estimable when expected agreement is one, this should be a clear table error. This regression is introduced by the PR's manual Fleiss calculation.

8. Krippendorff interval/ratio silently manufacture equal distances for factor labels (R/raterAgreement.R:97-107).

   All factor/character values are mapped to 1..K, regardless of method. For labels 0,10,100, interval alpha was 0.67593 on the manufactured codes versus 0.56197 using the actual scores; ratio alpha was 0.63373 versus 0.39198. Nominal may map labels and ordinal may use ordered codes, but interval and ratio should require suitable numeric scale data. Ratio data also need domain validation.

9. Degenerate Kendall rankings complete with a table of NaNs (R/raterAgreement.R:508-541).

   If every rater is constant, the analysis status is complete while W, chi-square, F, and both p-values are NaN. Validate the result after irr::kendall() and show a variation/estimability error.

10. Uncorrected bootstrap CIs are distorted by ties introduced by resampling (R/raterAgreement.R:464-468, 571-572).

    Sampling rows with replacement creates tied ranks even when the source rankings have none, but every replicate receives correctForTies = FALSE and the warning checks only the original result. For three identical tie-free rankings 1:5, 1,000 replicates, seed 1:

    • uncorrected: W = 1, SE = 0.09726, CI [0.5,1], no warning;
    • corrected: W = 1, SE = 0, CI [1,1].

    At minimum, CI plus uncorrected W needs a warning; statistically, the bootstrap path needs a defined treatment of resampling-induced ties.

11. Krippendorff table dependencies omit seed options (R/raterAgreement.R:351-360 versus state dependencies at 410-418).

    The consuming table does not depend on setSeed/seed, while the bootstrap state does. This is the same invalidation pattern addressed for the method option and can leave a seed-only CI change stale. Kendall's table correctly lists both.

12. Cohen row-mode output metadata are incorrect (R/raterAgreement.R:165, 192-200).

    For exactly two row raters, the pair label concatenates every selected subject name. Subject counts also include completely unrated subject rows; a three-subject analyzed example was reported as four. One overall count is also insufficient when pairwise sample sizes differ.

UI/documentation accuracy

Several newly added info strings remain factually inaccurate:

• inst/qml/IntraclassCorrelation.qml:73-75 says averaged ratings always produce ICC(A,k), although the backend selects ICC1/2/3,k and the agreement/consistency interpretation is model-specific.
• inst/qml/StandardErrorOfMeasurement.qml:214-218 says minimumGroupSize applies to all methods except Mollenkopf-Feldt/IRT; the backend uses it only for Thorndike, Feldt, ANOVA, and generalized Lord.
• R/unidimensionalReliabilityBayesian.R:1427 calls the optional probability ribbon a credible interval; the credible interval is a separate error bar.
• inst/qml/UnidimensionalReliabilityBayesian.qml:565-567 says mean/median changes plots, but it changes summary outputs, not posterior density plots.
• The retained sample count at inst/qml/UnidimensionalReliabilityBayesian.qml:282-290 is exactly ceiling((samples-burnin)/thinning), not generally the raw quotient.
• The placeholder info at R/raterAgreement.R:49-63 describes selected coefficients although the object exists only when none are selected; the Fleiss table info says three or more raters while the backend and QML permit two.

The F-test comment at R/raterAgreement.R:524-525 also describes the wrong small-sample dimension: the concern is especially few judges/raters, not subjects. Marozzi (2014) explicitly notes that Legendre did not examine this F test, although the implemented formula itself is correct.

Verification and remaining test gaps

• Exact R 4.5.2 full suite: 105 passed, 0 failed, 1 skipped.
• Rater Agreement suite: 27 passed.
• Ordinary nondegenerate Kendall/F results match irr and vegan.
• Missing tests: ordered-factor weighted Cohen, labelled factors in row mode, zero effective overlap, all-missing Krippendorff, one rater, n=2/m=2, single-category Fleiss, constant Kendall, and uncorrected bootstrap on tie-free rankings.
• Measurement-level changes are read through variables.types, but that option is absent from result/bootstrap dependency lists; this still needs a live JASP invalidation check.
• The migration now looks correct from Desktop source semantics, but a representative 0.95.5/0.96.x .jasp fixture with a non-default bootstrap count would still make it regression-testable.
• Windows, Ubuntu, and i18n checks pass. macOS and coverage retain shared dependency failures also present on master.
• Exact R CMD build succeeds but the newly added snapshot paths generate many >100-byte tar-path warnings; git diff --check still flags two whitespace lines in renv/activate.R. NEWS also omits the F test.

Given the weighted-Cohen result, factor handling, missing/effective-data failures, and invalid F-domain output, I think the PR should remain blocked pending another backend pass.

[juliuspfadt]

All addressed in d1296bb (backend + tests), f902d6f (info strings), 371ab32 (NEWS/whitespace):

Correctness blockers

1. Fixed. psych ignores factor level order entirely (also when passed factors), so Cohen's kappa now feeds it level codes from the union of the columns' levels. Your low/medium/high example reproduces the codes-based weighted kappa; regression test added.
2. Fixed at the root. .raterAgreementHandleData now encodes discrete columns onto the union of levels before transposing and rebuilds factors (union levels, declared order) afterwards, so row mode is equivalent to column mode for all coefficients. Row-mode test with labelled ordered factors added (Kendall + ordinal Krippendorff match the column-mode references).
3. Fixed. Fleiss validates the listwise-complete data (≥ 3 complete rows); Cohen's psych call is wrapped and reports a clean error for zero pairwise overlap; Krippendorff validates pairable observations before computing.
4. Fixed. New .krippAlphaValidate: ≥ 2 raters, at least one subject with ≥ 2 ratings, non-constant ratings, finite metric data, numeric data for interval/ratio, non-negative for ratio. All-NA now errors instead of α = 1.
5. Fixed. Cohen/Krippendorff/Kendall now error on < 2 raters (post-transpose), matching Fleiss. One-rater row-mode test added.
6. Fixed. F columns are NA with an explanatory footnote when df₁ = n − 1 − 2/m ≤ 0; n=2/m=2 test added.

Behavioral

7. Fixed. Single-category data now gives a clear "not estimable" error before the manual computation (which also removes the malformed apply() path).
8. Fixed. Interval/ratio now parse labels as numeric scores (your 0/10/100 case reproduces the scores-based α, test added); non-numeric labels error with a pointer to nominal/ordinal; ratio additionally requires non-negative values. Nominal/ordinal keep the union-codes mapping.
9. Fixed. Non-finite W (constant rankings) → "not estimable" error, test added.
10. Fixed (option b). Bootstrap replicates now always use the tie correction — resampling-induced ties are artifacts, and mixing them into the uncorrected formula distorted the CI. A footnote states this when the point estimate is uncorrected. Your identical-rankings example now gives SE = 0, CI [1,1] (test added). This changed the stored CI expectations slightly for the uncorrected+CI test.
11. Fixed. Table depends on setSeed/seed; variables.types added to the deps of everything that branches on it (Cohen, Fleiss, Kendall table + Kendall bootstrap state). The types invalidation still deserves a live GUI check — type changes also change the dataset, which should invalidate everything anyway.
12. Fixed. Pair labels come from the (post-transpose) data columns — row-mode raters are named Rater1..m; the subject count requires ≥ 2 ratings per row; with missing data the table now shows a per-pair n column.

Info strings — all six corrected as described (ICC k-version of selected type; SEM group size = Thorndike/Feldt/ANOVA/Lord generalized; probability-region shading vs CI error bar; point estimate affects tables; retained samples = ceiling; placeholder/Fleiss texts). The F-test comment now points at small numbers of raters and cites Marozzi (2014) with the Legendre caveat.

Housekeeping — NEWS gained the F test (Added) and a Fixed summary; the two trailing-whitespace lines in renv/activate.R are stripped. The >100-byte tar paths come from the long vdiffr snapshot names — renaming them means regenerating all snapshots, so I'd prefer to leave that for a dedicated cleanup (the warnings are benign for a non-CRAN module).

Verification — full suite 129 passing locally (was 106; +23 from the new regression tests: ordered-factor weighted Cohen, row-mode labelled factors, zero overlap, all-missing Krippendorff, one rater, n=2/m=2, single-category Fleiss, constant rankings, tie-free uncorrected bootstrap, interval-scores, non-numeric interval labels). The two items that remain manual: live invalidation on a variable-type change, and the 0.96 → current .jasp migration (verified by hand earlier today).

[juliuspfadt]

Ran an additional multi-angle self-review over the full PR diff; a033eac fixes what it found (all verified by running the code, not just reading it):

Correctness

• Union-of-levels order: appearance-order unions scrambled ordinal codes when raters carried different level subsets (e.g. a rater who never used a middle category: levels {1,3} ∪ {1,2,3} → order 1,3,2, so "2" coded above "3"). Numeric labels are now ordered by value; text labels are merged preserving each column's declared order. Regression test with a missing-middle-category rater validates against irr on the label values.
• Fleiss with mixed factor/numeric columns: unlist(dataset) returns factor codes when columns mix factors and numerics, breaking the category derivation (spurious "single category" errors); additionally as.matrix() on mixed columns applies format() and pads numbers (" 7" vs "7"), corrupting label matching inside irr too. Categories now derive from labels via the union levels, and columns are converted to character explicitly before irr. Test with labels 7/10 across factor and numeric columns.
• Row mode with mixed column types: one discrete column forced all columns through label codes, silently replacing numeric magnitudes (4 ranked above 7.1). Mixed categorical/continuous row-mode input is now a validation error.
• Constant rankings, tie correction off: irr::kendall returns a meaningless finite W = 0 there, bypassing the NaN guard — now detected via zero within-rater variance, so both correction settings error consistently.
• Krippendorff deps: variables.types added to the table and bootstrap state, matching the other coefficients.

Kendall CI policy refinement: with replicates tie-corrected (as agreed for #10), an uncorrected point estimate could fall outside its own CI (constructed: W = 0.762 with CI [1,1]). Bootstrap CIs are therefore only offered for the tie-corrected coefficient; the uncorrected estimate gets a footnote explaining why. The F test now also shows NA with a footnote at W = 1 instead of ∞.

Cleanup: shared .raterAgreementUnionCodes() replaces the three hand-copied encoders (the seam most of these bugs lived in), shared CI-column helper replaces five copies, the Krippendorff bootstrap transposes once outside the loop, dead code removed.

Side effect worth noting: the new union ordering makes the weighted-Cohen expectations for the disjoint-category stress data (facGender/facExperim/debBinMiss20) land back on the original master values. Full suite: 138 passing.

[FBartos]

Follow-up review after the latest fixes, performed against exact PR head a033eac91dbf02c8483ed213d65745aa59aa6790.

The latest commit resolves several earlier findings, including the corrected/uncorrected Kendall CI mismatch, numeric partial-level ordering, mixed row-mode handling, the Kripp dependency declarations, and the perfect-concordance F output. However, I can still reproduce the following issues.

Correctness blockers

1. Partial textual ordinal level sets still make results depend on rater order.

   .raterAgreementUnionLevels() greedily starts from the longest level set and inserts missing labels, but it does not correctly merge all later ordering constraints. With three raters declaring low < high, medium < high, and low < medium, the same ratings give:

   • columns r1,r2,r3: weighted Cohen average 0.0212963; ordinal Krippendorff alpha -0.0810064
   • columns r2,r3,r1: weighted Cohen average 0.2125545; ordinal Krippendorff alpha 0.1402254

   Rater permutation cannot change either coefficient. This needs a proper constraint-based ordered merge, with a clear error for contradictory or ambiguous schemas. A regression test should use textual subsets where no single rater declares the complete scale.

2. The new shared encoder corrupts Kendall's W for mixed ordinal/scale raters in column mode.

   .kendallWRatings() now sends every column through .raterAgreementUnionCodes() whenever any column is discrete. Consequently, continuous values are ordered by first appearance instead of numeric magnitude when the union also contains text labels. In my reproduction, the same data give W = 0.7189542, then W = 0.3137255 after merely reordering subjects; the direct irr::kendall() reference is 0.6143791.

   Kendall ranks within each rater, so numeric columns should remain numeric and factor columns should use their own declared level codes. The shared cross-rater union is not appropriate here.

3. Krippendorff validation still accepts non-estimable effective coincidence data and invalid bootstrap replicates.

   With one pairable A/A item plus singleton B ratings, JASP reports a point estimate of NaN but a bootstrap CI of [1, 1]. The current validation counts variation from singleton ratings that never participate in a coincidence. During bootstrapping, single-category resamples for which irr returns alpha = 1 are also accepted as valid.

   Validation should be based on the effective coincidence data, require a finite point estimate, and reject non-estimable bootstrap replicates rather than merely checking for a thrown error.

4. Cohen's kappa still validates the raw row count rather than each analyzed rater pair.

   Current reproductions:

   • both raters constant: blank kappa with no table error
   • one pairwise-complete subject: NaN
   • two pairwise-complete subjects: kappa 1, despite the analysis's stated minimum of three subjects/items

   Each pair needs its own complete-case count, category/denominator validation, and finite-result check before its result is included in the table or average.

Secondary issues

5. Invalid corrected Kendall inputs bootstrap before the table performs its validation. One-rater and all-constant inputs can pass .kendallWBootRA()'s initial guard and run every requested bootstrap replicate before the table reports the error. Since the UI allows up to 10,000,000 replicates, the rater-count and variation checks should be shared with or run before the bootstrap.

6. Cohen pair labels are malformed when variable names contain spaces. For example, Rater one and Rater two are displayed as Rater - one Rater two. The labels should be constructed directly from possiblePairs and colnames(dataset), rather than replacing the first space in the names returned by psych.

Verification at this head: the full R 4.5.2 module suite passes (137 passed, 0 failed, 1 existing empty-test skip), git diff --check is clean, and the focused reproductions above were run after updating to a033eac. The passing tests therefore do not currently cover these cases.

[juliuspfadt]

All six addressed in dc2a127; each of your reproductions was run before and after.

Correctness blockers

1. Fixed. .raterAgreementUnionLevels() now does a topological sort over the precedence constraints declared by every column, instead of a greedy insert. Your three-rater schema (low<high, medium<high, low<medium) merges to low < medium < high and gives identical results for all six column permutations (weighted Cohen 0.364923747276688, ordinal alpha 0.294117647058823 for my regression fixture). Genuinely ambiguous schemas (low<high, medium<high, with low vs medium undetermined) and cyclic ones now set ordered = FALSE; weighted Cohen and ordinal Krippendorff error out with a clear message, while order-free coefficients (Fleiss, nominal alpha) still compute. The flag survives the raters-in-rows transpose, which rebuilds every column with one shared level set and would otherwise hide the ambiguity. Regression tests use textual subsets where no rater declares the complete scale, plus the ambiguous and cyclic cases.

2. Fixed — and this one was mine. The a033eac consolidation routed Kendall through the cross-rater union encoder; you're right that it must not. .kendallWRatings() is back to per-column encoding (numeric columns stay numeric, factor columns use their own declared level codes), since Kendall ranks within each rater. Added a mixed ordinal/scale test asserting subject-permutation invariance and equality with the direct irr::kendall() reference.

3. Fixed. Validation is now based on the effective coincidence data: only subjects rated at least twice contribute, so your one-pairable-A/A-item-plus-singleton-Bs case errors ("the pairable ratings do not vary") instead of reporting NaN with a CI of [1,1]. The table additionally requires a finite point estimate, and bootstrap replicates whose pairable ratings collapse to one category are rejected rather than accepted at alpha = 1.

4. Fixed. Each pair is now computed and validated on its own pairwise-complete data: >= 3 jointly rated subjects, ratings must vary, estimate must be finite. Your three reproductions now yield clear errors instead of a blank kappa, NaN, and kappa = 1. When only some pairs are unusable the usable ones are still reported, with a note naming the excluded pairs and their reason; the average is taken over the valid pairs only.

Secondary

5. Fixed. The bootstrap now mirrors the table's rater-count and variance guards, so one-rater and constant inputs skip resampling entirely (constant data with 100,000 replicates requested returns immediately; there's a test asserting it).

6. Fixed. Labels are built from combn + colnames(dataset). Rater one/Rater two now render as Rater one - Rater two. Side effect worth flagging: Cohen rows now follow combn order (V1-V2, V1-V3, V1-V4, …) rather than psych's column-major order — the kappas and their pairings are unchanged, and the stored expectations were regenerated accordingly.

Two deliberate test changes. The legacy test.csv weighted-Cohen block used facGender {f,m}, facExperim {control,experimental} and debBinMiss20 {0,1} — disjoint categories, i.e. exactly the ambiguous schema from (1); it only ever computed because that block never sets variables.types. It now asserts the ambiguity error, and weighted-kappa value coverage (with SE/CI, cross-checked against psych) moved to a new test on a valid shared ordinal scale. Fleiss category display order for that block follows the appearance-order fallback (nominal, so cosmetic).

Full suite: 163 passing locally.

[juliuspfadt]

On your closing point — "the passing tests therefore do not currently cover these cases" — 3a168e4 adds a property-based invariance suite (tests/testthat/test-other-raterAgreementInvariance.R) so this class of bug is caught by construction rather than by example.

Every coefficient is asserted invariant under operations that are no-ops on the ratings:

• permuting the raters, and permuting the subjects/items (also with missing data);
• relabelling the categories — order-preserving for the ordinal coefficients, any bijection for the nominal ones;
• expressing the same data with raters in rows instead of columns;
• a strictly increasing per-rater transform (Kendall's W ranks within raters);
• an affine rescaling (Krippendorff interval), with a negative control asserting that ratio alpha is scale- but not shift-invariant, so the comparison is demonstrably able to detect a change.

Several fixtures deliberately give the raters only subsets of the scale (no rater observes every category) — the configuration that makes the cross-rater level merge observable, i.e. your finding 1. A guard also fails the property if a coefficient errored, so a table that failed to compute cannot pass vacuously.

The suite is mutation-verified rather than merely green: reintroducing each historical bug makes it fail — greedy level merge → 6 failures, Kendall via the cross-rater union codes → 3 failures, t() before encoding → 1 failure + 1 error. On the current code it is clean (174 assertions, ~7 s; module suite 337 passing).

NEWS.md now also records the user-visible behaviour changes from this PR: tie correction on by default with bootstrap CIs restricted to the corrected coefficient, the common-ordinal-scale requirement for weighted Cohen / ordinal alpha, per-pair Cohen validation and the combn row order, the Krippendorff coincidence validation, and the Fleiss standard errors.

CI at dc2a127: Windows, both Ubuntu jobs and i18n pass. macOS and coverage still fail while installing dependencies — dlopen(dcurver.so): symbol not found '___kmpc_for_static_fini' → mirt fails to install — which is the shared failure also present on master, before any module test runs.

[FBartos]

Follow-up review against exact current PR head 3a168e40724abe8b751959659f558e63531c8ade.

The latest two commits resolve all six issues from my preceding comment. The new topological merge also passed an independent comparison against 1,000 randomized brute-force ordering cases. Three additional correctness cases remain outside the new tests, however.

1. Numeric-looking ordinal labels bypass the declared order

R/raterAgreement.R:79-80 sorts the union numerically whenever all labels parse as numbers and immediately returns ordered = TRUE. This bypasses both the factor-level precedence graph and its contradiction/ambiguity detection.

An order-preserving relabeling of the same ordinal data from factor levels low, medium, high to factor levels 1, 3, 2 changes:

• weighted Cohen average: 0.4058692 -> -0.1825397
• ordinal Krippendorff alpha: 0.4100 -> -0.1800

The ratings and declared category order are identical; only the labels changed. Two raters declaring contradictory numeric orders are also silently accepted instead of receiving the new common-scale error.

Factor levels should remain authoritative even when their labels look numeric. Numeric sorting should be limited to genuinely numeric columns for which no factor-level order exists. A regression test should use a non-monotonic but order-preserving numeric relabel such as low, medium, high -> 1, 3, 2, plus contradictory numeric factor orders.

2. Pairwise weighted Cohen drops unused positions from the common scale

At R/raterAgreement.R:325, each pair is passed to psych::cohen.kappa() without its levels argument. psych therefore refactors that pair using only the categories observed by those two raters, even though the union encoder retained the full declared scale.

For a declared scale A < B < C < D, with the pair observing only A, C, and D:

• current JASP result: 0.1304348
• psych::cohen.kappa(..., levels = 1:4) reference: 0.1025641

Dropping the unused middle category B changes the linear/quadratic distances among the observed categories. Each pair should receive the full common levels, or an explicit weight matrix based on that scale. The regression test needs a four-level scale with an unused interior category.

3. Row-mode Kendall ignores an ambiguous merged ordinal scale

raterAgreementHandleData() correctly records an ambiguous row-mode merge in the levelOrderAmbiguous attribute (R/raterAgreement.R:157-165). Weighted Cohen and ordinal alpha inspect this state, but .kendallWRatings(), the Kendall table, and .kendallWBootRA() do not.

With subject columns declaring partial constraints A < C, B < C, and C < D, merely changing the subject-column order changes Kendall's W:

• s1, s2, s3: 0.5764706
• s2, s1, s3: 0.8588235

In row mode the transposed rater columns need one common category order to rank subjects. Under the PR's existing policy, Kendall should reject this ambiguous scale, and the same guard must run before bootstrapping. A row-mode invariance/error test with partial ambiguous level sets would cover this path.

Verification at this head:

• full R 4.5.2 module suite: 336 passed, 0 failed, 1 existing empty-test skip
• focused Rater Agreement and invariance tests: pass
• git diff --check: clean
• worktree: clean

The passing invariance suite currently uses numeric relabels whose numeric and declared orders coincide, full common factor levels for row/column equivalence, and no weighted-Cohen pair missing an interior scale category; therefore it does not exercise these three counterexamples.

[juliuspfadt]

All three addressed in 250867e; each of your reproductions was checked before and after.

Correctness blockers

1. Fixed. .raterAgreementUnionLevels() no longer short-circuits to a numeric sort whenever the union of labels parses as numbers. Numeric sorting is now only used as a column's local order when that column carries no declared (factor) order of its own; any factor column always goes through the order-preserving topological merge, using its own levels() regardless of whether the labels look numeric. Your low,medium,high -> 1,3,2 relabel now reproduces the same weighted Cohen (0.710144927536232) and ordinal Krippendorff (0.675925925925926) as the text-label versions. Two raters declaring contradictory numeric orders (e.g. levels 1,2,3 vs 1,3,2) now correctly hit the common-scale error instead of being silently sorted. Regression tests added for both.

2. Fixed. Each pair now passes the full common-scale codes to psych::cohen.kappa() via levels = (seq_along(unionLevels$levels)), so a pair missing an interior category (e.g. observing only A, C, D on a declared A<B<C<D scale) keeps that category's correct position/weight instead of psych silently narrowing the scale to what the pair happens to observe. Test added with a 4-level scale and an unused interior category.

3. Fixed. .kendallWBootRA() and .computeKendallWTable() now check the levelOrderAmbiguous attribute that .raterAgreementHandleData() already sets during the raters-in-rows transpose, and refuse to compute with the same "requires a common ordinal scale" error used by weighted Cohen / ordinal Krippendorff. Your A<C, B<C, C<D partial-constraint example now errors identically regardless of subject-column order, and the bootstrap is skipped (not just the table) when the merge is ambiguous. Regression tests cover both the table and the bootstrap path, asserting invariance across column orderings.

Verification: full module suite passes (raterAgreement + the invariance suite, no regressions), 7 new regression tests added reproducing your three repro cases plus the negative/contradiction cases, NEWS.md updated. I also re-checked the earlier Copilot threads still marked "unresolved" on GitHub (SEM ciTable/combinedPlot/ciPlotsContainer dependOn, RA bootstrap state guarded on $object, QML enabled bindings) — those are already fixed by earlier commits in this PR (a9479f5, d600975, cbb4262), just never marked resolved in the UI.