[AI-3] Store the model catalogue and capability verdicts by tangopium · Pull Request #24882 · opf/openproject

tangopium · 2026-08-22T14:55:04Z

Ticket

What are you trying to accomplish?

PR 4 of 11 in the AI-3 stack. Persists what the server offers and what is known about it: LlmModel rows synced from the catalogue after every credential change and on demand via the Refresh models action, and LlmCapabilityVerdict rows in three states (supported, unsupported, unknown), because the OpenAI model list carries no capability information and "we cannot tell" is the honest answer. Verdicts are seeded from the ruby_llm registry where the model id is catalogued, administrator assertions survive re-detection, and everything is discarded when the credentials point at a different deployment. Models the server stops offering are deactivated rather than deleted, so anything referencing one still has something to name. No UI beyond the refresh action; the list follows in PR 5.

Merge checklist

Added/updated tests
Behind the llm_connection feature flag

Stacked on #24881.

github-actions · 2026-08-22T14:55:33Z

Caution

The provided work package version does not match the core version

Details:

Work package URL: https://community.openproject.org/work_packages/66020
Work package version: not set
Core version: 17.9.0

Please make sure that:

The work package version OR your pull request target branch is correct

github-actions · 2026-08-22T14:55:35Z

Caution

The Enterprise plan field is not set on the work package

Details:

Work package URL: https://community.openproject.org/work_packages/66020

Please make sure that:

The work package Enterprise plan field is set

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c176e37fd0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-08-22T15:50:50Z

Warning

Flaky specs

rspec ./spec/features/roles/report_spec.rb[1:3]
rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]
rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:2]

🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #24882, linked for reference only):

- `rspec ./spec/features/roles/report_spec.rb[1:3]`
- `rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]`
- `rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:2]`

Treat this as a standalone task, unrelated to PR #24882. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #24882 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @tangopium to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @tangopium, and request a review from @tangopium.
On every commit, set @tangopium as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

… capabilities Persists the models a connection offers as LlmModel records, refreshed by LlmConnections::SyncModelsService after every credential change and on demand through the new Refresh models action. Models the server stops offering are deactivated rather than deleted, so anything pointing at one still has something to name. What is known about each model is stored as LlmCapabilityVerdict rows in three states: supported, unsupported and unknown, because the OpenAI model list carries no capability information and the honest answer is very often that we cannot tell. Verdicts are seeded from the ruby_llm registry where the model id is catalogued, survive re-detection when an administrator asserted them, and are discarded wholesale when the credentials point at a different deployment. No UI beyond the refresh action yet; the model list and manual entries follow in later parts. Part 4 of the AI-3 stack. https://community.openproject.org/work_packages/66020

github-actions · 2026-08-22T16:13:44Z

Warning

Flaky specs

rspec ./modules/backlogs/spec/features/inbox_column_spec.rb[1:7:1]
rspec ./modules/wikis/spec/features/admin/internal_provider_spec.rb[1:1]

🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #24882, linked for reference only):

- `rspec ./modules/backlogs/spec/features/inbox_column_spec.rb[1:7:1]`
- `rspec ./modules/wikis/spec/features/admin/internal_provider_spec.rb[1:1]`

Treat this as a standalone task, unrelated to PR #24882. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #24882 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @tangopium to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @tangopium, and request a review from @tangopium.
On every commit, set @tangopium as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

github-actions · 2026-08-22T16:38:39Z

Warning

Flaky specs

rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]

🤖 Ask Copilot to investigate

Copy the prompt below into a new comment on this PR to delegate the investigation to GitHub Copilot. It will look into the flakiness and open a separate pull request with you as reviewer.

@copilot The following spec(s) are flaky in CI (first seen on PR #24882, linked for reference only):

- `rspec ./spec/features/work_packages/table/switch_types_spec.rb[1:1:1]`

Treat this as a standalone task, unrelated to PR #24882. Create a new branch from origin/dev and open a new pull request targeting dev — do not stack it on PR #24882 or reuse that branch.

Follow the playbook in docs/development/testing/handling-flaky-tests/README.md to find the root cause and fix the underlying race — do not skip, delete, or weaken the spec to make it pass; disabling is a last resort per the playbook, and only with a bug ticket. Verify the fix by running the spec(s) repeatedly (e.g. `script/bulk_run_rspec --run-count 10`).

If you cannot reproduce the flake or are not confident in a fix after reasonable investigation, do not fabricate a change or skip the spec to force CI green. Instead, leave the pull request in draft and document what you tried, the suspected cause, and any leads in its description, then assign @tangopium to take over.

Once the fix is verified, title the PR after the spec(s) it fixes, and use the PR description to explain the root cause, how the change resolves it, and the before/after results. Label the PR `flaky-spec`, assign @tangopium, and request a review from @tangopium.
On every commit, set @tangopium as the sole co-author with a `Co-authored-by:` trailer (use their GitHub no-reply email so it links to their account), so it is traceable who dispatched the fix.

thykel

👋 The concept looks sound to me, but a couple of things leave me scratching my head. Additionally, this needs way more test coverage for the new models, sync service, and enrichment mechanism.

thykel · 2026-08-29T21:07:44Z

+  # -- an LLM server that has not finished starting.
+  class SyncModelsJob < ApplicationJob
+    def perform
+      connection = LlmConnection.first


I don't see a reason to implement that single-connection constraint here in addition to the model. We can just iterate the whole table to reduce friction later.

thykel · 2026-08-29T21:17:46Z

+
+      cards.each do |card|
+        model = connection.models.find_or_initialize_by(external_id: card.fetch(:id))
+        model.update!(display_name: card[:display_name],


Shouldn't we want to also retain the original name if we got nothing from the server? Something like:

Suggested change

model.update!(display_name: card[:display_name],

model.update!(display_name: card[:display_name].presence || model.display_name),

thykel · 2026-08-29T21:20:34Z

+      states = relevant.index_with { |capability| published.include?(capability) ? :supported : :unsupported }
+      states.merge(embeddings: embedding ? :supported : :unsupported)


Aren't we missing the edge case of unknown here?

tangopium mentioned this pull request Aug 22, 2026

[AI-3] List the cached models on the connection page #24883

Open

2 tasks

tangopium mentioned this pull request Aug 22, 2026

[#66020] Add LLM connection settings to admin AI configuration #24851

Closed

3 tasks

chatgpt-codex-connector Bot reviewed Aug 22, 2026

View reviewed changes

Comment thread app/services/llm_connections/sync_models_service.rb

Comment thread app/services/llm_connections/sync_models_service.rb

Comment thread app/services/llm_connections/sync_models_service.rb Outdated

Comment thread app/models/llm_connection.rb

tangopium force-pushed the feature/66020-pr04-model-records-sync-capabilities branch from c176e37 to 2eda2e5 Compare August 22, 2026 15:05

tangopium changed the title ~~[#66020] Store the model catalogue and capability verdicts~~ [AI-3] Store the model catalogue and capability verdicts Aug 22, 2026

tangopium force-pushed the feature/66020-pr04-model-records-sync-capabilities branch from 2eda2e5 to 69ec8da Compare August 22, 2026 15:26

tangopium force-pushed the feature/66020-pr04-model-records-sync-capabilities branch from 69ec8da to 028a030 Compare August 22, 2026 15:51

tangopium force-pushed the feature/66020-pr04-model-records-sync-capabilities branch from 028a030 to 281818a Compare August 22, 2026 16:16

tangopium requested a review from thykel August 22, 2026 21:39

tangopium self-assigned this Aug 22, 2026

This was referenced Aug 25, 2026

[AI-3] Add the LLM connection record and discovery transport #24879

Open

[AI-3] Add the LLM connection administration page #24880

Open

thykel reviewed Aug 29, 2026

View reviewed changes

Navigation Menu

Uh oh!

[AI-3] Store the model catalogue and capability verdicts - #24882

[AI-3] Store the model catalogue and capability verdicts#24882
tangopium wants to merge 1 commit into
feature/66020-pr03-disconnect-and-key-removalfrom
feature/66020-pr04-model-records-sync-capabilities

tangopium commented Aug 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

thykel left a comment •

edited

Loading

Uh oh!

thykel Aug 29, 2026

Uh oh!

thykel Aug 29, 2026

Uh oh!

thykel Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

	model.update!(display_name: card[:display_name],
	model.update!(display_name: card[:display_name].presence \|\| model.display_name),

		states = relevant.index_with { \|capability\| published.include?(capability) ? :supported : :unsupported }
		states.merge(embeddings: embedding ? :supported : :unsupported)

Uh oh!

Conversation

tangopium commented Aug 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket

What are you trying to accomplish?

Merge checklist

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

thykel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thykel Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

thykel Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

thykel Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

tangopium commented Aug 22, 2026 •

edited

Loading

thykel left a comment •

edited

Loading