[AI-3] Add the LLM connection record and discovery transport #24879
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| #-- copyright | ||
| # OpenProject is an open source project management software. | ||
| # Copyright (C) the OpenProject GmbH | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License version 3. | ||
| # | ||
| # OpenProject is a fork of ChiliProject, which is a fork of Redmine. The copyright follows: | ||
| # Copyright (C) 2006-2013 Jean-Philippe Lang | ||
| # Copyright (C) 2010-2013 the ChiliProject Team | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License | ||
| # as published by the Free Software Foundation; either version 2 | ||
| # of the License, or (at your option) any later version. | ||
| # | ||
| # This program is distributed in the hope that it will be useful, | ||
| # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| # GNU General Public License for more details. | ||
| # | ||
| # You should have received a copy of the GNU General Public License | ||
| # along with this program; if not, write to the Free Software | ||
| # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. | ||
| # | ||
| # See COPYRIGHT and LICENSE files for more details. | ||
| #++ | ||
|
|
||
| # The connection to an OpenAI-API-compatible LLM server. | ||
| # | ||
| # Only a single connection is supported today. That is enforced by a validation | ||
| # rather than by the schema, so lifting the restriction later is a one-line change: | ||
| # every association is already scoped by +llm_connection_id+ and the STI +type+ | ||
| # column is in place. | ||
|
Comment on lines
+33
to
+36
Contributor
There was a problem hiding this comment. Do we already have plans to support multiple connections? If they are definitely coming, we might want to consider dropping that singleton approach, and just relying on a simple |
||
| class LlmConnection < ApplicationRecord | ||
| include Redmine::Ciphering | ||
|
Contributor
There was a problem hiding this comment. 🤔 We don't usually do encryption at rest and the only other code paths that use this method are leftovers from Redmine. So, this doesn't look like something we want to lean into. I'm gonna do a bit more skulking and then get back to you.
Contributor
There was a problem hiding this comment. @tangopium Let's drop the attribute encryption altogether please -- the current consensus is that we do not support it, for various good reasons. |
||
|
|
||
| SINGLETON_NAME = "default" | ||
|
|
||
| has_many :health_reports, as: :subject, dependent: :delete_all | ||
|
Contributor
There was a problem hiding this comment. @NobodysNightmare How do we feel about this one? Is that
Contributor
There was a problem hiding this comment. Uh totally. I am happy if we start using it in more places, though it's good to be aware that more consumers exist, because it's a "naturally grown abstraction", not one that was designed before it was implemented. I didn't look into this usage here, but if there is:
then it sounds like a good fit to me. |
||
|
|
||
| validates :base_url, presence: true | ||
| validate :only_one_connection, on: :create | ||
|
|
||
| class << self | ||
| # The connection record, whether or not it has been persisted yet. | ||
| # | ||
| # Identifying attributes are left unset here and filled in by | ||
| # LlmConnections::SetAttributesService as system changes, so that they do not | ||
| # register as user-made changes to non-writable attributes. | ||
| def instance | ||
| first || new | ||
| end | ||
|
|
||
| # Cheap enough to call from a menu visibility lambda. | ||
| def enabled? | ||
| exists?(enabled: true) | ||
| end | ||
|
|
||
| # Whether LLM-backed features may run right now. This is the predicate | ||
| # sibling features gate on; see #77783. | ||
| def available? | ||
| OpenProject::FeatureDecisions.llm_connection_active? && | ||
| enabled? && | ||
| instance.configured? | ||
| end | ||
|
Comment on lines
+64
to
+68
Contributor
There was a problem hiding this comment. The The The intent behind that Then, I would suggest |
||
| end | ||
|
|
||
| def api_key | ||
| read_ciphered_attribute(:api_key) | ||
| end | ||
|
|
||
| def api_key=(value) | ||
| write_ciphered_attribute(:api_key, value) | ||
| end | ||
|
|
||
| # Deliberately does not consider +last_connected_at+: a connection provisioned | ||
| # from the environment is never probed, and must still count as configured. | ||
| def configured? | ||
| base_url.present? | ||
| end | ||
|
|
||
| def server_flavour | ||
| options["server_flavour"].presence&.to_sym | ||
| end | ||
|
|
||
| private | ||
|
|
||
| def only_one_connection | ||
| return unless self.class.where.not(id:).exists? | ||
|
|
||
| errors.add(:base, :singleton) | ||
| end | ||
| end | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| #-- copyright | ||
| # OpenProject is an open source project management software. | ||
| # Copyright (C) the OpenProject GmbH | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License version 3. | ||
| # | ||
| # OpenProject is a fork of ChiliProject, which is a fork of Redmine. The copyright follows: | ||
| # Copyright (C) 2006-2013 Jean-Philippe Lang | ||
| # Copyright (C) 2010-2013 the ChiliProject Team | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License | ||
| # as published by the Free Software Foundation; either version 2 | ||
| # of the License, or (at your option) any later version. | ||
| # | ||
| # This program is distributed in the hope that it will be useful, | ||
| # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| # GNU General Public License for more details. | ||
| # | ||
| # You should have received a copy of the GNU General Public License | ||
| # along with this program; if not, write to the Free Software | ||
| # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. | ||
| # | ||
| # See COPYRIGHT and LICENSE files for more details. | ||
| #++ | ||
|
|
||
| module Llm | ||
| # Resolves a connection's api_format to the adapter that speaks it. | ||
| # | ||
| # There is no universal model-discovery standard, so each dialect needs its own | ||
| # translation: OpenAI-compatible servers answer GET /models with data[].id, | ||
| # Gemini uses /v1beta/models with richer metadata, Bedrock needs AWS signing | ||
| # rather than a bearer token, and Azure indirects through deployment names. | ||
| # | ||
| # Only the OpenAI adapter is implemented. The seam exists so that adding one is | ||
| # a new class rather than a migration. | ||
| module Adapters | ||
| class UnsupportedFormat < StandardError; end | ||
|
|
||
| # Formats an administrator can choose. "openai" covers OpenAI itself and the | ||
| # great majority of gateways and self-hosted inference servers; the rest are | ||
| # RubyLLM providers whose model lists come from its registry. | ||
| OPENAI_COMPATIBLE = "openai" | ||
|
|
||
| FORMATS = %w[ | ||
|
Contributor
There was a problem hiding this comment. So this is currently more or less a mirror of a similar listing in I guess it makes sense to have it here to be more explicit. But in that case, I think we need a way to detect any drift after gem updates. Something like: |
||
| openai | ||
| anthropic | ||
| gemini | ||
| mistral | ||
| deepseek | ||
| openrouter | ||
| perplexity | ||
| xai | ||
| ollama | ||
| gpustack | ||
| azure | ||
| bedrock | ||
| vertexai | ||
| ].freeze | ||
|
|
||
| def self.for(connection) | ||
| format = connection.api_format.to_s | ||
| raise UnsupportedFormat, format unless FORMATS.include?(format) | ||
|
|
||
| if format == OPENAI_COMPATIBLE | ||
| # Queried live, so the list is what this endpoint actually serves. | ||
| Openai.new(connection) | ||
| else | ||
| RegistryBacked.new(connection) | ||
| end | ||
| end | ||
| end | ||
| end | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| #-- copyright | ||
| # OpenProject is an open source project management software. | ||
| # Copyright (C) the OpenProject GmbH | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License version 3. | ||
| # | ||
| # OpenProject is a fork of ChiliProject, which is a fork of Redmine. The copyright follows: | ||
| # Copyright (C) 2006-2013 Jean-Philippe Lang | ||
| # Copyright (C) 2010-2013 the ChiliProject Team | ||
| # | ||
| # This program is free software; you can redistribute it and/or | ||
| # modify it under the terms of the GNU General Public License | ||
| # as published by the Free Software Foundation; either version 2 | ||
| # of the License, or (at your option) any later version. | ||
| # | ||
| # This program is distributed in the hope that it will be useful, | ||
| # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| # GNU General Public License for more details. | ||
| # | ||
| # You should have received a copy of the GNU General Public License | ||
| # along with this program; if not, write to the Free Software | ||
| # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. | ||
| # | ||
| # See COPYRIGHT and LICENSE files for more details. | ||
| #++ | ||
|
|
||
| module Llm | ||
| module Adapters | ||
| # Servers speaking the OpenAI API: OpenAI itself, and the great majority of | ||
| # gateways and self-hosted inference servers. | ||
| class Openai | ||
| def initialize(connection) | ||
| @connection = connection | ||
| end | ||
|
|
||
| # Normalised model cards. | ||
| # | ||
| # The raw card is kept alongside, because the fields worth having are the | ||
| # non-standard ones: the OpenAI schema itself carries only id, object, | ||
| # created and owned_by, while vLLM adds max_model_len -- the operator's | ||
| # actual --max-model-len, and the only trustworthy context window for this | ||
| # deployment. | ||
| # | ||
| # @return [Array<Hash>] cards with :id and :raw | ||
| def models | ||
| @models ||= Array(client.models["data"]).filter_map do |card| | ||
| id = card["id"] | ||
| next if id.blank? | ||
|
|
||
| { id:, raw: card } | ||
| end | ||
| end | ||
|
|
||
| def embeddings(model_id:, input:) | ||
|
|
||
| client.embeddings(model: model_id, input:) | ||
| end | ||
|
|
||
| # Which server we are talking to decides which non-standard metadata is | ||
| # worth reading later. +owned_by+ is the documented hint; the structural | ||
| # fallback catches an operator who overrode it. | ||
| def server_flavour | ||
| cards = models | ||
| owner = cards.first&.dig(:raw, "owned_by").to_s.downcase | ||
|
|
||
| return owner if %w[vllm sglang llamacpp openai].include?(owner) | ||
|
|
||
| cards.any? { |card| card[:raw].key?("max_model_len") || card[:raw].key?("root") } ? "vllm" : "unknown" | ||
| end | ||
|
|
||
| private | ||
|
|
||
| attr_reader :connection | ||
|
|
||
| def client | ||
| @client ||= Llm::Client.new(base_url: connection.base_url, | ||
| api_key: connection.api_key, | ||
| headers: connection.custom_headers) | ||
| end | ||
| end | ||
| end | ||
| end | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to drag in a deprecation warning.
Please make sure it's cleaned up -- looks like it just needs a little configuration tweak.