Skip to content

Conversation

@KatieMSB
Copy link
Collaborator

  • Identified the issue which this PR solves.
  • Read the CONTRIBUTING document.
  • Code builds clean without any errors or warnings.
  • Added appropriate tests for any new functionality.
  • All new and existing tests passed.
  • Added comments in the code, where necessary.
  • Ran make check to catch common errors. Fixed any that came up.

Description:
Adds new uik_logs table for storing the most recent requests to universal integration keys.

Which issue(s) this PR fixes:
Part of #4187

Out of Scope:
Follow up work will be to make logs viewable in the UI to aid in debugging integrations with universal integration keys.

Additional Considerations:
This implementation focuses on capturing and storing the most recent request log for each universal integration key to support basic debugging functionality as a MVP.
In the future, we may want to expand this system to store multiple logs per key, with smarter deduplication strategies or time-based retention. Ideas include:

  • Retaining the last N unique logs, with a configurable maximum (e.g. 15 per integration key), potentially deduplicated by payload and result type.
  • Tracking both the most recent successful request and the most recent error, as different error types (e.g. parse, execution, send) may need to be surfaced separately.
  • Adding support for aggregate metadata, such as first/last seen timestamps and occurrence counts, similar to Kubernetes’ event log behavior.
  • Setting TTLs for stored logs, such as keeping full payloads for 48 hours while retaining metadata longer.
  • Future UI support could show a breakdown of what actions were generated, including any errors encountered, and could allow for test payload replay and pinning important logs.

@KatieMSB KatieMSB changed the title uik: save more recent request to universal integration key uik: save most recent request to universal integration key Jul 23, 2025
}

if !utf8.Valid(data) {
_ = h.upsertUikLog(ctx, keyID, req, nil, LogStatusParseError, "invalid UTF-8")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should log these, maybe remove the error return value from the method and handle the error internally (by logging it)


CREATE TABLE uik_logs (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
integration_key_id uuid NOT NULL REFERENCES integration_keys(id) ON DELETE CASCADE,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit/preference: we can just add UNIQUE to the column definition at table creation time instead of creating an explicit index below

integration_key_id uuid NOT NULL UNIQUE REFERENCES integration_keys(id) ON DELETE CASCADE,

Comment on lines +101 to +106
ctx context.Context,
keyID uuid.UUID,
req *http.Request,
rawBody []byte,
status string,
errMsg string,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: with so many args it may be worth using a struct, but if we use explicit types it could at least prevent mixing up status/err and enforce explicit values

Suggested change
ctx context.Context,
keyID uuid.UUID,
req *http.Request,
rawBody []byte,
status string,
errMsg string,
ctx context.Context,
keyID uuid.UUID,
req *http.Request,
rawBody []byte,
status RequestStatus,
err error,

Alternatively, we could just take err and wrap it appropriately to tell the difference

h.upsertUIKLog(ctx, id, req, body, wrapError(err, ParseError))

and then inspect it within the method

}

// upsertUikLog will save a log of the most recent request to a universal integration key in the db
func (h *Handler) upsertUikLog(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: https://go.dev/wiki/CodeReviewComments#initialisms

Suggested change
func (h *Handler) upsertUikLog(
func (h *Handler) upsertUIKLog(

Comment on lines +38 to +41
LogStatusSuccess = "success"
LogStatusParseError = "parse_error"
LogStatusExecError = "exec_error"
LogStatusSendError = "send_error"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe these could be error types? since success is sort of on it's own

Comment on lines +120 to +127
ContentType: sql.NullString{
String: req.Header.Get("Content-Type"),
Valid: req.Header.Get("Content-Type") != "",
},
UserAgent: sql.NullString{
String: req.UserAgent(),
Valid: req.UserAgent() != "",
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just make these non-null and allow the empty string? not a huge deal but would simplify things

data, err := io.ReadAll(req.Body)
if errutil.HTTPError(ctx, w, err) {
_ = h.upsertUikLog(ctx, keyID, req, nil, LogStatusParseError, "read error: "+err.Error())
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the logger rather than fmt.Print* so it goes to the right place and format

const (
Example Flag = "example"
UnivKeys Flag = "univ-keys"
UikLogs Flag = "uik-logs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: https://go.dev/wiki/CodeReviewComments#initialisms

Suggested change
UikLogs Flag = "uik-logs"
UIKLogs Flag = "uik-logs"

also, I'm wondering if there's a different way to describe this than "logs" and "logs support" since it doesn't have anything to do with application logs, and the table itself only records the last status/result, and maintains no running "log" of events.

Maybe uik_requests for the table, and the expflag I'm less sure, maybe uik-debug?

Comment on lines +51 to +70
-- name: IntKeyLog :exec
INSERT INTO uik_logs (
integration_key_id,
content_type,
user_agent,
raw_body,
status,
error_message,
received_at
) VALUES (
$1, $2, $3, $4, $5, $6, $7
)
ON CONFLICT (integration_key_id) DO UPDATE
SET
content_type = EXCLUDED.content_type,
user_agent = EXCLUDED.user_agent,
raw_body = EXCLUDED.raw_body,
status = EXCLUDED.status,
error_message = EXCLUDED.error_message,
received_at = EXCLUDED.received_at;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For testing/consistency, we'll want to use DB time -- if we pass in the received_at time, we'll need to do a separate query to get the current time. Instead, we can just rely on the default of now() and on the conflict update set the value to the current DB time.

Suggested change
-- name: IntKeyLog :exec
INSERT INTO uik_logs (
integration_key_id,
content_type,
user_agent,
raw_body,
status,
error_message,
received_at
) VALUES (
$1, $2, $3, $4, $5, $6, $7
)
ON CONFLICT (integration_key_id) DO UPDATE
SET
content_type = EXCLUDED.content_type,
user_agent = EXCLUDED.user_agent,
raw_body = EXCLUDED.raw_body,
status = EXCLUDED.status,
error_message = EXCLUDED.error_message,
received_at = EXCLUDED.received_at;
-- name: IntKeyLog :exec
INSERT INTO uik_logs (
integration_key_id,
content_type,
user_agent,
raw_body,
status,
error_message
) VALUES (
$1, $2, $3, $4, $5, $6
)
ON CONFLICT (integration_key_id) DO UPDATE
SET
content_type = EXCLUDED.content_type,
user_agent = EXCLUDED.user_agent,
raw_body = EXCLUDED.raw_body,
status = EXCLUDED.status,
error_message = EXCLUDED.error_message,
received_at = now();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants