Skip to content

Suppress raw JSON logs in Complement CI when debug disabled. #19745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
reivilibre wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

Suppress raw JSON logs in Complement CI when debug disabled. #19745

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d08d390
Make Complement CI output less unwieldy
reivilibre Apr 29, 2026
16ed294
Newsfile
reivilibre Apr 29, 2026
edf5953
Add complement_tests.yml to Complement trigger
reivilibre Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 33 additions & 4 deletions .github/workflows/complement_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ jobs:
# real-time.
run: |
set -o pipefail

COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh --in-repo -p 1 -json -run 'TestSynapseVersion/Synapse_version_matches_current_git_checkout' 2>&1 | tee /tmp/gotest-sanity-check-complement.log
shell: bash
env:
Expand All @@ -122,12 +123,26 @@ jobs:
# -json: Output JSON format so that gotestfmt can parse it.
#
Copy link
Copy Markdown
Contributor

@MadLittleMods MadLittleMods Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My vote would be to leave it as-is.

Although, I don't run into problems when viewing the logs now that the formatted view only shows failures.

And re-running with debug logging is another step when I want to see the output which I've wanted to multiple times recently:

On top of it generally being useful of the CI job making progress.

Copy link
Copy Markdown
Contributor Author

@reivilibre reivilibre Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, happy to leave it in if this is something you rely on, by all means this PR was just a convenient place to put that question.

Would be curious if there are other ideas to make this less unbearable.

Suppose we could upload the formatted fail-only logs as a downloadable artifact? Feels like 'a workaround' but one I'm happy to take. Thoughts?

Copy link
Copy Markdown
Contributor

@MadLittleMods MadLittleMods Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to debug some flakes and the webUI to get the logs was just so sluggish.

Viewing this page is laggy? (with only the Formatted Complement test logs (only failing are shown) section expanded) https://github.com/element-hq/synapse/actions/runs/25128863747/job/73673674666

You can expand the ❌ section to see the logs for the test.

GitHub's fake scroll doesn't help with being able to copy-paste everything easily but I don't experience any problems outside of that. The failed section is 19k lines by itself (55k lines total for the entire raw log of the CI job) which is a lot to manage regardless of what we do.

Perhaps some GitHub extension you're using is making it worse than usual?

I tried to download the log file and the download got interrupted, presumably because of the size.
Multiple attempts did not help and it was not resumable.

The raw logs also download fine for me. It's only 16.5MB. You can see that I also did that when figuring out why gotestfmt was failing to archive the logs

upload the formatted fail-only logs as a downloadable artifact

Additionally having GitHub workflow artifacts of whatever logs sounds fine to me.

# tee /tmp/gotest-complement.log: We tee the output to a file so that we can re-process it
# later on for better formatting with gotestfmt. But we still want the command
# later on for better formatting with gotestfmt.
# But we still want the ability for the command
# to output to the terminal as it runs so we can see what's happening in
# real-time.
# When not running in pipeline debug mode, we only show the last 50 lines
# of the JSON output from the build log because it is not human-readable,
# the extreme volume makes it very unwieldy to access in the web UI
# and the resultant log download extremely large.
run: |
set -o pipefail
COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest-complement.log

# To see the full build log, start a re-run with
# the debug mode enabled in the GitHub UI.
Comment on lines +137 to +138
Copy link
Copy Markdown
Contributor

@MadLittleMods MadLittleMods Apr 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make this more clear, to accomplish this, when viewing a CI job, Re-run all jobs -> Scroll down to Enable debug logging and check it -> Re-run jobs

if [ "$RUNNER_DEBUG" = "1" ]; then
debug_pipe="cat"
else
debug_pipe="tail -n 50"
fi

COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh -p 1 -json 2>&1 | tee /tmp/gotest-complement.log | $debug_pipe
shell: bash
env:
POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
Expand All @@ -151,12 +166,26 @@ jobs:
# are underpowered and don't like running tons of Synapse instances at once.
# -json: Output JSON format so that gotestfmt can parse it.
#
# tee /tmp/gotest-in-repo-complement.log: We tee the output to a file so that we can re-process it
# later on for better formatting with gotestfmt. But we still want the command
# tee /tmp/gotest-complement.log: We tee the output to a file so that we can re-process it
# later on for better formatting with gotestfmt.
# But we still want the ability for the command
# to output to the terminal as it runs so we can see what's happening in
# real-time.
# When not running in pipeline debug mode, we only show the last 50 lines
# of the JSON output from the build log because it is not human-readable,
# the extreme volume makes it very unwieldy to access in the web UI
# and the resultant log download extremely large.
run: |
set -o pipefail

# To see the full build log, start a re-run with
# the debug mode enabled in the GitHub UI.
if [ "$RUNNER_DEBUG" = "1" ]; then
debug_pipe="cat"
else
debug_pipe="tail -n 50"
fi

COMPLEMENT_DIR=`pwd`/complement synapse/scripts-dev/complement.sh --in-repo -p 1 -json 2>&1 | tee /tmp/gotest-in-repo-complement.log
shell: bash
env:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ jobs:
- '.ci/**'
- 'scripts-dev/complement.sh'
- '.github/workflows/tests.yml'
- '.github/workflows/complement_tests.yml'

linting:
- 'synapse/**'
Expand Down
1 change: 1 addition & 0 deletions changelog.d/19745.misc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Suppress raw JSON logs in Complement CI when debug disabled.
Loading