Notifications via docker compose question

[cookiemonsteruk]

Hello. I wonder if someone is willing to assist with this query.
I have just moved from the AnalogJ repo in the last 24-36 hours by replacing all the binaries on the endpoints and changing the image in the docker compose file. I use binaries with scheduled tasks on them to send and a hub with docker compose. I do not use config.yaml files on any of them. All using defaults except the hub which is using a docker compose file.
This is the setup that has worked before for notifications. I am using mattermost, hosted as another container on the same host i.e. sits alongside scrutiny.
However this is my problem, the notifications aren't working. Testing with curl -X POST http://localhost:8080/api/health/notify on the docker container of the hub errors with {"errors":["failed to send using mattermost: timed out"],"success":false}.
Frustratingly there are no errors on the mattermost server that I can spot.
I am hoping that my problem is that I need to change something on the compose for this repo.

This is the setting in my docker compose:

include:
  - path: ../mattermost/docker/docker-compose.yml
  - path: ../mattermost/docker/docker-compose.without-nginx.yml
    env_file: ../mattermost/docker/.env

services:
  scrutiny:
    restart: unless-stopped
    container_name: scrutiny
#    image: ghcr.io/analogj/scrutiny:master-omnibus
    image: ghcr.io/starosdev/scrutiny:latest-omnibus
    cap_add:
      - SYS_ADMIN
    ports:
      - "8080:8080" # webapp
      - "8086:8086" # influxDB admin
    volumes:
      - /run/udev:/run/udev:ro
      - ./config:/opt/scrutiny/config
      - ./influxdb:/opt/scrutiny/influxdb
    environment:
    #  notify:
    #    urls:
      - SCRUTINY_NOTIFY_URLS=mattermost://mydomain/x3zfc96h7brizyk9d1xy7hbgrc/opsnotify

As mentioned earlier. This worked fine when the image was analogj. That was the only change on the compose file above.

Additional info:
A test succeeds from outside scrutiny, still inside the host:
$ curl -i -X POST -H 'Content-Type: application/json' -d '{"text": "Hello, this is some text\nThis is more text. :tada:"}' https://mydomain/hooks/x3zfc96h7brizyk9d1xy7hbgrc

HTTP/2 200
content-type: text/plain
permissions-policy:
referrer-policy: no-referrer
vary: Accept-Encoding
x-content-type-options: nosniff
x-request-id: u1mcu3atetdimrjrk47nffqydc
x-version-id: 10.11.8.19424819464.6158b7bb5e5b8e159789e6fee16b9e07.false
date: Tue, 03 Mar 2026 12:54:51 GMT
content-length: 2
strict-transport-security: max-age=63072000; includeSubDomains; preload
ok

this confirms the webhook is working, can receive messages. I have tested with and without the "hooks" element in the url, which when getting it to work with analogj proved to need to be dropped from the scrutiny configuration in the compose.
Logs for this successful test in mattermost:
{"timestamp":"2026-03-03 12:54:51.973 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","method":"POST","url":"/hooks/x3zfc96h7brizyk9d1xy7hbgrc","request_id":"u1mcu3atetdimrjrk47nffqydc","status_code":"200"}

This now is a test with the health/notify endpoint and logs from the time window. In case it helps. We can ignore the "your license does not support..." as is I believe an outstanding mattermost issue.
Sending:

penguin@vanth1:~/scrutiny$ date
Tue Mar  3 13:13:39 UTC 2026
penguin@vanth1:~/scrutiny$ curl -X POST http://localhost:8080/api/health/notify
{"errors":["failed to send using mattermost: timed out"],"success":false}penguin@vanth1:~/scrutiny$
penguin@vanth1:~/scrutiny$ date
Tue Mar  3 13:13:58 UTC 2026

Receiving end:

{"timestamp":"2026-03-03 13:13:36.395 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"POST","url":"/api/v4/users/status/ids","request_id":"xyy193djofr63fdi9uzpg9ofih","user_id":"d7g4ik9rrbr35bnfz9o3
joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:13:52.568 Z","level":"debug","msg":"Able to write to database.","caller":"api4/system.go:194
","path":"/api/v4/system/ping","request_id":"jzbnjuusbpbotmhejt1hoetzie","ip_addr":"","user_id":"","method":"GET"}
{"timestamp":"2026-03-03 13:13:52.568 Z","level":"debug","msg":"Able to write files to local storage.","caller":"filesto
re/localstore.go:83"}
{"timestamp":"2026-03-03 13:13:52.568 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/system/ping","request_id":"jzbnjuusbpbotmhejt1hoetzie","status_code":"200"}
{"timestamp":"2026-03-03 13:14:04.396 Z","level":"debug","msg":"websocket.NextReader: closing websocket","caller":"platf
orm/web_conn.go:1032","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","conn_id":"f8x537odrbrdmgfkppnhbzgywe","origin_client":"web
","error":"read tcp 172.18.0.4:8065->192.168.5.1:62630: read: connection reset by peer"}
{"timestamp":"2026-03-03 13:14:04.396 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/websocket","request_id":"hnykdd7ju7dqbmc1otqtoryrcr","user_id":"d7g4ik9rrbr35bnfz9o3joaefe"}
{"timestamp":"2026-03-03 13:14:09.541 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/users/me/channels","request_id":"m5ruo4nnatdh7jseoogxx9rxzh","user_id":"d7g4ik9rrbr35bnfz9o3
joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.542 Z","level":"debug","msg":"Scheduled posts feature requires a license","caller":"we
b/context.go:118","path":"/api/v4/posts/scheduled/team/kubikbbprpfsjqqwmbs58gcx1c","request_id":"hre8i4q4s38yjcoy16asb6e
mxh","ip_addr":"192.168.5.1","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","method":"GET","err_where":"","http_code":400,"error
":"Scheduled posts feature requires a license"}
{"timestamp":"2026-03-03 13:14:09.542 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/posts/scheduled/team/kubikbbprpfsjqqwmbs58gcx1c","request_id":"hre8i4q4s38yjcoy16asb6emxh","
user_id":"d7g4ik9rrbr35bnfz9o3joaefe","status_code":"400"}
{"timestamp":"2026-03-03 13:14:09.549 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/users/d7g4ik9rrbr35bnfz9o3joaefe/channel_members","request_id":"tzpfx9my7id7frngw9t3rjxcrr",
"user_id":"d7g4ik9rrbr35bnfz9o3joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.550 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/users/d7g4ik9rrbr35bnfz9o3joaefe/teams/kubikbbprpfsjqqwmbs58gcx1c/channels/categories","requ
est_id":"u3agbw365jyw8x3nrxz39f8q7o","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.553 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/channels/n9xnj6sxzfy1bqr4z5ybhnryzc/posts","request_id":"6cq8xz8e4pdnzyt7ifqcdgtfcy","user_i
d":"d7g4ik9rrbr35bnfz9o3joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.563 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/plugins/webapp","request_id":"yago18jxebyiif5gipawf9zorr","user_id":"d7g4ik9rrbr35bnfz9o3joa
efe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.566 Z","level":"debug","msg":"Received HTTP request","caller":"app/plugin_api.go:1119"
,"plugin_id":"playbooks","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0"
,"method":"GET","url":"/api/v0/runs?page=0&participant_id=d7g4ik9rrbr35bnfz9o3joaefe&per_page=0&team_id=kubikbbprpfsjqqw
mbs58gcx1c","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","request_id":"h4grz4ezkinjuxgii4fj15bjey","plugin_caller":"github.com
/mattermost/mattermost-plugin-playbooks/server/api/logger.go:44"}
{"timestamp":"2026-03-03 13:14:09.567 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/users/me/teams/unread","request_id":"8xo7bkn86t8ixewrrguzakq81y","user_id":"d7g4ik9rrbr35bnf
z9o3joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:09.572 Z","level":"debug","msg":"Handled HTTP request","caller":"app/plugin_api.go:1119",
"plugin_id":"playbooks","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","request_id":"h4grz4ezkinjuxgii4fj15bjey","user_agent":"M
ozilla/5.0 (Windows NT 10.0; Win64; x64; rv:148.0) Gecko/20100101 Firefox/148.0","method":"GET","time":"7","status":"200
","url":"/api/v0/runs?page=0&participant_id=d7g4ik9rrbr35bnfz9o3joaefe&per_page=0&team_id=kubikbbprpfsjqqwmbs58gcx1c","p
lugin_caller":"github.com/mattermost/mattermost-plugin-playbooks/server/api/logger.go:57"}
{"timestamp":"2026-03-03 13:14:09.582 Z","level":"debug","msg":"Your license does not support Custom Profile Attributes.
","caller":"web/context.go:118","path":"/api/v4/custom_profile_attributes/fields","request_id":"f3j4awe3eidg7dr4hfoqxrop
ao","ip_addr":"192.168.5.1","user_id":"d7g4ik9rrbr35bnfz9o3joaefe","method":"GET","err_where":"Api4.listCPAFields","http
_code":403,"error":"Api4.listCPAFields: Your license does not support Custom Profile Attributes."}
{"timestamp":"2026-03-03 13:14:09.582 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"GET","url":"/api/v4/custom_profile_attributes/fields","request_id":"f3j4awe3eidg7dr4hfoqxropao","user_id":"d7g4i
k9rrbr35bnfz9o3joaefe","status_code":"403"}
{"timestamp":"2026-03-03 13:14:09.583 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"POST","url":"/api/v4/users/ids","request_id":"6f3iweq7g3dhtrsq3i59ydy4qa","user_id":"d7g4ik9rrbr35bnfz9o3joaefe"
,"status_code":"200"}
{"timestamp":"2026-03-03 13:14:10.620 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"POST","url":"/api/v4/users/status/ids","request_id":"41skaidbpfb4bb1e35ihn5iasy","user_id":"d7g4ik9rrbr35bnfz9o3
joaefe","status_code":"200"}
{"timestamp":"2026-03-03 13:14:13.649 Z","level":"debug","msg":"Received HTTP request","caller":"web/handlers.go:175","m
ethod":"POST","url":"/api/v4/users/status/ids","request_id":"oakhby53tj8qzq3cqbfxb3rtrh","user_id":"d7g4ik9rrbr35bnfz9o3
joaefe","status_code":"200"}

[cookiemonsteruk]

Problem solved. Or rather, the testing was flawed.
Unbeknownst to me the place where to test from matters. If I send the test from the scrutiny directory for docker, it fails.

|-- mattermost
|   `-- docker
|-- scrutiny
|   |-- compose.yaml
|   |-- compose.yaml.backup
|   |-- config
|   `-- influxdb
`-- scrutiny-backup
    |-- compose.yaml
    |-- compose.yaml.backup
    |-- config
    `-- influxdb

If I send the test from the parent directory, it succeeds. Clearly the ability to connect to localhost:8080 is something I need to understand better but this query can be closed for now. Thanks for reading.