Upload works and HTTP-TPC pull fails with Authentication Error

[vokac]

It is not clear to me why it is possible to successfully upload data to the dCache 9.2.45, but with same credentials HTTP-TPC pull transfers to (some) directories fails (GGUS ticket), e.g. with

$ export SRC=https://se1.farm.particle.cz/atlas/atlasdatadisk/SAM/1M
$ export DST=https://lcgdpmse.dnp.fmph.uniba.sk/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1
$ export TSRC=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT30M"}' "$SRC" | jq -r '.macaroon')
$ export TDST=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST,MANAGE"], "validity": "PT30M"}' "$DST" | jq -r '.macaroon')

$ python -c "import pymacaroons; print(pymacaroons.Macaroon.deserialize('$TDST').inspect())"
location Optional[/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1]
identifier zrz3a/QT
cid iid:gK6XrQa5
cid id:10038;2003,2003,2000;vokac
cid before:2026-03-03T22:32:16.344341735Z
cid path:/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1
cid activity:UPLOAD,DELETE,LIST,MANAGE
signature ecfca08cecfe08785912d3b67626ebe804489a2ef97367532479743120af3a1c

it is possible to upload

$ curl -v --capath /etc/grid-security/certificates -L -X PUT -H "Authorization: Bearer $TDST" --upload-file /tmp/1M "$DST"
*   Trying 2001:4118:1a:2c10:0:a5:5e:0:443...
* Connected to lcgdpmse.dnp.fmph.uniba.sk (2001:4118:1a:2c10:0:a5:5e:0) port 443 (#0)
...
> PUT /dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1 HTTP/1.1
> Host: lcgdpmse.dnp.fmph.uniba.sk
> User-Agent: curl/7.76.1
> Accept: */*
> Authorization: Bearer SE-Token-DST
> Content-Length: 1048576
> Expect: 100-continue
> 
...
< HTTP/1.1 307 Temporary Redirect
< Date: Tue, 03 Mar 2026 22:03:18 GMT
< Server: dCache/9.2.45
< Location: https://lcgstorage13.dnp.fmph.uniba.sk:24320/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1?dcache-http-uuid=97db49d1-714c-4a7b-b714-afde5e3c04ce&dcache-http-ref=https%3A%2F%2Flcgdpmse.dnp.fmph.uniba.sk%3A443
< Connection: close
< 
...
*   Trying 2001:4118:1a:2c10:0:a5:5e:13:24320...
* Connected to lcgstorage13.dnp.fmph.uniba.sk (2001:4118:1a:2c10:0:a5:5e:13) port 24320 (#1)
...
> PUT /dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1?dcache-http-uuid=97db49d1-714c-4a7b-b714-afde5e3c04ce&dcache-http-ref=https%3A%2F%2Flcgdpmse.dnp.fmph.uniba.sk%3A443 HTTP/1.1
> Host: lcgstorage13.dnp.fmph.uniba.sk:24320
> User-Agent: curl/7.76.1
> Accept: */*
> Content-Length: 1048576
> Expect: 100-continue
> 
...
< HTTP/1.1 201 Created
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 26
< Location: https://lcgdpmse.dnp.fmph.uniba.sk/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1
< X-OC-MTime: accepted
< Server: dCache/9.2.45
< 

but HTTP-TPC fails writing in the same directory

$ curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'RequireChecksumVerification: false' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
*   Trying 2001:4118:1a:2c10:0:a5:5e:0:443...
* Connected to lcgdpmse.dnp.fmph.uniba.sk (2001:4118:1a:2c10:0:a5:5e:0) port 443 (#0)
...
> COPY /dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89/test1 HTTP/1.1
> Host: lcgdpmse.dnp.fmph.uniba.sk
> User-Agent: curl/7.76.1
> Accept: */*
> RequireChecksumVerification: false
> Credential: none
> Authorization: Bearer SE-Token-DST
> TransferHeaderAuthorization: Bearer SE-TokenSRC
> Source: https://se1.farm.particle.cz/atlas/atlasdatadisk/SAM/1M
> 
...
< HTTP/1.1 401 Permission denied
< Date: Tue, 03 Mar 2026 22:02:53 GMT
< Server: dCache/9.2.45
< WWW-Authenticate: Basic realm=""
< Transfer-Encoding: chunked
< 

This directory has following (NFS) ACLs

root@lcgdpmse:~# nfs4_getfacl /mnt/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89
# file: /mnt/dpm/dnp.fmph.uniba.sk/home/atlas/atlasscratchdisk/rucio/data24_13p6TeV/9d/89
A:fd:10022:rwaDdx
A:fdg:2002:rwaDdx
A:fdg:2000:rwaDdx
A:fd:EVERYONE@:rx
A::OWNER@:rwaDxtTcC
A::GROUP@:rwaDxtc
A::EVERYONE@:rxtc

There are directories with same ACLs that works with HTTP-TPC & SE-Tokens, also when I was using OIDC tokens it was even possible to do HTTP-TPC to the directory that doesn't work with SE-Tokens.

I would expect if client have upload permissions than HTTP-TPC pull should never fails with HTTP/1.1 401 Permission denied.

(we observe similar behavior also with a storage that use older dCache 8.2.40)

[vokac]

Any idea how to at least better debug / understand / fix this issue. This is not transient issue, behavior is same even after dCache service restart. How to soleve this mistery

# get ATLAS storage tokens
$ export TDST=$(oidc-token --scope=storage.modify:/ --aud=lcgdpmse.dnp.fmph.uniba.sk ATLAS-storage)
$ export TSRC=$(oidc-token --scope=storage.read:/ --aud=se1.farm.particle.cz ATLAS-storage)

# upload with X.509 VOMS proxy succeeds
$ curl --capath /etc/grid-security/certificates --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) -L -X PUT --upload-file /tmp/1M "$DST"
1048576 bytes uploaded

# HTTP-TPC pull with WLCG JWT token works
$ curl --capath /etc/grid-security/certificates -L -X COPY -H 'RequireChecksumVerification: false' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
success: Created

# HTTP-TPC pull with X.509 VOMS proxy + same WLCG JWT tokens fails (doesn't report `failed: error`)
$ curl -v --capath /etc/grid-security/certificates --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) -L -X COPY -H 'RequireChecksumVerification: false' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
...
< HTTP/1.1 401 login failed
< Date: Fri, 06 Mar 2026 13:57:25 GMT
< Server: dCache/9.2.45
...

# SE-tokens also all fails with HTTP-TPC (but individually works for download & upload)
$ export TSRC=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:DOWNLOAD"], "validity": "PT30M"}' "$SRC" | jq -r '.macaroon')
$ export TDST=$(curl --silent --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates -X POST -H 'Content-Type: application/macaroon-request' -d '{"caveats": ["activity:UPLOAD,DELETE,LIST,MANAGE"], "validity": "PT30M"}' "$DST" | jq -r '.macaroon')
$ curl -v --capath /etc/grid-security/certificates -L -X COPY -H 'RequireChecksumVerification: false' -H 'Credential: none' -H "Authorization: Bearer $TDST" -H "TransferHeaderAuthorization: Bearer $TSRC" -H "Source: $SRC" "$DST"
...
< HTTP/1.1 401 Permission denied
< Date: Fri, 06 Mar 2026 14:02:56 GMT
< Server: dCache/9.2.45
...

Where this "HTTP/1.1 401 Permission denied" comes from? I guess this is some validation before even trying to touch source with HTTP-TPC, because than I would expect HTTP 200 with failure: error in the body. But why this validation fails with COPY while everything works fine with PUT and same credentials?