Large file sync uploads fill host disk with temporary files in the Apache container

[svefro]

⚠️ This issue respects the following points: ⚠️

• This is not a troubleshooting question, general support matter, or webserver/proxy problem, but likely a bug (if unsure, ask the Community Help Forum).
• This issue is not already reported on Github OR solved at the Community Help Forum (I've searched!).
• I'm using a maintained major version of Nextcloud Server and tested against the latest patch level. (Supported major versions and current patch levels).
• I agree to follow Nextcloud's Code of Conduct.
• I've tried my best to provide clear reproduction steps that someone unfamiliar with this bug could use to reproduce it.

Bug description

While syncing large amounts of files (about 30 to 80 GB of Photos) from the windows sync clients
Nextcloud consumes all available disk space on the host disk and stops.

i'm using nexcloud-aio in docker and have the data dir on another drive.
the drive that fills up is the one containing the host os. not the one that contains the data folder.

Most of the disk space is consumed in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/744/fs
I think this is the apache containers filesystem

Steps to reproduce

1.Syncing large amounts of files (about 30 to 80 GB of Photos) from the windows sync client
2. observe diskusage by running sudo ncdu / --exclude /mnt at regular intervals, navigate to /var/lib/ and follow the disk usage
3. /tmp folder grow to fill all available disk space
4. docker containers crash
5. if i manage to restart all containers before diskspace is consumed, diskspace is reclaimed and everything works. i also have to retry, or stop + start the sync client
6. if i wait until all disk usage is consumed. im unable to restart the docker containers due to no disk space.
7. i have to restart lxc in proxmox, after restart i open aio interface and press stop all containers, then start all containers
8. diskusage goes from 100% (40gig) down to about 25Gig

Expected behavior

nextcloud should clear temp folder before consuming all available space. admin option to set limit.
it needed the sync client should stop uploading files untill they are processed on the server.

Nextcloud Server version

35

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.5

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

running the docker aio image. no php on host

List of activated Apps

running the docker aio image. no php on host

Nextcloud Signing status

reports No errors have been found. right now but that is just because of a restart just now

Nextcloud Logs

not avaliable right now. will come back to fill in

Additional info

No response

[joshtrichards]

I can't think of a scenario off-hand where Nextcloud creates temporary filenames with that exact syntax in the configured temp directory. Nextcloud's temp files typically have a prefix like oc_tmp. The filesize also doesn't match the default upload chunking size.

running the docker aio image. no php on host

See https://github.com/nextcloud/all-in-one#how-to-run-occ-commands

i'm using nexcloud-aio in docker and have the data dir on another drive.

Can you be more precise? For example, do you mean the data directory at the container environment level or Nextcloud's datadirectory parameter or something else? Can you provide the Compose you're using to run AIO?

the drive that fills up is the one containing the host os. not the one that contains the data folder.
If that container is, indeed

Most of the disk space is consumed in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/744/fs
I think this is the apache containers filesystem

I think you can determine for certain which container via the mount command output on your host (or maybe ctr).

AIO does explicitly not utilize /tmp on the specified data directory so the fallback could end up on the overlay fs I guess, but that would only be applicable to the nextcloud app container not the apache one:

https://github.com/nextcloud/all-in-one/blob/75c29db47849df004a4fc53b7abc7b84c09636fd/Containers/nextcloud/entrypoint.sh#L606-L612

[svefro]

After all files were uploaded, this stopped happening. disk usage in lxc is now stable at 25gb.

The infinite diskusage was on the lxc disk. not the nextcloud data directory. the data directory is mounted from another disk on the proxmox host.

For information, it was about 500gigs of photos, videos +++

I assumed it was the apache container since the files was prefixed with php.

I'm going to switch to a vm. set everything up again, and do the same file upload.

I found out after i reported this that nextcloud had som issues downloading and writing a cities file.
This was due to permission issues in the data dir.

Could that have caused it? if it downloads it to tmp folder, but is unable to save it in data, then after a while downloads it again. and every time leaves a temp file???