Enforce disk resource limits without causing performance regression

[dralley]

#7113 was added to mitigate against #7064

Unfortunately it is a bit heavy-handed and comes with a performance cost as a result of using unnecessarily small batch sizes by default. Using smaller batches prevents so many different packages from being in-flight at once (consuming disk space), but it comes at the cost of more round-trips to the database.

Ideally, we would have a resource limitation mechanism which adjusts dynamically, rather than simply using a batch size selected for the worst-case.