Download all images per gbifID with -NN suffix, plus circuit breaker …

[Jiahao419]

This is a consolidation of changes I made to image_install_parallel.py while taking over the pipeline. The new SQLite-tracked downloader in gardoslab/herbaria-data#1 is based on this same logic — opening this PR mainly to record the work in the original repo's history, as Tom suggested.

What changed

• Download every image per gbifID, not just the first. Each image is saved as <gbifID>-00.jpg, <gbifID>-01.jpg, etc. A gbifID is written to processed_ids.txt only when all of its images succeed; partial failures go to failed_ids.txt for retry.
• Hierarchical storage path — <base>/<id[:3]>/<id[3:6]>/<id>-NN.jpg — to avoid millions of files in a single directory.
• Per-host circuit breaker (threshold raised from 50 → 500) plus a timed cooldown for HTTP 429 / timeouts, so one bad host (e.g. swbiodiversity.org, mnhn.fr) can't block the whole run.
• IIIF manifest parsing: manifests are expanded into direct image URLs at 1600 / 1200 / 800 px so the downloader actually has something to fetch.
• Retry-failed-first on startup: IDs in failed_ids.txt are tried before new ones; an ID is removed from failed_ids only on full success.
• Removed the duplicate check against legacy datasets (GBIF-F24, harvard-herbaria) — it was loading those multimedia.txt files at startup, which contributed to the early memory issues.
• .gitignore: exclude local scratch files (backups, dry runs, sorted/merged checkpoint copies).

Status

• Last batch run on this code (job 5752217, 5/20–5/21) added ~620k processed IDs on top of the inherited baseline (13.6M processed, 19.3M failed).
• Memory issues were resolved separately by submitting with -l mem_per_core=8G; no code change needed.
• Job has been canceled so the new SQLite-tracked pipeline can take over.