hostedreap

Deletes hosted feature layer rows that no longer exist in your source file.

The Append REST operation has no delete semantics. upsert inserts and updates, skipDeletes only honours delete records already carried inside a sync payload, and nothing in the parameter list infers "this row vanished from the source". So after an upsert-append your hosted layer keeps every row the source dropped. This tool does the delete half only, backing up full rows with geometry first, and it refuses when truncate-and-reload is the better answer.

Read When to use truncate and reload instead before you use this. Most layers do not need a diff-delete.

$ python hostedreap.py --self-test
self-test OK: 125 assertions passed, 0 network calls, 0 credentials.

Requirements

Python 3.8 or newer. Standard library only for everything except password sign-in.

If you export a token, no third-party package is needed at all. The arcgis package is imported only inside the sign-in function, guarded by try/except ModuleNotFoundError, and only when you pass --username. If it is missing the tool tells you to export a token or re-run with "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe".

Quick start

python hostedreap.py --self-test

That runs the full offline suite: no network, no credentials, no ArcGIS Online organisation. Then a real dry run against your own layer:

set ARCGIS_TOKEN=<your token>
python hostedreap.py --layer-url https://.../FeatureServer/0 --key-field ASSET_ID ^
  --source keep.csv --source-field ASSET_ID

Dry run is the default. Nothing is deleted until you add --apply.

Usage

Flag	Default	What it does
--layer-url	required	Layer endpoint, e.g. https://.../FeatureServer/0
--key-field	required	Field in the layer that joins to the source
--source	required	.csv, .json, or one-key-per-line file of keys that still exist
--source-field	none	Column name inside a csv source, or key inside a json list of objects
--apply	off	Actually delete. Without it the run is a dry run
--max-delete-fraction	0.20	Refuse if the diff deletes more than this share of matchable rows
--easy-case-max-rows	50000	Row ceiling below which truncate-and-reload is judged simpler
--ignore-easy-case	off	Proceed even when truncate-and-reload would be simpler
--allow-null-keys	off	Proceed when some layer rows have a null or blank key
--case-insensitive	off	Fold key case when matching
--page-size	2000	Rows per read window, capped at the layer's maxRecordCount
--delete-chunk-size	500	OIDs per deleteFeatures request
--backup-dir	hostedreap_backups	Where the pre-delete backup is written
--resume BACKUP.json	none	Finish an interrupted run from its backup. Requires --apply
--unregister-replicas	off	Unregister sync replicas first. Breaks every checked-out offline map
--oid-field	layer's own	Override the OBJECTID field name
--portal	https://www.arcgis.com	Portal URL for --username sign-in
--token-env	ARCGIS_TOKEN	Environment variable holding a token
--username	none	Sign in interactively instead of using a token
--password-env	ARCGIS_PASSWORD	Environment variable holding the password
--self-test	off	Offline assertions, no network
--version		Print the version

Exit codes: 0 success or clean dry run, 1 error or a failed delete chunk, 3 refused.

Configuration

Credential precedence, highest first:

1. $ARCGIS_TOKEN (or whatever --token-env names).
2. --username plus $ARCGIS_PASSWORD (or whatever --password-env names).
3. --username plus an interactive getpass prompt.

No credential is ever written to a file, logged, or printed. Request parameters are never logged, because they carry the token.

No environment variable can enable deleting. Deleting requires --apply on the command line, every time. There is no config file, no HOSTEDREAP_* override, and no way for a scheduled job to inherit destructive behaviour from its environment.

Why the obvious version is wrong

The obvious version is query(where="1=1"), set-difference the keys, delete_features(where=...). Three things break it.

where=1=1 times out. On a layer of any size a single sweep either times out or comes back truncated, and resultOffset paging drifts and repeats rows while the layer is edited underneath you. This tool walks (low, high] OID windows instead, splits any window the server flags as truncated, then reconciles the total read against returnCountOnly. A read that comes up short raises rather than diffing, because a short read nominates live rows for deletion.

Empty result pages read as success. A deleteFeatures response with an empty deleteResults list, or one that silently omits an OID you asked about, is a full-chunk failure. Treating either as done is exactly how a nightly job reports success while the rows are still there. Every OID sent must come back with success: true or it is counted failed and left for --resume.

Key types do not match. A layer Double reads back as 5.0 and the same id exported to CSV is the text "5.0". Comparing raw values nominates every such row for deletion. Keys are normalised so 5, 5.0, "5.0" and " 5 " are one key, while "A5.0", "1.2.3" and "007" stay literal.

Sync replicas block bulk deletes. A large delete against a sync-enabled service bloats the delta tables and can fail mid-run. The tool refuses when replicas are registered, and with --unregister-replicas it unregisters, disables sync, deletes, then restores sync in a finally block so a failed run does not leave sync off.

When to use truncate and reload instead

If your layer supports truncate, has no attachments, no relationships, no sync, and holds fewer than --easy-case-max-rows rows, do not use this tool. Truncate and reload is one atomic step, it cannot half-succeed, and it needs no diff:

layer.manager.truncate()
layer.append(item_id=<uploaded item>, upload_format='filegdb', source_table_name=<name>)

hostedreap detects that case and refuses with exit code 3, printing those two lines. Diff-delete is worth its complexity only when you are keeping OBJECTIDs, editor tracking, attachments, relationship classes, or a layer too large to reload. Pass --ignore-easy-case when that is a deliberate choice.

Limitations

• Attribute join only. There is no spatial or fuzzy matching, and no composite key: one field joins to one source column.
• The layer must not be edited during the run. The read reconciliation catches concurrent edits and aborts, rather than diffing against a moving target.
• Restore is manual. The backup is Esri feature JSON with geometry, complete enough to append back, but the tool does not do the appending.
• Attachments are not backed up. Rows carrying attachments lose them permanently on delete.
• The --max-delete-fraction guard uses matchable rows, so a layer where most keys are null offers much weaker protection than the row count suggests.

Contributing

Issues and pull requests welcome. Run python hostedreap.py --self-test before opening one, and add assertions for whatever you changed. The suite must stay offline.

Author

Built by Asir Khan.

License

MIT.

Related

Other single-file tools in this portfolio that pair with this one:

• restfake - a fake service to rehearse the delete against before you point it at a real one
• safe-republish - the same refusal applied to a truncate and append