Skip to content

Commit 862f29e

Browse files
committed
better wording & description
1 parent 140fff9 commit 862f29e

2 files changed

Lines changed: 20 additions & 18 deletions

File tree

docs/04_upgrading/upgrading_to_v4.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ Pickle could store arbitrary Python objects. JSON cannot, so the values in a req
218218

219219
- A `tuple` comes back as a `list`.
220220
- Non-string `dict` keys come back as strings, so `{1: 'a'}` becomes `{'1': 'a'}`.
221-
- A value JSON cannot represent (`datetime`, `set`, `Decimal`, a custom object) is no longer stored silently. The request is skipped and the failure is logged. Pydantic models are still supported and are dumped with `model_dump(mode='json')`, so model fields JSON cannot natively represent (such as `datetime`) are stored in their JSON form.
221+
- Non-JSON-serializable values, such as `datetime`, `set`, `Decimal`, or custom objects, are skipped and logged. Pydantic models are supported via `model_dump(mode='json')`, which converts non-JSON-native fields into JSON-compatible values, such as ISO-8601 strings for `datetime` fields.
222222

223223
Convert such values to a JSON-friendly form before yielding the request:
224224

@@ -230,4 +230,4 @@ yield scrapy.Request(url, meta={'since': datetime(2024, 1, 1)})
230230

231231
# After (v4): store a JSON-serializable value.
232232
yield scrapy.Request(url, meta={'since': datetime(2024, 1, 1).isoformat()})
233-
```
233+
```Pickle

src/apify/scrapy/_serialization.py

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,23 @@
11
"""JSON serialization of Scrapy requests and cached responses for storage on the Apify platform.
22
3-
Scrapy requests and cached responses are stored in the Apify request queue and key-value store which hold JSON,
4-
so they are serialized as JSON here rather than pickled.
5-
6-
Only `body` (`bytes`) and `headers` (`{bytes: [bytes]}`) are not natively JSON-serializable; both sit at fixed keys
7-
and are base64-encoded in place. A `str` `body` is encoded as its UTF-8 bytes and comes back as `bytes`, matching
8-
Scrapy, which always stores `body` as `bytes`. Pydantic models such as Crawlee's `UserData` are dumped via
9-
`model_dump(mode='json')`, so model fields JSON cannot natively represent (e.g. `datetime`) are stored in their
10-
JSON form. Everything else, notably `meta` and `cb_kwargs`, must already be JSON-serializable, otherwise
11-
serialization fails with a clear error naming the offending value. No in-band sentinel is used, so no user value
12-
can collide with the encoding.
13-
14-
Known limitations of the pickle -> JSON switch (a documented breaking change): JSON has fewer types than pickle,
15-
so values in `meta`/`cb_kwargs` are subject to JSON's coercions. A `tuple` round-trips as a `list` and non-string
16-
`dict` keys round-trip as strings (e.g. `{1: 'a'}` becomes `{'1': 'a'}`). Values JSON cannot represent at all
17-
(`datetime`, `set`, `Decimal`, arbitrary objects, ...) are not coerced silently: serialization raises and the request
18-
is skipped loudly rather than stored in a corrupted form.
3+
Scrapy requests and cached responses are stored in the Apify request queue and key-value store, which hold JSON.
4+
They are therefore serialized as JSON here.
5+
6+
Only `body` (`bytes`) and `headers` (`{bytes: [bytes]}`) are not natively JSON-serializable. Both live at fixed keys
7+
and are base64-encoded in place. A `str` `body` is encoded as UTF-8 bytes and deserialized as `bytes`, matching
8+
Scrapy, which always stores `body` as `bytes`.
9+
10+
Pydantic models, such as Crawlee's `UserData`, are dumped with `model_dump(mode='json')`, which converts
11+
non-JSON-native fields into JSON-compatible values. For example, `datetime` fields are stored as ISO-8601 strings.
12+
13+
All other values, notably `meta` and `cb_kwargs`, must already be JSON-serializable. Non-JSON-serializable values,
14+
such as `datetime`, `set`, `Decimal`, or arbitrary objects, fail serialization with a clear error naming the
15+
offending value. The request is skipped rather than stored in a corrupted form.
16+
17+
No in-band sentinel is used, so user values cannot collide with the encoding.
18+
19+
Known JSON limitations: values in `meta` and `cb_kwargs` are subject to JSON coercions. A `tuple` round-trips as
20+
a `list`, and non-string `dict` keys round-trip as strings; for example, `{1: 'a'}` becomes `{'1': 'a'}`.
1921
"""
2022

2123
from __future__ import annotations

0 commit comments

Comments
 (0)