|
1 | 1 | """JSON serialization of Scrapy requests and cached responses for storage on the Apify platform. |
2 | 2 |
|
3 | | -Scrapy requests and cached responses are stored in the Apify request queue and key-value store which hold JSON, |
4 | | -so they are serialized as JSON here rather than pickled. |
5 | | -
|
6 | | -Only `body` (`bytes`) and `headers` (`{bytes: [bytes]}`) are not natively JSON-serializable; both sit at fixed keys |
7 | | -and are base64-encoded in place. A `str` `body` is encoded as its UTF-8 bytes and comes back as `bytes`, matching |
8 | | -Scrapy, which always stores `body` as `bytes`. Pydantic models such as Crawlee's `UserData` are dumped via |
9 | | -`model_dump(mode='json')`, so model fields JSON cannot natively represent (e.g. `datetime`) are stored in their |
10 | | -JSON form. Everything else, notably `meta` and `cb_kwargs`, must already be JSON-serializable, otherwise |
11 | | -serialization fails with a clear error naming the offending value. No in-band sentinel is used, so no user value |
12 | | -can collide with the encoding. |
13 | | -
|
14 | | -Known limitations of the pickle -> JSON switch (a documented breaking change): JSON has fewer types than pickle, |
15 | | -so values in `meta`/`cb_kwargs` are subject to JSON's coercions. A `tuple` round-trips as a `list` and non-string |
16 | | -`dict` keys round-trip as strings (e.g. `{1: 'a'}` becomes `{'1': 'a'}`). Values JSON cannot represent at all |
17 | | -(`datetime`, `set`, `Decimal`, arbitrary objects, ...) are not coerced silently: serialization raises and the request |
18 | | -is skipped loudly rather than stored in a corrupted form. |
| 3 | +Scrapy requests and cached responses are stored in the Apify request queue and key-value store, which hold JSON. |
| 4 | +They are therefore serialized as JSON here. |
| 5 | +
|
| 6 | +Only `body` (`bytes`) and `headers` (`{bytes: [bytes]}`) are not natively JSON-serializable. Both live at fixed keys |
| 7 | +and are base64-encoded in place. A `str` `body` is encoded as UTF-8 bytes and deserialized as `bytes`, matching |
| 8 | +Scrapy, which always stores `body` as `bytes`. |
| 9 | +
|
| 10 | +Pydantic models, such as Crawlee's `UserData`, are dumped with `model_dump(mode='json')`, which converts |
| 11 | +non-JSON-native fields into JSON-compatible values. For example, `datetime` fields are stored as ISO-8601 strings. |
| 12 | +
|
| 13 | +All other values, notably `meta` and `cb_kwargs`, must already be JSON-serializable. Non-JSON-serializable values, |
| 14 | +such as `datetime`, `set`, `Decimal`, or arbitrary objects, fail serialization with a clear error naming the |
| 15 | +offending value. The request is skipped rather than stored in a corrupted form. |
| 16 | +
|
| 17 | +No in-band sentinel is used, so user values cannot collide with the encoding. |
| 18 | +
|
| 19 | +Known JSON limitations: values in `meta` and `cb_kwargs` are subject to JSON coercions. A `tuple` round-trips as |
| 20 | +a `list`, and non-string `dict` keys round-trip as strings; for example, `{1: 'a'}` becomes `{'1': 'a'}`. |
19 | 21 | """ |
20 | 22 |
|
21 | 23 | from __future__ import annotations |
|
0 commit comments