Does not return proper prev_batch on /sync

[kroeckx]

I'm not sure what exactly synapse is returning, but whatever it at least seems it doesn't follow the specs.

This related to Sytest's GET /rooms/:room_id/messages returns a message test, and matrix-org/sytest#1083

What the test does is, as returned by Conduit:

> client/r0/register {"password":"an0th3r s3kr1t","username":"anon-20210803_173811-34","bind_email":false,"auth":{"type":"m.login.dummy"}}
<  200 {"access_token":"dB[...]","user_id":"@anon-20210803_173811-34:localhost:8800","device_id":"UNsEkGLo
TK"}
> client/r0/createRoom {"visibility":"private","preset":"public_chat"}
< 200 {"room_id":"!z3iyEeHFmrLTUAwQUF:localhost:8800"}
> client/r0/sync filter={"room":{"rooms":[]},"account_data":{},"presence":{"types":[]}}, set_presence=offline
< 200 {"next_batch":"860","rooms":{"join":{"!z3iyEeHFmrLTUAwQUF:localhost:8800":{"summary":{"m.joined_member_count":1,"m.invited_member_count":0},"unread_notifications":{"highlight_count":0,"notification_count":0},"timeline":{"limited":true,"prev_batch":"829","events":[{"content":{"creator":"@anon-20210803_173811-34:localhost:8800","room_version":"6"},"event_id":"$fmcyYiUPeJiC972jVrZqllGUMJ8DJ2ziwLxX93hEKoQ","origin_server_ts":1628012337746,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.create","unsigned":{}},{"content":{"is_direct":false,"membership":"join"},"event_id":"$JSkeEV_Cq0aexMV16JJDMmcO-fz2Qns3POa0GBGXAwM","origin_server_ts":1628012337762,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"@anon-20210803_173811-34:localhost:8800","type":"m.room.member","unsigned":{}},{"content":{"users":{"@anon-20210803_173811-34:localhost:8800":100}},"event_id":"$s60zf9B2hYfQOxs6RdOiQpQeETLkXc66Rni6wB3xbVs","origin_server_ts":1628012337778,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.power_levels","unsigned":{}},{"content":{"join_rule":"public"},"event_id":"$LcE6-fL4W-81Fu5gEdZTDcqQd9qIwCdNKnRO_SH2Irw","origin_server_ts":1628012337792,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.join_rules","unsigned":{}},{"content":{"history_visibility":"shared"},"event_id":"$Bto6KcfZ9_wWHv3K141d35SkE-KKx9yNn6U7vFhchhg","origin_server_ts":1628012337808,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.history_visibility","unsigned":{}},{"content":{"guest_access":"forbidden"},"event_id":"$iK5LimMns1J3o3RSQ3QV8a8iuhcu_NTo13g15M7iJgY","origin_server_ts":1628012337824,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.guest_access","unsigned":{}}]},"state":{"events":[{"content":{"history_visibility":"shared"},"event_id":"$Bto6KcfZ9_wWHv3K141d35SkE-KKx9yNn6U7vFhchhg","origin_server_ts":1628012337808,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.history_visibility","unsigned":{}},{"content":{"is_direct":false,"membership":"join"},"event_id":"$JSkeEV_Cq0aexMV16JJDMmcO-fz2Qns3POa0GBGXAwM","origin_server_ts":1628012337762,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"@anon-20210803_173811-34:localhost:8800","type":"m.room.member","unsigned":{}},{"content":{"join_rule":"public"},"event_id":"$LcE6-fL4W-81Fu5gEdZTDcqQd9qIwCdNKnRO_SH2Irw","origin_server_ts":1628012337792,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.join_rules","unsigned":{}},{"content":{"creator":"@anon-20210803_173811-34:localhost:8800","room_version":"6"},"event_id":"$fmcyYiUPeJiC972jVrZqllGUMJ8DJ2ziwLxX93hEKoQ","origin_server_ts":1628012337746,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.create","unsigned":{}},{"content":{"guest_access":"forbidden"},"event_id":"$iK5LimMns1J3o3RSQ3QV8a8iuhcu_NTo13g15M7iJgY","origin_server_ts":1628012337824,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.guest_access","unsigned":{}},{"content":{"users":{"@anon-20210803_173811-34:localhost:8800":100}},"event_id":"$s60zf9B2hYfQOxs6RdOiQpQeETLkXc66Rni6wB3xbVs","origin_server_ts":1628012337778,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.power_levels","unsigned":{}}]}}}},"account_data":{"events":[{"content":{"global":{"content":[{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight"}],"default":true,"enabled":true,"pattern":"anon-20210803_173811-34","rule_id":".m.rules.contains_user_name"}],"override":[{"actions":["dont_notify"],"conditions":[],"default":true,"enabled":false,"rule_id":".m.rule.master"},{"actions":["dont_notify"],"conditions":[{"key":"content.msgtype","kind":"event_match","pattern":"m.notice"}],"default":true,"enabled":true,"rule_id":".m.rule.suppress_notices"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.member"},{"key":"content.membership","kind":"event_match","pattern":"invite"},{"key":"state_key","kind":"event_match","pattern":"@anon-20210803_173811-34:localhost:8800"}],"default":true,"enabled":true,"rule_id":".m.rule.invite_for_me"},{"actions":["dont_notify"],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.member"}],"default":true,"enabled":true,"rule_id":".m.rule.member_event"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight"}],"conditions":[{"kind":"contains_display_name"}],"default":true,"enabled":true,"rule_id":".m.rule.contains_display_name"},{"actions":["notify",{"set_tweak":"highlight"}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.tombstone"},{"key":"state_key","kind":"event_match","pattern":""}],"default":true,"enabled":false,"rule_id":".m.rule.tombstone"},{"actions":["notify",{"set_tweak":"highlight"}],"conditions":[{"key":"content.body","kind":"event_match","pattern":"@room"},{"key":"room","kind":"sender_notification_permission"}],"default":true,"enabled":true,"rule_id":".m.rule.roomnotif"},{"actions":["dont_notify"],"conditions":[{"key":"type","kind":"event_match","pattern":"m.reaction"}],"default":true,"enabled":true,"rule_id":".m.rule.reaction"}],"room":[],"sender":[],"underride":[{"actions":["notify",{"set_tweak":"sound","value":"ring"},{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.call.invite"}],"default":true,"enabled":true,"rule_id":".m.rules.call"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"is":"2","kind":"room_member_count"},{"key":"type","kind":"event_match","pattern":"m.room.encrypted"}],"default":true,"enabled":true,"rule_id":".m.rules.encrypted_room_one_to_one"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"is":"2","kind":"room_member_count"},{"key":"type","kind":"event_match","pattern":"m.room.message"}],"default":true,"enabled":true,"rule_id":".m.rules.room_one_to_one"},{"actions":["notify",{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.message"}],"default":true,"enabled":true,"rule_id":".m.rules.message"},{"actions":["notify",{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.encrypted"}],"default":true,"enabled":true,"rule_id":".m.rules.encrypted"}]}},"type":"m.push_rules"}]}}
> client/r0/rooms/!z3iyEeHFmrLTUAwQUF:localhost:8800/send/m.room.test/1000007 {}
< 200 {"event_id":"$aVbO5UM8PBIdJZr8MN3vXxSWLcqsjhJZ8ivUjX4bzrw"}
> client/r0/sync?timeout=1000&since=860&set_presence=offline
< 200 {"next_batch":"865","rooms":{"join":{"!z3iyEeHFmrLTUAwQUF:localhost:8800":{"unread_notifications":{"highlight_count":0,"notification_count":0},"timeline":{"prev_batch":"865","events":[{"content":{},"event_id":"$aVbO5UM8PBIdJZr8MN3vXxSWLcqsjhJZ8ivUjX4bzrw","origin_server_ts":1628012337925,"sender":"@anon-20210803_173811-34:localhost:8800","type":"m.room.test","unsigned":{"transaction_id":"1000007"}}]},"state":{"events":[{"content":{"history_visibility":"shared"},"event_id":"$Bto6KcfZ9_wWHv3K141d35SkE-KKx9yNn6U7vFhchhg","origin_server_ts":1628012337808,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.history_visibility","unsigned":{}},{"content":{"guest_access":"forbidden"},"event_id":"$iK5LimMns1J3o3RSQ3QV8a8iuhcu_NTo13g15M7iJgY","origin_server_ts":1628012337824,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.guest_access","unsigned":{}}]}}}}}
> client/r0/rooms/!z3iyEeHFmrLTUAwQUF:localhost:8800/send/m.room.message/0 {"msgtype":"m.text","body":"Here is the message content"}
< 200 {"event_id":"$rceux4p0AGQrdSoRnN8Tt6Sm-PlV6er90k_cIld2e50"}
> client/r0/sync
> 200 {"next_batch":"870","rooms":{"join":{"!z3iyEeHFmrLTUAwQUF:localhost:8800":{"summary":{"m.joined_member_count":1,"m.invited_member_count":0},"unread_notifications":{"highlight_count":0,"notification_count":0},"timeline":{"limited":true,"prev_batch":"829","events":[{"content":{"creator":"@anon-20210803_173811-34:localhost:8800","room_version":"6"},"event_id":"$fmcyYiUPeJiC972jVrZqllGUMJ8DJ2ziwLxX93hEKoQ","origin_server_ts":1628012337746,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.create","unsigned":{}},{"content":{"is_direct":false,"membership":"join"},"event_id":"$JSkeEV_Cq0aexMV16JJDMmcO-fz2Qns3POa0GBGXAwM","origin_server_ts":1628012337762,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"@anon-20210803_173811-34:localhost:8800","type":"m.room.member","unsigned":{}},{"content":{"users":{"@anon-20210803_173811-34:localhost:8800":100}},"event_id":"$s60zf9B2hYfQOxs6RdOiQpQeETLkXc66Rni6wB3xbVs","origin_server_ts":1628012337778,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.power_levels","unsigned":{}},{"content":{"join_rule":"public"},"event_id":"$LcE6-fL4W-81Fu5gEdZTDcqQd9qIwCdNKnRO_SH2Irw","origin_server_ts":1628012337792,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.join_rules","unsigned":{}},{"content":{"history_visibility":"shared"},"event_id":"$Bto6KcfZ9_wWHv3K141d35SkE-KKx9yNn6U7vFhchhg","origin_server_ts":1628012337808,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.history_visibility","unsigned":{}},{"content":{"guest_access":"forbidden"},"event_id":"$iK5LimMns1J3o3RSQ3QV8a8iuhcu_NTo13g15M7iJgY","origin_server_ts":1628012337824,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.guest_access","unsigned":{}},{"content":{},"event_id":"$aVbO5UM8PBIdJZr8MN3vXxSWLcqsjhJZ8ivUjX4bzrw","origin_server_ts":1628012337925,"sender":"@anon-20210803_173811-34:localhost:8800","type":"m.room.test","unsigned":{"transaction_id":"1000007"}},{"content":{"body":"Here is the message content","msgtype":"m.text"},"event_id":"$rceux4p0AGQrdSoRnN8Tt6Sm-PlV6er90k_cIld2e50","origin_server_ts":1628012338024,"sender":"@anon-20210803_173811-34:localhost:8800","type":"m.room.message","unsigned":{"transaction_id":"0"}}]},"state":{"events":[{"content":{"history_visibility":"shared"},"event_id":"$Bto6KcfZ9_wWHv3K141d35SkE-KKx9yNn6U7vFhchhg","origin_server_ts":1628012337808,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.history_visibility","unsigned":{}},{"content":{"is_direct":false,"membership":"join"},"event_id":"$JSkeEV_Cq0aexMV16JJDMmcO-fz2Qns3POa0GBGXAwM","origin_server_ts":1628012337762,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"@anon-20210803_173811-34:localhost:8800","type":"m.room.member","unsigned":{}},{"content":{"join_rule":"public"},"event_id":"$LcE6-fL4W-81Fu5gEdZTDcqQd9qIwCdNKnRO_SH2Irw","origin_server_ts":1628012337792,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.join_rules","unsigned":{}},{"content":{"creator":"@anon-20210803_173811-34:localhost:8800","room_version":"6"},"event_id":"$fmcyYiUPeJiC972jVrZqllGUMJ8DJ2ziwLxX93hEKoQ","origin_server_ts":1628012337746,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.create","unsigned":{}},{"content":{"guest_access":"forbidden"},"event_id":"$iK5LimMns1J3o3RSQ3QV8a8iuhcu_NTo13g15M7iJgY","origin_server_ts":1628012337824,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.guest_access","unsigned":{}},{"content":{"users":{"@anon-20210803_173811-34:localhost:8800":100}},"event_id":"$s60zf9B2hYfQOxs6RdOiQpQeETLkXc66Rni6wB3xbVs","origin_server_ts":1628012337778,"sender":"@anon-20210803_173811-34:localhost:8800","state_key":"","type":"m.room.power_levels","unsigned":{}}]}}}},"account_data":{"events":[{"content":{"global":{"content":[{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight"}],"default":true,"enabled":true,"pattern":"anon-20210803_173811-34","rule_id":".m.rules.contains_user_name"}],"override":[{"actions":["dont_notify"],"conditions":[],"default":true,"enabled":false,"rule_id":".m.rule.master"},{"actions":["dont_notify"],"conditions":[{"key":"content.msgtype","kind":"event_match","pattern":"m.notice"}],"default":true,"enabled":true,"rule_id":".m.rule.suppress_notices"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.member"},{"key":"content.membership","kind":"event_match","pattern":"invite"},{"key":"state_key","kind":"event_match","pattern":"@anon-20210803_173811-34:localhost:8800"}],"default":true,"enabled":true,"rule_id":".m.rule.invite_for_me"},{"actions":["dont_notify"],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.member"}],"default":true,"enabled":true,"rule_id":".m.rule.member_event"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight"}],"conditions":[{"kind":"contains_display_name"}],"default":true,"enabled":true,"rule_id":".m.rule.contains_display_name"},{"actions":["notify",{"set_tweak":"highlight"}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.tombstone"},{"key":"state_key","kind":"event_match","pattern":""}],"default":true,"enabled":false,"rule_id":".m.rule.tombstone"},{"actions":["notify",{"set_tweak":"highlight"}],"conditions":[{"key":"content.body","kind":"event_match","pattern":"@room"},{"key":"room","kind":"sender_notification_permission"}],"default":true,"enabled":true,"rule_id":".m.rule.roomnotif"},{"actions":["dont_notify"],"conditions":[{"key":"type","kind":"event_match","pattern":"m.reaction"}],"default":true,"enabled":true,"rule_id":".m.rule.reaction"}],"room":[],"sender":[],"underride":[{"actions":["notify",{"set_tweak":"sound","value":"ring"},{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.call.invite"}],"default":true,"enabled":true,"rule_id":".m.rules.call"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"is":"2","kind":"room_member_count"},{"key":"type","kind":"event_match","pattern":"m.room.encrypted"}],"default":true,"enabled":true,"rule_id":".m.rules.encrypted_room_one_to_one"},{"actions":["notify",{"set_tweak":"sound","value":"default"},{"set_tweak":"highlight","value":false}],"conditions":[{"is":"2","kind":"room_member_count"},{"key":"type","kind":"event_match","pattern":"m.room.message"}],"default":true,"enabled":true,"rule_id":".m.rules.room_one_to_one"},{"actions":["notify",{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.message"}],"default":true,"enabled":true,"rule_id":".m.rules.message"},{"actions":["notify",{"set_tweak":"highlight","value":false}],"conditions":[{"key":"type","kind":"event_match","pattern":"m.room.encrypted"}],"default":true,"enabled":true,"rule_id":".m.rules.encrypted"}]}},"type":"m.push_rules"}]}}
> client/r0/rooms/!z3iyEeHFmrLTUAwQUF:localhost:8800/messages dir=b, from=829
< {"start":"829"}

The specification isn't really clear about what needs to be returned for a /sync without since, but that should actually not be a problem. Conduit returns all the events for the room in the test. I assume that Synapse is actually doing the same, or that it will at least include the test message. Under the assumption that the test message was returned in the response to the /sync, the prev_batch for that room should then point before the test message.

If Sytest then asks for the messages since prev_batch, backwards, and it was returned in /sync, it should not get the test message since it was already returned, but it does get it. So as far as I can see, the reasons the Sytest currently passes are:

• The prev_batch value doesn't point to the message before the returned events
• It's not returning messages starting at prev_batch, but actually messages after it.
• /sync did not return the test message

If I then changed it to return messages between next_batch and prev_batch, it should have returned the message if it was included in the /sync, but it didn't.

[babolivier]

I've had a quick test and it looks to me like it's working as intended.

I've done a sync which brought up event $Ar1eHli4J1It0ZtVm16ZRWO2mlDCu_CE4q6YO_Dr3F8 in room !BmAgWlfcrgaRfViMVl:matrix.org. This sync has a next_batch of s8309819_93441792_133776_10682404_669543_1254_1159557_9500936_33 and a prev_batch for that room of s8309817_93441792_133776_10682404_669543_1254_1159557_9500936_33.

If I request /_matrix/client/r0/rooms/!BmAgWlfcrgaRfViMVl:matrix.org/messages?dir=b&from=s8309817_93441792_133776_10682404_669543_1254_1159557_9500936_33 (using the prev_batch as from), it doesn't show event $Ar1eHli4J1It0ZtVm16ZRWO2mlDCu_CE4q6YO_Dr3F8, which is expected since it's only supposed to show events sent before it.

If I request /_matrix/client/r0/rooms/!BmAgWlfcrgaRfViMVl:matrix.org/messages?dir=b&from=s8309819_93441792_133776_10682404_669543_1254_1159557_9500936_33&to=s8309817_93441792_133776_10682404_669543_1254_1159557_9500936_33 (using the prev_batch as to and the next_batch as from) it does show event $Ar1eHli4J1It0ZtVm16ZRWO2mlDCu_CE4q6YO_Dr3F8, which is expected since that event happened between the positions these tokens refer to.

So it looks like Synapse returns a prev_batch that points to before the event and a next_batch that points to after the event, which is what the spec says.

It is weird that this looks quite obviously not like what's happening in matrix-org/sytest#1083, though my poking around leads me to believe this is a symptom of something else and not the source of the issue itself.

[richvdh]

nope, I think this is a real bug, and it's been around for ever: it's a duplicate of #1509.

[babolivier]

Oh, my bad then, somehow I didn't manage to replicate it locally :/
Good that it's already being tracked though.