IMAP response parsing issues

[rustmailer]

Hi there,

First of all, I'd like to say thank you for this library. I'm using async-imap as the core IMAP client for my mail archiving project, Bichon, and it has been a great foundation for me.

We’ve recently encountered two issues related to IMAP command parsing that we are trying to debug. I’ve done some investigation on my end, but I’m struggling to determine the exact root cause, so I’m hoping to get some advice.

version:

async-imap = { version = "0.11.1", default-features = false, features = [
    "runtime-tokio",
    "compress",
] }

The Problem

Error 50000: Io( Custom { kind: Other, error: "Error(Error { input:, code: TakeWhile1 }) during parsing of \"* CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS MOVE SNIPPET=FUZZY LITERAL+ NOTIFY SPECIAL-USE QUOTA\\r\\nA0002 OK Capability completed (0.001 + 0.000 secs).\\r\\n\"", }, )[2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39][40][41][42][43][44][45][46][47]

Error(Error { input: [42, 32, 49, 49, 55, 56, 52, 32, 70, 69, 84, 67, 72, 32, 40, 85, 73, 68, 32, 56, 55, 53, 54, 55, 32, 73, 78, 84, 69, 82, 78, 65, 76, 68, 65, 84, 69, 32, 34, 48, 54, 45, 77, 97, 121, 45, 50, 48, 50, 51, 32, 49, 56, 58, 52, 56, 58, 51, 48, 32, 43, 48, 50, 48, 48, 34, 32, 82, 70, 67, 56, 50, 50, 46, 83, 73, 90, 69, 32, 53, 55, 57, 57, 56, 52, 53, 32, 41, 13, 10], code: TakeWhile1 }) during parsing of \"* 11784 FETCH (UID 87567 INTERNALDATE \\\"06-May-2023 18:48:30 +0200\\\" RFC822.SIZE 5799845 )\\r\\n\"",\n },\n)

What I've observed

I suspect that this method is the source of both errors.

fn decode(&mut self) -> io::Result<Option<ResponseData>> {
        if self.buffer.used() < self.decode_needs {
            // We know that there is not enough data to decode anything
            // from previous attempts.
            return Ok(None);
        }

        let block = self.buffer.take_block();
        // Be aware, now self.buffer is invalid until block is returned or reset!

        let res = ResponseData::try_new_or_recover(block, |buf| {
            let buf = &buf[..self.buffer.used()];
            log::trace!("decode: input: {:?}", std::str::from_utf8(buf));
            match imap_proto::parser::parse_response(buf) {
                Ok((remaining, response)) => {
                    // TODO: figure out if we can use a minimum required size for a response.
                    self.decode_needs = 0;
                    self.buffer.reset_with_data(remaining);
                    Ok(response)
                }
                Err(nom::Err::Incomplete(Needed::Size(min))) => {
                    log::trace!("decode: incomplete data, need minimum {min} bytes");
                    self.decode_needs = self.buffer.used() + usize::from(min);
                    Err(None)
                }
                Err(nom::Err::Incomplete(_)) => {
                    log::trace!("decode: incomplete data, need unknown number of bytes");
                    self.decode_needs = 0;
                    Err(None)
                }
                Err(other) => {
                    self.decode_needs = 0;
                    Err(Some(io::Error::other(format!(
                        "{:?} during parsing of {:?}",
                        other,
                        String::from_utf8_lossy(buf)
                    ))))
                }
            }
        });
        match res {
            Ok(response) => Ok(Some(response)),
            Err((heads, err)) => {
                self.buffer.return_block(heads);
                match err {
                    Some(err) => Err(err),
                    None => Ok(None),
                }
            }
        }
    }

from: https://github.com/chatmail/async-imap/blob/v0.11.1/src/imap_stream.rs

The first error is quite puzzling: I’ve verified that the logged CAPABILITY data parses correctly in isolation. Could there be a discrepancy between the actual data passed to the parser and what is reported in the error log?

I have also isolated a second parsing issue. The parser fails when encountering the string RFC822.SIZE 5799845 ) (specifically the space before the closing parenthesis). If I remove that space (RFC822.SIZE 5799845), it parses correctly.

I suspect this might be an issue with imap-proto's grammar strictness, but I am not certain if this behavior is intended or if the parser should handle such variations in whitespace. Does imap-proto have any tolerance for extra whitespace in these response structures?

Any insights or pointers you could provide would be incredibly helpful.

Thank you again for your time and for this project!

rustmailer/bichon#190
rustmailer/bichon#197

[link2xt]

Is there anything to do here other than waiting for djc/tokio-imap#190 to be resolved?