302 Redirect after Login

[imukai]

I can successfully populate and submit a login form with Scrapy. As can be common with logins the server sends back a 302 redirect to a main page that Scrapy sees as an error (a WebException).

I tried enabling .AllowAutoRedirect as well as AllowMetaRedirect on the ScrapingBrowser object with no change in behavior.

Please advise.

[imukai]

My current workaround is to trap for the exception and manually call the next page, but this seems less than ideal as relying on exceptions for normal flow is usually bad.

catch (AggregateException ex) {
    var inner = ex.InnerExceptions.First() as WebException;
    redoUrl = inner?.Response?.Headers["Location"];
    }

[dararish]

@imukai as far as I can tell, HttpWebRequest throws an exception whenever a non-200 status code is returned. See #19 for a fix.