Project Euler download.py crashes on large batch

[jayvdb]

When doing a large batch download, e.g. starting from 1, download.py crashes with URLError due to Connection timed out.

Problem 232
Downloading...
Traceback (most recent call last):
  File "download.py", line 98, in <module>
    main()
  File "download.py", line 92, in main
    downloaded = save_problem(problem_num)
  File "download.py", line 68, in save_problem
    soup = download(url)
  File "download.py", line 57, in download
    data = browser.open(url).get_data()
  File "/usr/lib/python2.7/site-packages/mechanize/_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/mechanize/_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "/usr/lib/python2.7/site-packages/mechanize/_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1142, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/site-packages/mechanize/_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>

[jontsai]

Yeah, probably not much you can do here. Do a smaller batch or increase the sleep timeout between each download.

If we make a fix here, PE can always change settings on their end. Maybe the best fix for this script is to add some comments about what a user can do to mitigate issues like this happening.