Right now, when a random API error occurs (e.g. in my case I got 403 in one out of tens of thousands requests to the same model, for whatever reason), Runner returns and empty string.
This is usually OK for text questions (we get one empty string instead of the answer, doesn't matter much, this is very rare). But in logprobs case, we should have a dictionary and we get an error of "str has no attribute items" or something like that.
Furthermore, the bad result gets cached so we need to clean cache, e.g. like this:
try:
df = question.df(MODELS)
except Exception as e:
for _, models in MODELS.items():
for model in models:
print(f"Clearing cache for {model}")
question.clear_cache(model)
I'm not sure what is the best approach here. Maybe runner should return empty dict if the question asks for a dict? But this feels potentially more misleading than empty strings.
Maybe runner should raise some exception (UnexpectedError or whatever), and Question should capture it and handle? This is likely much better than what we have now.
Right now, when a random API error occurs (e.g. in my case I got 403 in one out of tens of thousands requests to the same model, for whatever reason), Runner returns and empty string.
This is usually OK for text questions (we get one empty string instead of the answer, doesn't matter much, this is very rare). But in logprobs case, we should have a dictionary and we get an error of "str has no attribute items" or something like that.
Furthermore, the bad result gets cached so we need to clean cache, e.g. like this:
I'm not sure what is the best approach here. Maybe runner should return empty dict if the question asks for a dict? But this feels potentially more misleading than empty strings.
Maybe runner should raise some exception (UnexpectedError or whatever), and Question should capture it and handle? This is likely much better than what we have now.