extremely slow execution of sighting.get_pipeline_status()

[naknomum]

source unknown, but very likely related to needing to block while querying sage. maybe multiple times? on zebra, can take 1-2 minutes to get a result. this is especially bad during full Sighting elasticsearch re-indexing, as it hits this method 35000 times.

possibly solutions:

• sighting.get_pipeline_state() (used by ES index) calls get_pipeline_status() - this is inefficient to get the whole complex status when the state can possibly short circuit early. some optimization possible here.
• get_pipeline_status() might have some optimizations available in the methods it calls; in particular, some sort of sage timeout improvement or (better still) avoidance
• probably the best solution is to store the state as an attribute on Sighting. (and try to only use get_pipeline_state() and avoid get_pipeline_status() e.g. in the api Sighting schema) this is a little more work, but computing on-the-fly is probably best avoided