Batch job: write script to pull all current SED transcripts

[andrewmarklloyd]

• Convert PDF to text format that can be scraped
• Input into BigTable as unclassified

[andrewmarklloyd]

Currently have a proof of concept script to get all pdf URLs and download the PDFs. I have another function to convert the pdf to text. I need to figure out how to associate the transcript URL with the episode id in softwaredaily.com MongoDB or something equivalent. The algorithm will be something like the following:

• get all transcript objects from Wordpress API
• for each transcript object:
  • use URL for pdf to download pdf
  • convert transcript pdf to text
  • upload text to Big Table as unclassified