Skip to content

Working Agenda

Jump to bottom
Jiaying edited this page Jul 18, 2020 · 9 revisions

Review work in 2017

Data collection

Portals List

  • Build search terms
    Use python to get the portals in full-list.xlsx, and then build search terms according to different rules and length limits of the three Databases.
  • Conduct search(6.27)

Data cleaning

  • Filter Rules:
  1. Language: English;
  2. Type: Journal article or Conference paper
  3. Time: 2018, 2019, 2020

Statistics

Paper download

  • Method One：APIs(Failed)
  1. Scopus isn’t a full-text database, so I can only get abstract and other basic info by its API
  2. Another full-text database ScienceDirect which is also provided by Elsevier doesn’t have all the papers we need. And my application of full-text APIKey hasn’t been approved by Elsevier support center.
  • Method Two: Browser Extension(Succeed)
  1. There is a Document Download Manager extension which can download full-text of search results by one click, however there’s a limitation of 50 one time, so it still requires lots of time. You can try the Firefox extension, since I contact the support center and they told me the chrome extension doesn’t work.
  2. I asked the support center are there any methods to change this limitation and I am waiting for their reply.
  3. I also tried to find the source code of this extension but failed, and the CRX extension, which is used to see the source code of Firefox extensions also failed.

Paper Extraction

  • Get readable text
  1. Python: Depending on the layout of the PDF, if it is neat, it will be effective, but if it is stange, then it may not work very well.
  2. R: Transform PDFs with OCR first and then analyze
  3. Adobe: Transform the PDFs first and then extract
  4. APIs: Lose some papers from Scopus, or we can find a integrated API
  • Extract Objectives Use python to extract emails and portals appear in pdfs.
  1. Email-list: 7229 emails are extracted and there are 5907 emails left after removing deplicate items. Note: Part of emails may have some redudant characters in the end, eg: xxx@edu.comNames, I have cleaned some wrong emails by rules, but these are irregular.
  2. Portals: There are 3503 files after remove duplicated items among these three databases. I conclude the file names and corresponding urls retrieved from it, which we can use "Ctrl+F" to locate the urls. However, some articles do not have any matched url, since the layout of pdfs are complicated so that disorganize these urls. For these articles, we have to check manually. After removing those items with zero url, 3112 articles are left.

Clone this wiki locally