FTP Metadata Crawler script

[davidonlaptop]

Write a script that crawls an FTP to collect recursively the metadata of the files.

Script input parameters:

• FTP_Site (e.g. ftp://ftp.ncbi.nlm.nih.gov/)

Script output (TSV Format):

• rows:
  • one row per file
• columns:
  • directory (/snp/organisms/human_9606/VCF)
  • filename (00-All.vcf.gz)
  • linktarget (All_20150603.vcf.gz) e.g. for when the entry is a symbolic link.
  • filesize (19) in bytes
  • DateModified (2015-06-08T15:16:20Z) in ISO 8601 format in the UTC timezone.
  • other metadata available ?

Particular behavior

• The script should throttle its speed so not to flood the FTP server with too many requests. A good rule of thumb would be not more than 4 requests per second. This should be a variable in the script.

Example

Example of how this script could be called:

./files-crawler.sh ftp://ftp.ncbi.nlm.nih.gov/   >  files.csv