Download from the SRA using ffq (and ffs)

ffq is a tool that can be used to fetch metadata from several data repositories. It can also be used to generate links to raw data hosted at the EBI (via FTP), AWS, GCP, and NCBI. I wrote a short Bash script called ffs that can generate the commands needed to download the raw data using the output from ffq. To use ffs just clone my repository somewhere and add the script directory to your PATH. For example:

cd
git clone https://github.com/davetang/research_parasite.git

export PATH=$PATH:$HOME/research_parasite/script

Type ffs to get the usage.

ffs

Program: ffs (Generate download script from ffq)
Version: 0.1.0

Usage:   /home/dtang/github/research_parasite/script/ffs <command> [options]

Commands:

     ftp            download using wget
     aspera         download using ascp
     aws            download using aws
     help           display this help message
     version        display version

To generate a script to download from the EBI's FTP site, use the following commands.

ffq --ftp SRX079566 | ffs ftp - > ftp.sh
cat ftp.sh

if [[ ! -f SRR292241_1.fastq.gz ]]; then
   wget -c ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/SRR292241/SRR292241_1.fastq.gz
fi
if [[ ! -f SRR292241_2.fastq.gz ]]; then
   wget -c ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR292/SRR292241/SRR292241_2.fastq.gz
fi
if [[ ! -f SRR390728_1.fastq.gz ]]; then
   wget -c ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR390/SRR390728/SRR390728_1.fastq.gz
fi
if [[ ! -f SRR390728_2.fastq.gz ]]; then
   wget -c ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR390/SRR390728/SRR390728_2.fastq.gz
fi

To generate a script to download from AWS using AWS CLI, use the following commands.

ffq --aws SRX079566 | ffs aws - > aws.sh
cat aws.sh

aws s3 sync s3://sra-pub-run-odp/sra/SRR292241 SRR292241 --no-sign-request
aws s3 sync s3://sra-pub-run-odp/sra/SRR390728 SRR390728 --no-sign-request

To generate a script to download using Aspera Connect, which is what I recommend since it is the fastest, use the following commands. (The generated commands are intended for use with my Docker image.)

ffq --ftp SRX079566 | ffs aspera - > aspera.sh
cat aspera.sh

if [[ ! -f SRR292241_1.fastq.gz ]]; then
   ascp -QT -l 300m -P33001 -i ${HOME}/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR292/SRR292241/SRR292241_1.fastq.gz .
fi
if [[ ! -f SRR292241_2.fastq.gz ]]; then
   ascp -QT -l 300m -P33001 -i ${HOME}/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR292/SRR292241/SRR292241_2.fastq.gz .
fi
if [[ ! -f SRR390728_1.fastq.gz ]]; then
   ascp -QT -l 300m -P33001 -i ${HOME}/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR390/SRR390728/SRR390728_1.fastq.gz .
fi
if [[ ! -f SRR390728_2.fastq.gz ]]; then
   ascp -QT -l 300m -P33001 -i ${HOME}/asperaweb_id_dsa.openssh era-fasp@fasp.sra.ebi.ac.uk:vol1/fastq/SRR390/SRR390728/SRR390728_2.fastq.gz .
fi

Use my Aspera Connect Docker image to download the data at blazing speeds (compared to FTP and AWS).

mkdir data && chmod 777 data && cd data
docker run --rm -it -u parasite -v $(pwd):$(pwd) -w $(pwd) davetang/aspera_connect:4.2.5.306 /bin/bash
bash ../aspera.sh

SRR292241_1.fastq.gz 100%  369MB  238Mb/s    00:13
Completed: 378151K bytes transferred in 13 seconds
 (227709K bits/sec), in 1 file.
SRR292241_2.fastq.gz 100%  376MB  272Mb/s    00:13
Completed: 385855K bytes transferred in 14 seconds
 (218815K bits/sec), in 1 file.
SRR390728_1.fastq.gz 100%   96MB  265Mb/s    00:05
Completed: 99006K bytes transferred in 5 seconds
 (141136K bits/sec), in 1 file.
SRR390728_2.fastq.gz 100%   97MB  261Mb/s    00:07
Completed: 99628K bytes transferred in 7 seconds
 (105342K bits/sec), in 1 file.

The above speeds are from my home Internet connection, on a Amazon Renewed desktop (from probably 2017 given it's using a 7th generation Intel Core i5) that I turned into a home server.

I will update ffs to accept different options so that users can generate ascp commands for a local installation or commands that use curl instead of wget, etc.

Check out the GitHub repository for more information.