This is a test version of Biostars. For the public version, visit https://www.biostars.org.
efetch sra suddenly stopping

Hi There,

I am using the command below to download raw fastq files from BioProject accession: PRJNA354235 for metagenomic and metatranscriptomic research.

esearch -db sra -query PRJNA354235 | efetch --format runinfo | cut -d ',' -f 1| grep SRR |xargs fastq-dump --split-file

The command works just fine at first then suddenly at SRR6451769 the process stops and prints ncbi_error_report.txt. Is there a way to get around this or for the command to ignore an error and continue trying to move forward? The error report is below.

Thank you for your time!

<Report>
 <Run>
  <Date>
   <Start value="Fri Jan 3 2020 6:30:55 PM"/>
   <End value="Fri Jan 3 2020 6:33:13 PM"/>
  </Date>
  <Home name="HOME" value="/home/ubuntu"/>
  <Cwd>/home/ubuntu/data2</Cwd>
  <CommandLine argc="2">
   <Arg index="0" value="/usr/local/ncbi/sra-tools/bin/fastq-dump"/>
   <Arg index="1" value="SRR6451769"/>
  </CommandLine>
  <Result rc="RC(rcPS,rcCondition,rcWaiting,rcTimeout,rcExhausted)"/>
  <User admin="false"/>
 </Run>
 <Configuration>
  <Files count="3">
   <File name="/etc/ncbi/certs.kfg"/>
   <File name="/etc/ncbi/default.kfg"/>
   <File name="/etc/ncbi/vdb-copy.kfg"/>
  </Files>
  <refseq state="not found"/>
  <krypto state="pwfile: not found"/>
  <sra state="not found"/>
  <Config>  
  <ConfigurationFiles>
/etc/ncbi/certs.kfg
/etc/ncbi/default.kfg
/etc/ncbi/vdb-copy.kfg
    </ConfigurationFiles>
    <APPNAME>"fastq-dump"</APPNAME>
    <APPPATH>"/usr/local/ncbi/sra-tools/bin"</APPPATH>
    <AWS></AWS>
    <BUILD>"RELEASE"</BUILD>
    <HOME>"/home/ubuntu"</HOME>
    <HOST></HOST>
    <NCBI_HOME>"/home/ubuntu/.ncbi"</NCBI_HOME>
    <NCBI_SETTINGS>"/home/ubuntu/.ncbi/user-settings.mkfg"</NCBI_SETTINGS>
    <OS>"linux"</OS>
    <PWD>"/home/ubuntu/data2"</PWD>
    <USER>"ubuntu"</USER>
    <VDBCOPY>
      <DO_NOT_REDACT>"CS_KEY,FLOW_CHARS,KEY_SEQUENCE,LINKER_SEQUENCE"</DO_NOT_REDACT>
      <META>
        <IGNORE>"col,.seq,STATS,BASE_COUNT,HUFFMAN_TREE_POS,HUFFMAN_TREE_POS_SIZE,HUFFMAN_TREE_PRB,HUFFMAN_TREE_PRB_SIZE,HUFFMAN_TREE_SIG,HUFFMAN_TREE_SIG_SIZE,MSC454_CLIP_QUALITY_LEFT,MSC454_CLIP_QUALITY_RIGHT,MSC454_FLOW_CHARS,MSC454_KEY_SEQUENCE,NREADS,NUMBER_POS_CHANNELS,NUMBER_PRB_CHANNELS_1,NUMBER_PRB_COLUMNS,NUMBER_SIG_CHANNELS,PLATFORM,READ_0,READ_1,SPOT_COUNT"</IGNORE>
      </META>
      <NCBI_SRA_ABI_tbl_v1_1>
        <schema>"sra/abi.vschema"</schema>
        <tab>"NCBI:SRA:ABI:tbl:v2"</tab>
      </NCBI_SRA_ABI_tbl_v1_1>
      <NCBI_SRA_Illumina_tbl_v0a_1>
        <schema>"sra/illumina.vschema"</schema>
        <tab>"NCBI:SRA:Illumina:tbl:phred:v2"</tab>
      </NCBI_SRA_Illumina_tbl_v0a_1>
      <NCBI_SRA__454__tbl_v0_1>
        <schema>"sra/454.vschema"</schema>
        <tab>"NCBI:SRA:_454_:tbl:v2"</tab>
      </NCBI_SRA__454__tbl_v0_1>
      <READ_FILTER_COL_NAME>"READ_FILTER"</READ_FILTER_COL_NAME>
      <REDACTABLE_TYPES>"INSDC:color:text,INSDC:x2cs:bin,INSDC:2cs:bin,INSDC:2cs:packed,INSDC:dna:text,INSDC:4na:bin,INSDC:4na:packed,INSDC:x2na:bin,INSDC:2na:bin,INSDC:2na:packed,NCBI:SRA:pos16,INSDC:quality:phred,INSDC:quality:log_odds,NCBI:qual4,INSDC:position:one,INSDC:position:zero,NCBI:fsamp4,NCBI:isamp1"</REDACTABLE_TYPES>
      <REDACTVALUE>
        <INSDC_color_text>
          <VALUE>"'.'"</VALUE>
        </INSDC_color_text>
        <INSDC_dna_text>
          <VALUE>"'N'"</VALUE>
        </INSDC_dna_text>
        <NCBI_qual4>
          <VALUE>"-6"</VALUE>
        </NCBI_qual4>
        <TYPES>"INSDC_color_text,INSDC_dna_text,NCBI_qual4"</TYPES>
      </REDACTVALUE>
      <SCORE>
        <INSDC_2cs_bin>"2"</INSDC_2cs_bin>
        <INSDC_2cs_packed>"2"</INSDC_2cs_packed>
        <INSDC_2na_bin>"2"</INSDC_2na_bin>
        <INSDC_2na_packed>"2"</INSDC_2na_packed>
        <INSDC_SRA_read_type>"1"</INSDC_SRA_read_type>
        <INSDC_quality_log_odds>"1"</INSDC_quality_log_odds>
        <INSDC_quality_phred>"1"</INSDC_quality_phred>
        <INSDC_x2cs_bin>"1"</INSDC_x2cs_bin>
        <INSDC_x2na_bin>"1"</INSDC_x2na_bin>
      </SCORE>
      <_454_>
        <schema>"sra/454.vschema"</schema>
        <tab>"NCBI:SRA:_454_:tbl:v2"</tab>
      
      <_ABSOLID_>
        <schema>"sra/abi.vschema"</schema>
        <tab>"NCBI:SRA:ABI:tbl:v2"</tab>
      
      <_ILLUMINA_>
        <schema>"sra/illumina.vschema"</schema>
        <tab>"NCBI:SRA:Illumina:tbl:phred:v2"</tab>
      
    </VDBCOPY>
    <VDB_CONFIG></VDB_CONFIG>
    <VDB_ROOT></VDB_ROOT>
    <config>
      <default>"true"</default>
    </config>
    <kfg>
      <arch>
        <bits>"64"</bits>
        <name>"ip-172-31-89-243"</name>
      </arch>
      "/etc/ncbi"
      <name>"default.kfg"</name>
    </kfg>
    <repository>
      <remote>
        <main>
          <CGI>
            <resolver-cgi>"https://trace.ncbi.nlm.nih.gov/Traces/names/names.fcgi"</resolver-cgi>
          </CGI>
          <SDL.2>
            <resolver-cgi>"https://trace.ncbi.nlm.nih.gov/Traces/sdl/2/retrieve"</resolver-cgi>
          </SDL.2>
        </main>
        <protected>
          <CGI>
            <resolver-cgi>"https://trace.ncbi.nlm.nih.gov/Traces/names/names.fcgi"</resolver-cgi>
          </CGI>
        </protected>
      </remote>
      <user>
        <ad>
          <public>
            <apps>
              <file>
                <volumes>
                  <flat>"."</flat>
                </volumes>
              </file>
              <refseq>
                <volumes>
                  <refseqAd>"."</refseqAd>
                </volumes>
              </refseq>
              <sra>
                <volumes>
                  <sraAd>"."</sraAd>
                </volumes>
              </sra>
              <sraPileup>
                <volumes>
                  <ad>"."</ad>
                </volumes>
              </sraPileup>
              <sraRealign>
                <volumes>
                  <ad>"."</ad>
                </volumes>
              </sraRealign>
            </apps>
            <root>"."</root>
          </public>
        </ad>
        <main>
          <public>
            <apps>
              <file>
                <volumes>
                  <flat>"files"</flat>
                </volumes>
              </file>
              <nakmer>
                <volumes>
                  <nakmerFlat>"nannot"</nakmerFlat>
                </volumes>
              </nakmer>
              <nannot>
                <volumes>
                  <nannotFlat>"nannot"</nannotFlat>
                </volumes>
              </nannot>
              <refseq>
                <volumes>
                  <refseq>"refseq"</refseq>
                </volumes>
              </refseq>
              <sra>
                <volumes>
                  <sraFlat>"sra"</sraFlat>
                </volumes>
              </sra>
              <sraPileup>
                <volumes>
                  <flat>"sra"</flat>
                </volumes>
              </sraPileup>
              <sraRealign>
                <volumes>
                  <flat>"sra"</flat>
                </volumes>
              </sraRealign>
              <wgs>
                <volumes>
                  <wgsFlat>"wgs"</wgsFlat>
                </volumes>
              </wgs>
            </apps>
          </public>
        </main>
      </user>
    </repository>
    <tls>
      <ca.crt>
        <ncbi1>"-----BEGIN CERTIFICATE-----\x0D\x0AMIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh\x0D\x0AMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\x0D\x0Ad3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD\x0D\x0AQTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT\x0D\x0AMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j\x0D\x0Ab20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG\x0D\x0A9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB\x0D\x0ACSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97\x0D\x0Anh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt\x0D\x0A43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P\x0D\x0AT19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4\x0D\x0AgdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO\x0D\x0ABgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR\x0D\x0ATLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw\x0D\x0ADQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr\x0D\x0AhMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg\x0D\x0A06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF\x0D\x0APnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls\x0D\x0AYSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk\x0D\x0ACAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4wOTAeBggrBgEFBQcD\x0D\x0ABAYIKwYBBQUHAwEGCCsGAQUFBwMDDBdEaWdpQ2VydCBHbG9iYWwgUm9vdCBDQQ==\x0D\x0A-----END CERTIFICATE-----\x0D\x0A"</ncbi1>
        <ncbi2>"-----BEGIN CERTIFICATE-----\x0D\x0AMIIDxTCCAq2gAwIBAgIQAqxcJmoLQJuPC3nyrkYldzANBgkqhkiG9w0BAQUFADBs\x0D\x0AMQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\x0D\x0Ad3cuZGlnaWNlcnQuY29tMSswKQYDVQQDEyJEaWdpQ2VydCBIaWdoIEFzc3VyYW5j\x0D\x0AZSBFViBSb290IENBMB4XDTA2MTExMDAwMDAwMFoXDTMxMTExMDAwMDAwMFowbDEL\x0D\x0AMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3\x0D\x0ALmRpZ2ljZXJ0LmNvbTErMCkGA1UEAxMiRGlnaUNlcnQgSGlnaCBBc3N1cmFuY2Ug\x0D\x0ARVYgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMbM5XPm\x0D\x0A+9S75S0tMqbf5YE/yc0lSbZxKsPVlDRnogocsF9ppkCxxLeyj9CYpKlBWTrT3JTW\x0D\x0APNt0OKRKzE0lgvdKpVMSOO7zSW1xkX5jtqumX8OkhPhPYlG++MXs2ziS4wblCJEM\x0D\x0AxChBVfvLWokVfnHoNb9Ncgk9vjo4UFt3MRuNs8ckRZqnrG0AFFoEt7oT61EKmEFB\x0D\x0AIk5lYYeBQVCmeVyJ3hlKV9Uu5l0cUyx+mM0aBhakaHPQNAQTXKFx01p8VdteZOE3\x0D\x0AhzBWBOURtCmAEvF5OYiiAhF8J2a3iLd48soKqDirCmTCv2ZdlYTBoSUeh10aUAsg\x0D\x0AEsxBu24LUTi4S8sCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQF\x0D\x0AMAMBAf8wHQYDVR0OBBYEFLE+w2kD+L9HAdSYJhoIAu9jZCvDMB8GA1UdIwQYMBaA\x0D\x0AFLE+w2kD+L9HAdSYJhoIAu9jZCvDMA0GCSqGSIb3DQEBBQUAA4IBAQAcGgaX3Nec\x0D\x0AnzyIZgYIVyHbIUf4KmeqvxgydkAQV8GK83rZEWWONfqe/EW1ntlMMUu4kehDLI6z\x0D\x0AeM7b41N5cdblIZQB2lWHmiRk9opmzN6cN82oNLFpmyPInngiK3BD41VHMWEZ71jF\x0D\x0AhS9OMPagMRYjyOfiZRYzy78aG6A9+MpeizGLYAiJLQwGXFK3xPkKmNEVX58Svnw2\x0D\x0AYzi9RKR/5CYrCsSXaQ3pjOLAEFe4yHYSkVXySGnYvCoCWw9E1CAx2/S6cCZdkGCe\x0D\x0AvEsXCS+0yx5DaMkHJ8HSXPfqIbloEpw8nL+e/IBcm2PN7EeqJSdnoDfzAIJ9VNep\x0D\x0A+OkuE6N36B9KMEQwHgYIKwYBBQUHAwQGCCsGAQUFBwMBBggrBgEFBQcDAwwiRGln\x0D\x0AaUNlcnQgSGlnaCBBc3N1cmFuY2UgRVYgUm9vdCBDQQ==\x0D\x0A-----END CERTIFICATE-----\x0D\x0A"</ncbi2>
      </ca.crt>
    </tls>
    <tools>
      <ascp>
        <max_rate>"450m"</max_rate>
      </ascp>
    </tools>
    <vdb>
      <lib>
        <paths>
          <kfg>"/usr/local/ncbi/sra-tools/bin"</kfg>
        </paths>
      </lib>
    </vdb>
  </Config>
  <RemoteAccess available="true"/>
  <CurrentProtectedRepository found="false"/>
 </Configuration>
 <Object path="SRR6451769" type="table" fs_type="unexpected">
 </Object>
 <SOFTWARE>
  <VDBLibrary vers="2.7.32"/>
  <Build static="true">
   <Module name=""/>
  </Build>
  <Tool date="Aug 13 2019" name="/usr/local/ncbi/sra-tools/bin/fastq-dump" vers="2.10">
   <Binary path="/usr/local/ncbi/sra-tools/bin/fastq-dump" type="alias" md5="ae4edbb7ad442f57834b3b7f7596f30d">
    <Alias resolved="fastq-dump.2">
     <Alias resolved="fastq-dump.2.10.0">
      <Alias resolved="sratools.2.10.0"/>
     </Alias>
    </Alias>
   </Binary>
  </Tool>
 </SOFTWARE>
 <Env>
 </Env>
</Report>
fastq sra ncbi efetch esearch
• 1,273 views • link
updated by ATpoint ATpoint • written by barry • barry

This does not look like an error report even though it may be saved in a file with error in its name. There are 2598 samples in this set. How many are you able to successfully recover before you run into a problem.

• 0 views • link
written by GenoMax GenoMax

1 answer

I recommend not downloading fastq files via fastq-dump directly. This tool is too unstable for this. Using xargs doesn't improve things either as you have multiple instances of the tool running, potentially corrupting things. Better use GNU parallel or a loop.

Recommended: Either use prefetch to get the SRA file and then convert to fq with fastq-dump locally or download directly in fastq format from the ENA. Both approaches are outlined in this tutorial:

Fast download of FASTQ files from the European Nucleotide Archive (ENA)

• 0 views • link
written by ATpoint ATpoint

Log in to answer this question.

More posts like this