This is a test version of Biostars. For the public version, visit https://www.biostars.org.
sra-stat --statistics very slow on (ENA) SRA reads ERR5102882

Dear biostars out there :-),

I want to run sra-stat --statistics on a quite huge set of reads. After 50% it starts slowing down extremely and seems to never end.

A preliminary run of

sra-stat --xml --quick ERR5102882 > ERR5102882_sra-stat_quick.xml

revealed:

<Run accession="ERR5102882" spot_count="60717439" base_count="8906049287" base_count_bio="8906049287" cmp_base_count="8906049287">

So 60717439 spots = reads (= table lines ?) is quite a number. I tried to use --threads but this is not an option for this tool.

sra-stat --help suggests an --start and --stop option

So I plan to split up in smaller "packages" and afterwards merging the statistics together?

sra-stat --start 1 --stop 10717439 --xml --statistics --show_progress --verbose  ERR5102882 > ERR5102882_sra-stat_statistics_PART_01.xml

Does anybody know a better way?

Thanks in advance, Cheers, Katharina

sra-stat sratoolkit stastitics sra
• 2,072 views • link
updated by Ram Ram • written by KatharinaKopp • KatharinaKopp

I am not sure what exactly you are looking for from sra-stat but you can get the following information by using EntrezDirect:

$ efetch -db sra -id ERR5102882 -format runinfo

Run,ReleaseDate,LoadDate,spots,bases,spots_with_mates,avgLength,size_MB,AssemblyName,download_path,Experiment,LibraryName,LibraryStrategy,LibrarySelection,LibrarySource,LibraryLayout,InsertSize,InsertDev,Platform,Model,SRAStudy,BioProject,Study_Pubmed_id,ProjectID,Sam
ERR5102882,2021-02-02 00:45:59,2021-02-11 18:12:39,60717439,8906049287,60717439,146,3696,,https://sra-download.ncbi.nlm.nih.gov/traces/era2/ERR/ERR5102/ERR5102882,ERX4908318,unspecified,RNA-Seq,RANDOM,METAGENOMIC,PAIRED,300,0,ILLUMINA,NextSeq 500,ERP126368,PRJEB42502,
• 0 views • link
written by GenoMax GenoMax

Hi GenoMax,

Thanks for your answer. efetch can provide the same information as sra-stat --statistics regarding number of bases (A, T C, G) and Ns, avg read length & stdev and is much faster. I found these options to give a similar output to that of sra-stat --statistics :

time efetch -db sra -id ERR5102882 -format native -mode xml > QC/efetch/ERR5102882_efetch_native.xml

real 0m2,892s user 0m0,158s sys 0m0,070s

• 0 views • link
written by KatharinaKopp • KatharinaKopp

That command takes like 2 seconds on my machine. It is just collecting preparsed stats, it is not accessing the file itself.

$ sra-stat --xml --quick ERR5102882
<Run accession="ERR5102882" spot_count="60717439" base_count="8906049287" base_count_bio="8906049287" cmp_base_count="8906049287">
  <Member member_name="TAGGCATG" spot_count="58780928" base_count="8620137112" base_count_bio="8620137112" cmp_base_count="8620137112"/>
  <Member member_name="AAGGCATG" spot_count="544991" base_count="80185739" base_count_bio="80185739" cmp_base_count="80185739"/>
  <Member member_name="TAGGCATC" spot_count="30109" base_count="4432664" base_count_bio="4432664" cmp_base_count="4432664"/>
  <Member member_name="TAGACATG" spot_count="10415" base_count="1524966" base_count_bio="1524966" cmp_base_count="1524966"/>
  <Member member_name="TAGGCAGG" spot_count="163329" base_count="24224975" base_count_bio="24224975" cmp_base_count="24224975"/>
  <Member member_name="TAGGCAAG" spot_count="18056" base_count="2654481" base_count_bio="2654481" cmp_base_count="2654481"/>
  <Member member_name="TAGGGATG" spot_count="70360" base_count="10464980" base_count_bio="10464980" cmp_base_count="10464980"/>
  <Member member_name="TTGGCATG" spot_count="29834" base_count="4430032" base_count_bio="4430032" cmp_base_count="4430032"/>
  <Member member_name="TAGGCTTG" spot_count="500686" base_count="74230103" base_count_bio="74230103" cmp_base_count="74230103"/>
  <Member member_name="TAGGTATG" spot_count="16888" base_count="2480130" base_count_bio="2480130" cmp_base_count="2480130"/>
  <Member member_name="TAAGCATG" spot_count="153516" base_count="22551545" base_count_bio="22551545" cmp_base_count="22551545"/>
  <Member member_name="TATGCATG" spot_count="27328" base_count="4017006" base_count_bio="4017006" cmp_base_count="4017006"/>
  <Member member_name="TACGCATG" spot_count="93915" base_count="13807604" base_count_bio="13807604" cmp_base_count="13807604"/>
  <Member member_name="TAGGCATT" spot_count="33675" base_count="4952624" base_count_bio="4952624" cmp_base_count="4952624"/>
  <Member member_name="TAGGCCTG" spot_count="39764" base_count="5909248" base_count_bio="5909248" cmp_base_count="5909248"/>
  <Member member_name="TAGGCATA" spot_count="55616" base_count="8174831" base_count_bio="8174831" cmp_base_count="8174831"/>
  <Member member_name="TAGGCACG" spot_count="7083" base_count="1040139" base_count_bio="1040139" cmp_base_count="1040139"/>
  <Member member_name="TCGGCATG" spot_count="15392" base_count="2284810" base_count_bio="2284810" cmp_base_count="2284810"/>
  <Member member_name="TAGGCGTG" spot_count="26657" base_count="3943820" base_count_bio="3943820" cmp_base_count="3943820"/>
  <Member member_name="GAGGCATG" spot_count="29169" base_count="4331497" base_count_bio="4331497" cmp_base_count="4331497"/>
  <Member member_name="TAGTCATG" spot_count="13453" base_count="1987368" base_count_bio="1987368" cmp_base_count="1987368"/>
  <Member member_name="TAGGAATG" spot_count="12124" base_count="1778796" base_count_bio="1778796" cmp_base_count="1778796"/>
  <Member member_name="TGGGCATG" spot_count="19892" base_count="2938049" base_count_bio="2938049" cmp_base_count="2938049"/>
  <Member member_name="TAGCCATG" spot_count="17131" base_count="2520620" base_count_bio="2520620" cmp_base_count="2520620"/>
  <Member member_name="CAGGCATG" spot_count="7128" base_count="1046148" base_count_bio="1046148" cmp_base_count="1046148"/>
  <Size value="3875828769" units="bytes"/>
  <AlignInfo>
  </AlignInfo>
  <QualityCount>
    <Quality value="2" count="6068739"/>
    <Quality value="14" count="964457105"/>
    <Quality value="21" count="120769099"/>
    <Quality value="27" count="165764188"/>
    <Quality value="32" count="1267403419"/>
    <Quality value="36" count="6381586737"/>
  </QualityCount>
  <Databases>
    <Database>
      <Table name="SEQUENCE">
        <Statistics source="meta">
          <Rows count="60717439"/>
          <Elements count="8906049287"/>
        </Statistics>
      </Table>
    </Database>
  </Databases>
</Run>

Not sure what you can get from these information though.

• 0 views • link
written by ATpoint ATpoint

Hi ATpoint,

Thanks for your answer. The --quick option runs very fast, indeed. However, the --statistics option is actually very slow. They create slightly different outputs with not much added value regarding information when using --statistics (number of Ns, avg read length, stdev of read length). Maybe it could be useful to indicate and localize "bad" spots on the flow cell?

time sra-stat --xml **--quick** ERR5102882 > ERR5102882_sra-stat_quick.xml
real    0m6,113s
user    0m0,162s
sys 0m0,026s

time sra-stat --xml --statistics ERR5102882 > ERR5102882_sra-stat_statistics.xml                                                  
real    72m39,427s
user    5m11,348s
sys 1m20,296s

sra-stat --help
Usage:
  sra-stat [options] table
Summary:
  Display table statistics
Options:
...
-q|--quick                       quick mode: get statistics from metadata; do not scan the table 

-s|--statistics                  calculate READ_LEN average and standard deviation

output of --statistics:

<Run accession="ERR5102882" read_length="variable" spot_count="60717439" base_count="8906049287" base_count_bio="8906049287" spot_count_mates="60717439" base_count_bio_mates="8906049287" spot_count_bad="0" base_count_bio_bad="0" spot_count_filtered="0" base_count_bio_filtered="0">
  <Member member_name="AAGGCATG" spot_count="544991" base_count="80185739" base_count_bio="80185739" spot_count_mates="544991" base_count_bio_mates="80185739" spot_count_bad="0" base_count_bio_bad="0" spot_count_filtered="0" base_count_bio_filtered="0"/>
  <Member member_name="CAGGCATG" spot_count="7128" base_count="1046148" base_count_bio="1046148" spot_count_mates="7128" base_count_bio_mates="1046148" spot_count_bad="0" base_count_bio_bad="0" spot_count_filtered="0" base_count_bio_filtered="0"/>
  <Member member_name="GAGGCATG" spot_count="29169" base_count="4331497" base_count_bio="4331497" spot_count_mates="29169" base_count_bio_mates="4331497" spot_count_bad="0" base_count_bio_bad="0" spot_count_filtered="0" base_count_bio_filtered="0"/>
...
 <Size value="3875828769" units="bytes"/>
  <Bases cs_native="false" count="8906049287">
    <Base value="A" count="2132596134"/>
    <Base value="C" count="2119295117"/>
    <Base value="G" count="2646316212"/>
    <Base value="T" count="2001773085"/>
    <Base value="N" count="6068739"/>
  </Bases>
  <AlignInfo>
  </AlignInfo>
  <Statistics nreads="2" nspots="60717439">
    <Read index="0" count="60717439" average="73.33" stdev="5.50"/>
    <Read index="1" count="60717439" average="73.35" stdev="5.42"/>
  </Statistics>
  <QualityCount>
    <Quality value="2" count="6068739"/>
    <Quality value="14" count="964457105"/>
    <Quality value="21" count="120769099"/>
    <Quality value="27" count="165764188"/>
    <Quality value="32" count="1267403419"/>
    <Quality value="36" count="6381586737"/>
  </QualityCount>
  <Databases>
    <Database>
      <Table name="SEQUENCE">
        <Statistics source="meta">
          <Rows count="60717439"/>
          <Elements count="8906049287"/>
        </Statistics>
      </Table>
    </Database>
  </Databases>
</Run>
• 0 views • link
updated by GenoMax GenoMax • written by KatharinaKopp • KatharinaKopp

Wouldn’t ‚bad spots‘ anyway be removed by adapter and quality trim?

• 0 views • link
written by ATpoint ATpoint

0 answers

No answers yet.

Log in to answer this question.

More posts like this