This is a test version of Biostars. For the public version, visit https://www.biostars.org.
GATK BaseRecalibrator error.. how do I solve this?

I downloaded reference genome and known site vcf at GATK bundle (https://console.cloud.google.com/storage/browser/genomics-public-data/resources/broad/hg38/v0;tab=objects?pli=1&prefix=&forceOnObjectsSortingFiltering=false)

I sorted and duduplicated bam file then tried to use baserecalibrator. But I got errors like below.

gatk BaseRecalibrator -I sorted_dedup_test.bam -R $HG38 --known-sites resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf -O recal_data.table

Using GATK jar /home/sunghyepark_lab/packages/gatk-4.1.8.1/gatk-package-4.1.8.1-local.jar

Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/sunghyepark_lab/packages/gatk-4.1.8.1/gatk-package-4.1.8.1-local.jar BaseRecalibrator -I sorted_dedup_test.bam -R /home/sunghyepark_lab/test/test_files/HG38_Broad/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta --known-sites resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf -O recal_data.table

08:01:24.131 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/sunghyepark_lab/packages/gatk-4.1.8.1/gatk-package-4.1.8.1-local.jar!/com/intel/gkl/native/libgkl_compression.so

Jan 11, 2021 8:01:24 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.

08:01:24.316 INFO  BaseRecalibrator - ------------------------------------------------------------
08:01:24.317 INFO  BaseRecalibrator - The Genome Analysis Toolkit (GATK) v4.1.8.1
08:01:24.317 INFO  BaseRecalibrator - For support and documentation go to https://software.broadinstitute.org/gatk/
08:01:24.317 INFO  BaseRecalibrator - Executing as sunghyepark_lab@6655e26fb574 on Linux v3.10.0-327.3.1.el7_lustre.x86_64 amd64
08:01:24.317 INFO  BaseRecalibrator - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_252-b09
08:01:24.317 INFO  BaseRecalibrator - Start Date/Time: January 11, 2021 8:01:24 AM UTC
08:01:24.318 INFO  BaseRecalibrator - ------------------------------------------------------------
08:01:24.318 INFO  BaseRecalibrator - ------------------------------------------------------------
08:01:24.318 INFO  BaseRecalibrator - HTSJDK Version: 2.23.0
08:01:24.318 INFO  BaseRecalibrator - Picard Version: 2.22.8
08:01:24.318 INFO  BaseRecalibrator - HTSJDK Defaults.COMPRESSION_LEVEL : 2
08:01:24.318 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
08:01:24.318 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
08:01:24.318 INFO  BaseRecalibrator - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
08:01:24.319 INFO  BaseRecalibrator - Deflater: IntelDeflater
08:01:24.319 INFO  BaseRecalibrator - Inflater: IntelInflater
08:01:24.319 INFO  BaseRecalibrator - GCS max retries/reopens: 20
08:01:24.319 INFO  BaseRecalibrator - Requester pays: disabled
08:01:24.319 INFO  BaseRecalibrator - Initializing engine
08:01:24.933 INFO  FeatureManager - Using codec VCFCodec to read file file:///home2/sunghyepark_lab/storage/Whole_Exome_Sequencing/NMDA_encephal_202012/resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf

08:01:24.944 INFO  BaseRecalibrator - Shutting down engine
[January 11, 2021 8:01:24 AM UTC] org.broadinstitute.hellbender.tools.walkers.bqsr.BaseRecalibrator done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=2224029696

**org.broadinstitute.hellbender.exceptions.GATKException: Error initializing feature reader for path resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf**
        at org.broadinstitute.hellbender.engine.FeatureDataSource.getTribbleFeatureReader(FeatureDataSource.java:383)
        at org.broadinstitute.hellbender.engine.FeatureDataSource.getFeatureReader(FeatureDataSource.java:335)
        at org.broadinstitute.hellbender.engine.FeatureDataSource.<init>(FeatureDataSource.java:282)
        at org.broadinstitute.hellbender.engine.FeatureManager.addToFeatureSources(FeatureManager.java:246)
        at org.broadinstitute.hellbender.engine.FeatureManager.initializeFeatureSources(FeatureManager.java:209)
        at org.broadinstitute.hellbender.engine.FeatureManager.<init>(FeatureManager.java:156)
        at org.broadinstitute.hellbender.engine.ReadWalker.initializeFeatures(ReadWalker.java:68)
        at org.broadinstitute.hellbender.engine.GATKTool.onStartup(GATKTool.java:709)
        at org.broadinstitute.hellbender.engine.ReadWalker.onStartup(ReadWalker.java:50)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:138)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
        at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
        at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
        at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
        at org.broadinstitute.hellbender.Main.main(Main.java:289)

**Caused by: htsjdk.tribble.TribbleException$MalformedFeatureFile: Unable to parse header with error: Your input file has a malformed header: We never saw the required CHROM header line (starting with one #) for the input VCF file, for input source: resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf**
        at htsjdk.tribble.TribbleIndexedFeatureReader.readHeader(TribbleIndexedFeatureReader.java:263)
        at htsjdk.tribble.TribbleIndexedFeatureReader.<init>(TribbleIndexedFeatureReader.java:102)
        at htsjdk.tribble.TribbleIndexedFeatureReader.<init>(TribbleIndexedFeatureReader.java:127)
        at htsjdk.tribble.AbstractFeatureReader.getFeatureReader(AbstractFeatureReader.java:121)
        at org.broadinstitute.hellbender.engine.FeatureDataSource.getTribbleFeatureReader(FeatureDataSource.java:380)
        ... 14 more

**Caused by: htsjdk.tribble.TribbleException$InvalidHeader: Your input file has a malformed header: We never saw the required CHROM header line (starting with one #) for the input VCF file**
        at htsjdk.variant.vcf.VCFCodec.readActualHeader(VCFCodec.java:115)
        at htsjdk.tribble.AsciiFeatureCodec.readHeader(AsciiFeatureCodec.java:79)
        at htsjdk.tribble.AsciiFeatureCodec.readHeader(AsciiFeatureCodec.java:37)
        at htsjdk.tribble.TribbleIndexedFeatureReader.readHeader(TribbleIndexedFeatureReader.java:261)
        ... 18 more
gatk ngs wes
• 3,978 views • link
updated by Hamidah • Hamidah • written by kwanghoon • kwanghoon

what is the output of

file resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf

and 

head resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Output of file resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf

resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf: gzip compressed data, extra field

And output of head resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf

▒BC&▒}is▒H▒▒▒▒▒▒o:▒%! ▒y!kqiZ▒]▒▒▒%B▒(▒▒Ŷ▒׿▒X2▒▒▒▒꒪▒�p▒&q▒▒]O^$▒▒▒v:+o▒▒▒x▒▒▒▒▒▒▒▒▒w▒▒▒▒Ϧ▒▒▒zZ▒▒▒0▒▒g▒▒▒X?.▒=|▒h:_lօ▒▒▒_▒6Z▒▒▒x(▒▒▒x=v_-▒▒▒▒ޖ▒b5▒▒ro▒▒▒F▒▒j^▒▒C▒wy~q▒\ wZ▒K▒▒▒▒▒▒l▒ ▒▒7▒ͤ<▒▒vY▒}▒▒MY▒ʿo܇▒▒▒▒▒]9▒▒▒▒▒▒t▒*▒▒U)vo▒▒6+▒▒.▒▒Y▒▒▒f+▒▒▒z:/';▒▒;7▒K▒ک▒[▒|▒▒}5▒▒S/▒▒▒▒/;▒▒h▒ѩ▒▒s4▒▒▒▒ˎ▒9▒{bt▒ˎ▒9▒{bt▒ˎ▒9▒{bt▒eG▒▒=1:▒{▒▒]▒▒▒{▒▒▒▒▒
                                                                                                                      ▒▒▒h▒▒t▒▒▒▒x▒▒Y▒{j|/▒<K{O▒C▒gi▒▒p▒▒,▒=5▒\▒▒▒▒▒▒¡˳▒▒▒▒^8xy▒▒▒
                                                                                                                                                                                  ▒/▒▒▒S▒▒
                                                                                                                                                                                          ▒/▒▒▒S▒{▒▒▒Y▒{j|/M▒▒p▒▒▒▒&էy͑Ju▒▒XW▒-4Z▒▒▒▒T▒▒▒▒]▒▒▒▒▒▒▒▒▒f▒x▒(▒▒▒▒x▒~▒▒▒N2Z▒˚▒ߔ▒▒]٬ʷ▒▒▒.▒q3▒M▒▒r▒▒sRގ7▒▒k▒▒▒▒▒b▒0▒▒lo▒Z/▒7▒9,▒▒▒▒▒▒▒▒h▒▒m▒3▒<▒{G▒▒▒o▒▒▒▒ퟞ6▒ݲ▒,▒▒▒m1▒9k)a▒▒▒ˎ▒]▒▒▒▒;▒k▒i▒▒\▒כձ#▒▒o{r▒Q}*<▒\|▒▒▒▒▒I▒▒rR\▒▒Y▒▒▒▒0▒▒g▒▒▒C9_Ð▒|▒▒▒▒▒~▒▒▒▒5▒Ϫ;7▒bV~,g{'▒▒o▒W▒▒.ݗ▒E{▒ґ▒▒O▒▒߭▒▒▒/▒▒▒SެW▒▒▒#▒▒w▒l▒]-▒▒4▒D▒▒&▒Ï▒▒▒▒▒▒▒▒݇)\$▒▒xxc▒▒5ȃ*~<,▒/▒.▒▒w▒▒▒N▒▒h▒▒/W▒W/▒▒G▒f▒W-

Is it because of compressed file or file is crashed?

• 0 views • link
updated by Ram Ram • written by kwanghoon • kwanghoon

you should use {.vcf.gz} not {.vcf} from their resource bundle..

• 0 views • link
written by Hamidah • Hamidah

1 answer

you downloaded resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf as a VCF file but it looks like a BCF file (???). Fix this with:

bcftools view -O z -o tmp.vcf.gz resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf
mv tmp.vcf.gz resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz
bcftools index -t resources_broad_hg38_v0_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Wow it worked....

THANK YOU SO SO SO MUCH.

I don't know why resource bundle looks like BCF....

Anyway Thank you again!!!

• 0 views • link
written by kwanghoon • kwanghoon

Log in to answer this question.

More posts like this
  • GATK4 haplotypecaller calls variants on regions without read coverage
    written by thomas.heigl.ibk • thomas.heigl.ibk
    Hey friends. I am running my best practice GATK4 workflow and it seems to work. However, puzzling to me is that variants appear in the …
  • GATK4 VariantAnnotator deletes variants
    written by ManuelDB ManuelDB
    I am using VariantAnnotator gatk:4.1.3.0 without success and not clear error message is shown. However, same code in GATK3 works fine. I show the code …
  • CNNScoreVariants - GATK 4.5.0.0 error
    written by dophuochuy94 • dophuochuy94
    ``` /home/thang/huy/tools/gatk/gatk CNNScoreVariants \ -V 24WES042_20240424-971-2104_WES-11_I.haplo.vcf \ -R /home/thang/huy/ref/GRCh38.fa \ -O 24WES042_20240424-971-2104_WES-11_I.annotated.vcf ``` I already ran `conda activate gatk` My GATK version 4.5.0.0, Java 17. …
  • How does one create a custom VCF file?
    written by manfi • manfi
    I'm running an analysis where I use Mutect2 to look for reads that code for a particular pathogenic variant. The variant is given in a …
  • Error in VQSR first step
    written by Yoosef • Yoosef
    Hello. After inserting following codes for VQSR first step: java -jar gatk-package-4.3.0.0-local.jar VariantRecalibrator -O /home/yousef/Desktop/haplotype_hg38_SNP_Recal.vcf --resource:hapmap,known=false,training=true,truth=true,prior=15.0 '/media/yousef/EEFC0BDBFC0B9CC9/Sequencing/Next_generation_sequencing/Index_files/VQSR_data/resources_broad_hg38_v0_hapmap_3.3.hg38.vcf' --resource:omni,known=false,training=true,truth=false,prior=12.0 '/media/yousef/EEFC0BDBFC0B9CC9/Sequencing/Next_generation_sequencing/Index_files/VQSR_data/resources_broad_hg38_v0_1000G_omni2.5.hg38.vcf' --resource:1000G,known=false,training=true,truth=false,prior=10.0 '/media/yousef/EEFC0BDBFC0B9CC9/Sequencing/Next_generation_sequencing/Index_files/VQSR_data/resources_broad_hg38_v0_1000G_phase1.snps.high_confidence.hg38.vcf' --resource:dbsnp,known=true,training=false,truth=false,prior=2.0 '/media/yousef/EEFC0BDBFC0B9CC9/Sequencing/Next_generation_sequencing/Index_files/VQSR_data/resources_broad_hg38_v0_Homo_sapiens_assembly38.dbsnp138.vcf' --tranches-file /home/yousef/Desktop/Tranches.txt …
  • Problem with sequence dictionary detection in BaseRecalibrator
    written by Arda • Arda
    In recalibrating base quality scores step, I am trying to create a calibration table using BaseRecalibrator tool of GATK. Firstly, I have generated fai file …
  • While using GATK BaseRecalibration :No report file is created and i cannot figure out why is that…
    written by Phoebe Magdy • Phoebe Magdy
    I'm trying to make base recalibration using GATK for bam files and here is the command I used: ``` !gatk --java-options "-Xmx2G" BaseRecalibrator \ -R …
  • GATK Mutect2 errors during basic variant calling
    written by michael.flower.14 • michael.flower.14
    I've just installed GATK and am trying to do some basic variant calling. However when I try and run this line gatk Mutect2 -R $REF …
  • Problem with vcf format to run gatk FastaAlternateReferenceMaker
    written by pablo pablo
    Hello, I'm trying to run `gatk FastaAlternateReferenceMaker` to get the FASTA files from my vcf file. I run : `gatk FastaAlternateReferenceMaker -R my_reference.fa -O my_output.fasta …
  • GATK4.1.1.0---HaplotypeCaller :HaplotypeCallerEngine - Disabling physical phasing, which is support…
    written by nw_hyz • nw_hyz
    Hi everyone, I meet a difficult problem... java -Xmx40g -jar /data1/user/software/gatk-4.1.1.0/gatk-package-4.1.1.0-local.jar HaplotypeCaller -R genomic.fna -I x11fq/x11.sort.mkdup.bam -O x11fq/x11.g.vcf 12:33:15.225 INFO NativeLibraryLoader - Loading libgkl_compression.so from …