Thanks @kevin for your kind response..
it's really helpful. however, i have some confusions. The steps that you mentioned, have you tested while analyzing ancient genomes data?
After reading your guidelines i designed my command-line (based on the criteria that i mentioned in question previously) for ancient BAM samples as follows:
STEP1: I am interested in calling all sites variants/non-variant omitting --variants-only option in BCFTOOLS Since i am using single sample BAM omitting --per-sample-mF
samtools mpileup -C50 --redo-BAQ --min-BQ 30 --output-tags DP,AD -f ref.fa \
--BCF aln.bam | bcftools call --consensus-caller -Ob > Out.bcf
STEP2 and STEP3 as you suggested.
what does this -m-any used for in step 2 and what is --pval-threshold 1.0 in STEP 1 ?
In STEP4 i am still not sure how apply the following filters:
2. Filtered out variant calls with a phred posterior probability lower than 30.
3. Filtered out variants with significant strand and/or end distance bias (p<1e-4) or located within 5 bp of each other.
4. Discarded heterozygote calls if allelic balance for the minor allele was less than 0.25.
Why you set most of the options 0.05 in vcfutils.pl command? can you please guide me in applying above mentioned filters? or if you have tested your set parameters performed better in ancient DNA analysis, please share your experience.
Thanks a lot!