Hi everyone,

A bit of an open ended question here, as I am sure there are many possibilities. I would appreciate opinions on the best way to filter my VCF files. I know that the appropriate filters depend on the questions being asked and the type of data available, so here is some information. I have sequence data from multiple individuals from 2 populations, (around 15 individuals per populations) sequenced at 2x coverage per individual genome wide. I want to identify population level differences that are fixed or near fixation, so not too worried about losing singletons. The genotypes were called using GATK while the populations were pooled. Currently I think I have a quite crude filter which is to exclude sites where the MQ score is below 20. But I am thinking that maybe it is better to exclude individuals based on their genotype score instead(so replacing a bad genotype score call with ./.). Or perhaps a combination of basic hard filters will work best. If there is a good way to filter out on an sample by sample basis, excluding genotype calls for each sample when they have a low GQ score I think this is probably best, rather than filtering out the entire site because it has a MQ<20.

A final consideration is whether it matters to filter the population files seperately or to filter the single VCF file containing both populations before splitting.( I am worried if I filter the joint vcf file it might exclude sites that are only scoring low in one population and therefore I might be excluding interesting regions where there has been a large deletion in one of the populations) Though again filtering out individuals rather than sites should avoid this problem.

If anyone has a set of filters, or a single filter they think will remove most of the calls that are likely to be errors, due to low mapping scores etc (i can't really filter for low coverage as I always have low coverage) that would be really appreciated. Any examples command lines for doing this are also appreciated (already spent some time having to go back because of errors in this regard).

Really grateful for any thoughts or comments,

Best regards