If you are familiar with Python and the pandas package, I wrote the pyvcf.VcfFrame class which stores VCF data as pandas.DataFrame to allow fast computation and easy manipulation (click here to see the class). It also supports plotting as a bonus. I'm not sure if this approach is robust enough for you -- pyvcf.VcfFrame will be as fast and efficient as pandas.DataFrame gets -- but I once needed to "interactively" play with a lot of VCF data and doing it with CLI such as bcftools was not an option so I just ended up writing the class myself. It's 100% open source and free, and I even invite others to contribute if they can improve it. That being said, I'd be curious to learn what other tools are available to this end.

Thanks for the question. I am currently dealing with similar problem and haven't found a fully satisfying solution either. But I can share some tips.

I used to do some population-genomics analyses on a non-model organism, using RADseq and exome capture (WES) data for up to a few hundred samples. The uncompressed VCFs were maybe close to 1GB. These days, I work on human WGS datasets like SGDP, HGDP, and soon perhaps even the recent 30x resequencing of the g1k project data. These datasets have close to 100M sites and a some 930 samples. The uncompressed files are a few hundred GBs, compressed HGDP is just under 200GB.

I tried a few different approaches and tools to tackle some questions:

• vcfR is a simple modern R library for working with VCF data. It is great with smaller data, like those from RADseq or WES, but I would not try it on my current HGDP dataset, even on a cluster. Any operation would take half an hour or more. However it can read VCF files in chunks, so it may work for you in some way.
• plink can take VCF as input (with a --vcf flag) and do some analyses, like LD-based prunning and PCA or MDS. The new version (1.9 and 2) is very fast. If you have done all the quality-based filtering and have a final dataset, you could convert it to plink format and use it with admixtools2 (another R package) to get some interesting features out of the dataset, such as per-population allele frequency spectra, f-statistics and so on.
• bcftools query allows some manipulations and filtering of VCFs, while the output can be custom-defined. tabix can help with reducing the data to some region of interest. awk or mawk (faster awk) can help with further processing of the queries. Overall, bcftools is good for filtering, but not for sub-setting the data (at least I did not find a way yet). This can sometimes be solved by tabix, but not always. And tabix works on BGZF-compressed VCFs, not on BCFs.
• bgzip by itself can be useful. It can decompress files with multithreading (--threads INT), similar to pigz, and pipe the output (with -dc) to e.g. awk for queries that cannot be done with bcftools.
• BGZF-compressed VCF files can be indexed with both tabix and grabix, which can then provide random access to the VCF. While tabix extracts data based on genome coordinates, grabix instead uses file coordinates (line numbers) to extract data.

I'm also currently looking into Julia language and its ecosystem, but I haven't gotten very far yet (too busy). I know they have some packages for reading VCFs and doing some population genetics (mostly modelled after adegenet package in R, which btw could be also useful. Or pegas).

So overall I mostly use bcftools, awk and plink, with help of tools like bgzip and tabix or grabix. Not exactly what you asked for, but this is how it is for now (as far as I know).

But maybe somebody else can show us better ways.

I wrote vcf2table: http://lindenb.github.io/jvarkit/VcfToTable.html

SwingVcfView https://lindenb.github.io/jvarkit/SwingVcfView.html

(....)

Hello,

SCI-VCF can be a solution. It's user-friendly as it comes with a GUI, allowing interactive analysis like summarizing, filtering, comparison and visualization without the need for any programming.

Certain modules would become runtime-intensive for large files. However, the tool is shiny-based and can be deployed on servers for handling larger VCF files.

Some useful links: SCI-VCF | GitHub | Documentation

Hope this helps!