This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Converting PLINK binary files into python dataframe

I'm working with a genetic dataset (roughly 23,000 samples and 300,000 SNPs as features). I got my files in PLINK binary format files (.bed, .bim, .fam). Listed below are their sizes:

.bed file : 1.6G .bim file = 9.3M .fam file = 737K My aim is to convert them into (pandas) dataframes and then start my predictive analysis in Python (it's a machine learning project).

I was adviced to combine all 3 binary files into one vcf (variant call format) file. The result (vcf file) is a 26G file using PLINK software. There are python packages and codes for converting vcf files into pandas dataframes, but my remote system memory is limited (15 Gi). Due to the nature of the dataset, I can only work with university computers.

My question is, considering all my limitations, how do I convert my dataset into a dataframe that can be used in machine learning? Let me know if you need more details.

python frame vcf data plink genetic
• 2,457 views • link
written by bbehrooz • bbehrooz

1 answer

Pandas is notoriously slow at reading in large datasets

I would recommend an alternative like polars or datatable:

find more information here:

https://towardsdatascience.com/getting-started-with-the-polars-dataframe-library-6f9e1c014c5c

or something else from this list:

http://theautomatic.net/2021/10/09/faster-alternatives-to-pandas/

• 0 views • link
written by Istvan Albert Istvan Albert

Thank you so much for your input.

• 0 views • link
written by bbehrooz • bbehrooz

Log in to answer this question.

More posts like this