This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Space Required to run 15 GB File in python

I want to know how much ram memory is required to run a 15 GB data file and to clean datasets through python pandas.

I have a file of 14.8 GB which contains genes information. It contains a total of 45804630 rows and 39 columns

When tried to open it by taking 1000000 rows through pandas it was working fine

Code:

data = pd.read_csv("CosmicGenomeScreensMutantExport.tsv", sep= '\t',nrows=1000000)

But when I am trying to read all datasets at once, it hangs my pc. I only have 4 GB ram. So should I increase my RAM?

python pandas
• 2,443 views • link
updated by Ram Ram • written by anasjamshed anasjamshed

If you need to have a 15G file in memory then you obviously need more than 15G RAM, possibly a lot more depending on what precision you are reading your data (e.g. float16, float32, etc.). The pandas concat function could be of use to you, check e.g. this

• 0 views • link
written by 5heikki 5heikki

I am trying to use this code:

#Reading file in chunks as data is very large
chunks=pd.read_csv("CosmicGenomeScreensMutantExport.tsv",chunksize=1000000,sep='\t')

#Make dataframe by concatating list of chunks
df= pd.concat(list(chunks), ignore_index=True)

from my friend pc who has 8 gb so will it work?

• 0 views • link
written by anasjamshed anasjamshed

I don't understand why you would need to have the whole file in memory if you only need to "clean" it, whatever that means.. sounds like something that shouldn't require all the rows at the same time, so why not just process the file in chunks?

• 0 views • link
written by 5heikki 5heikki

but i want to clean all datasets that's why i need to upload all at once

• 0 views • link
written by anasjamshed anasjamshed

I'm just guessing here, but you probably need like minimum 64G RAM if you want to have your file in memory..

• 0 views • link
written by 5heikki 5heikki

Ohh no. How can I acquire 64 GB

• 0 views • link
written by anasjamshed anasjamshed

0 answers

No answers yet.

Log in to answer this question.

More posts like this