This is a test version of Biostars. For the public version, visit https://www.biostars.org.
R Studio memory on Mac

I am trying to import three really big files (each file size >100GB) to R Studio but it complains of memory. I am using this version:

platform       x86_64-apple-darwin15.6.0   
arch           x86_64                      
os             darwin15.6.0                
system         x86_64, darwin15.6.0        
status                                     
major          3                           
minor          6.0                         
year           2019                        
month          04                          
day            26                          
svn rev        76424                       
language       R                           
version.string R version 3.6.0 (2019-04-26)

I tried increasing the memory limit by using (.Renviron) on terminal but it gives the same error.

r
• 2,977 views • link
written by evelyn evelyn

How much actual memory does your mac have? There is no substitute for not having enough RAM unfortunately.

• 0 views • link
written by GenoMax GenoMax

My Mac has 16 GB memory.

• 0 views • link
written by evelyn evelyn

I don't think this is going to work as is.

Edit: @zx5784's solution of using virtual memory on disk may work. With such large data set you may need to have patience in equally large amounts.

• 0 views • link
written by GenoMax GenoMax

What kind of data is this? Alternatives:

  • HPC with tonnes of memory
  • Load data in chunks, do stuff, save results, load another chunk etc.
  • Use packages that uses hard drive as memory and "pretend" to have it as full object in R, e.g.: "bigmemory" package.

See CRAN TaskView for HPC: Large memory and out-of-memory data

• 0 views • link
written by zx8754 zx8754

These are vcf files.

• 0 views • link
written by evelyn evelyn

Do you need all the data at once, maybe slim it down first: filter on Samples, on Variants, or split on Chromosomes, etc. See bcftools for manipulating VCFs.

• 0 views • link
written by zx8754 zx8754

What operation are you trying to do with them? Perhaps there are command line alternatives that can be used instead of R. Are these files compressed or not?

• 0 views • link
written by GenoMax GenoMax

I am trying to use them for Upset R plot and these are not compressed.

• 0 views • link
written by evelyn evelyn

Is it to show overlapping Variants or Samples? Again you can get that info using bcftools, then file will be manageable for R.

• 0 views • link
written by zx8754 zx8754

0 answers

No answers yet.

Log in to answer this question.

More posts like this