r is running out of memory

enter image description here When i am trying loading gene expression counts downloaded from tcga (I have 859 samples with raw counts of 60000 rows). I GET THIS ERROR and R STOPPED FOR NO REASON! I check the memory to find out that my memory is nearly full.BUT i have 16GB RAM.

    counts_df <- counts_files %>% 
  lapply(function(x) {
    tmp <- read_tsv(x, col_names = F) %>% 
      purrr::set_names("gene_id", basename(x))
    cat(which(counts_files == x), "of", length(counts_files), "\n")
    return(tmp)
    }) %>%
  reduce(function(x, y) full_join(x, y, by = "gene_id")) %>% 
  dplyr::select(gene_id, metadata$file_name) %>% 
  set_names("gene_id", metadata$TCGA_id_full) %>% 
  dplyr::slice(1:(nrow(.)-5))

tcga memory r

• 2,901 views • link

written 5.0 years ago by jenyfer •

1 answer

It seems that R is using ~12GB of RAM. The remainder will be used up by Windows and associated processes.

You may consider one or more of these options:

restart the computer and run just R / RStudio (and nothing else)
avoid using RStudio - it uses up an unnecessary amount of extra RAM
avoid using the dplyr functions - one has greater control over the data flow with base R functions
avoid using %>%; instead, divide your code into different sections. You can use rm() and gc() between each section to remove unneeded objects and clear memory, respectively.
rent an Amazon EC2 instance that has more RAM
pre-filter the input object - most of those genes will have just 0 counts across all samples
import the data via data.table::fread()

If you want further assistance on improving the code, then please provide the following:

a link to the data that you are using
a sample of the data pasted here
a sample of how you want the data to appear after all processing, i.e., desired output

Kevin

• 0 views • link

written 5.0 years ago by Kevin Blighe

Thanks a lot for foryour kindness advise. I processed this on my old computer i bought 6 years ago with 8GB RAM. SAME code SAME pakages. and it produce results of 60000row and 789columns data。 However i wil still try your advise latter. thanks

• 0 views • link

written 5.0 years ago by jenyfer •