Hello,
I am trying to filter text file for missing and redundant markers in R. But R session gets aborted. I think this is because of the large file size because this code works on other small files. This file has SNP markers for 50 genotypes and is 25 Gb in size. So I want to do the same filtering for missing and redundant SNPs in the cluster and then import the file in R for further analysis. I am not sure how can I do it in the cluster.
Additionally, I think there is a problem in saving text file to RData file as well. I appreciate your time and effort for any help. Thank you!
1 answer
Growing objects in a loop is not a good idea. I am guessing you are running out of memory, as R keeps copying the same object over and over again with increased size, try something like below: load data, manipulate, results are in a list, then bind the lists:
library(data.table)
res <- rbindlist(
lapply(1:40, function(i){
load(paste0("file", i, ".RData"))
uni <- unique(uni)
na <- rowSums( is.na (uni))
uni[ -which(na > (0.05 * ncol( uni))), ]
})
)
Log in to answer this question.
-
Single cell cluster naming
written by sc_confusion •
-
beagle to gds for admixture mapping
written by nanodano •
-
convert beagle genotypes to vcf
written by bsp017 •
-
Is there any solution to reduce the size of .RData?
written by PNTick •
-
bcl2fastq discard complete read
written by DCZ •
-
Representation of common SNPs in text files
written by evelyn
-
R error: vector memory exhausted
written by evelyn
-
Error in (function...) thrown by WGCNA tutorial (R)
written by Kristin Muench
-
Split individual SNP genotypes into two columns per call in R when there are 2.44 million columns
written by brianscomer •
-
Plink Flag? I Have A Dataset That Is Formatted Weirdly And I Keep Getting An Error. How Do You Use …
written by Joe •
Have you tried this using
freadanddata.table? I think it might work, becausedata.tableis a lot more performant with tables in the millions-of-rows range of records thandata.frameever can be. If I recall correctly,freadeven has streaming support so it won't need to store the entire file in memory.Yes, I imported the file using
data.tableandfread. And then I used the above command to filter markers. But R session still gets aborted.Can you edit your question and add the exact code you're using please? You might also want to check stackoverflow on how to enhance the performance of
fread.I have edited the question and I have added an additional problem in saving text file to RData file which I think could be the reason.
Side note: It's not good practice to use keywords as variable names, like you're doing with
na.