Remove NA from file while reading
Heys,
I have a big dataset (37Gb) with a lot of missing data as NA. I know how to remove NAs once the file is read, but I was wondering if there is any way to remove the NAs while (or without) reading the file, so the read file is already without NAs. If not I have to read all the file and as you can imagine I will need a lot of RAM.
Thanks in advance!!
2 answers
Assuming you want a solution in R, I would do something along these lines using data.table::fread:
library(data.table)
dat <- fread(cmd="grep -v -w 'NA' bigfile.txt")
of course you would need to customize the shell command cmd to your case but hopefully you get the idea.
Use a stream-oriented approach to process your data. For example readr with R:
• 0 views
•
link
written
by
Istvan Albert
Log in to answer this question.
More posts like this
-
Error in mitch reports in R
written by Amr •
-
qc3C software for testing Hi-C data
written by gubrins
-
Decontamination of genome assembly
written by gubrins
-
Plink ped format
written by gubrins
-
iqtree vs raxml
written by gubrins
-
Filter number of N's fasta file
written by gubrins
-
How much weights a million reads
written by gubrins
-
Multisample BAM file
written by gubrins
-
where to practise bash
written by gubrins
-
fastQTL phenotypes with no cis-variants
written by Cindy •
You don't need to read the entire file into memory to remove the NA's. You can read and write the file line by line to remove the NA, using python or R. You may be able to edit in place with
sed.