This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Create txt file in a for loop

I have some paired-end sequencing data in a directory:

sam1_R1.fastq.gz
sam1_R2.fastq.gz
sam2_R1.fastq.gz
sam2_R2.fastq.gz

I want to generate txt files as follow (for each sample (sam) separately:

$less sam1.txt

(dir)/sam1_R1.fastq.gz
(dir)/sam1_R2.fastq.gz

$less sam2.txt

(dir)/sam2_R1.fastq.gz
(dir)/sam2_R2.fastq.gz
bash loop linux
• 2,615 views • link
updated by padwalmk padwalmk • written by Ati • Ati

what's the difference between the two dirs ? may be you wanted sam2_...

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Sorry! it was a typo!

• 0 views • link
written by Ati • Ati

try this:

$ find ../ -type f -name "*R1.fastq.gz" | sed 's_../__'

assuming that dir in OP is parent directory and files share the same pattern as in OP.

• 0 views • link
written by cpad0112 cpad0112

What does this do, cpad? It seems to be just a cd .. && ls *R1.fastq.gz ** cd dir. OP seems to want each a list of sample's fastq files in separate text files.

• 0 views • link
written by Ram Ram

it would print: <parent_directory>/filename (for eg. test/sam1_R1.fastq.gz, test/sam1_R2.fastq.gz ....so on, assuming that test being parent directory). @ RamRS

output would look like this (on dummy .fq files in a directory named "test"):

$ cat sam1.txt 
test/test5_R1.fq
test/test7_R1.fq
test/test1_R1.fq
test/test2_R1.fq
test/test8_R1.fq
test/test10_R1.fq
test/test4_R1.fq
test/test6_R1.fq
test/test9_R1.fq
test/test3_R1.fq
• 0 views • link
written by cpad0112 cpad0112

OK, but this is not what OP wants at all. It actually leads them down a slightly wrong path.

• 0 views • link
written by Ram Ram

is this OP wants? :

$ ls *.gz
sam1_R1.fastq.gz  sam1_R2.fastq.gz  sam2_R1.fastq.gz  sam2_R2.fastq.gz

$ ls *.fastq.gz  | sed 's/\(.*\)_\(.*\)/\1\\\1_\2/' | awk -F '\' '{print > $1}'

or 

$ find  -type f  -name "*.fastq.gz" -exec basename {} \;| awk -F '_'  '{print $1"\\"$1"_"$2>$1.txt}'

$ cat sam1
sam1\sam1_R1.fastq.gz
sam1\sam1_R2.fastq.gz

$ cat sam2
sam2\sam2_R1.fastq.gz
sam2\sam2_R2.fastq.gz

try this with awk:

$ awk 'FNR==1 {FN=FILENAME;gsub(/_R[^0-9],"",FN); print FN"\\"FILENAME > FN".txt" }' *.fastq.gz
• 0 views • link
written by cpad0112 cpad0112

I think that's close to what they want. I also think we should not be investing so much effort into a pure shell question that the OP has already solved.

• 0 views • link
written by Ram Ram

2 answers

I found the solution myself:

for sample in samples;
  do ls (dir)/*.fastq.gz | grep $sample > (dir)/$sample.txt
done
• 0 views • link
updated by Ram Ram • written by Ati • Ati

That is an extremely roundabout way to go about this. Why do you need ls in a loop if you already have a samples variable with a list of samples?

You could do something like padwalmk suggests.

• 0 views • link
written by Ram Ram

Here you go .....

ls *.gz | cut -f 1 -d "_" | sort | uniq |  while read line
do 
    touch "$line".txt 
    echo "/dir/$line"_R1.fastq.gz > "$line".txt 
    echo "/dir/$line"_R2.fastq.gz >> "$line".txt
done
• 0 views • link
updated by Ram Ram • written by padwalmk padwalmk
  1. Why the touch?
  2. Why not a simple echo -e "/dir/${line}_R1.fastq.gz\n/dir/${line}_R2.fastq.gz" > "${line}.txt"? That would be one line instead of 3.
  3. Why sort | uniq and not sort -u?
• 0 views • link
written by Ram Ram

Thank you for guiding, I am a novice and I have started shell scripting, so learning, but happy for your advice next time will try to implement it.

• 0 views • link
written by padwalmk padwalmk

That's great! You will enjoy learning the various ways to get a job done on the shell. If like me, you were taught to create files using the touch command, be aware that it's not necessary. >, 2>, etc - redirection operators as well as programs that write to files, such as tee and editors like emacs, nano, vim, vi can directly create files.

echo has an option which enables us to use -extended expressions, such as \t\ and\n.sort -uachieves the same assort | uniq, so it's better unless we need someuniqspecific features, such asuniq -c`.

• 0 views • link
written by Ram Ram

Thank you, Yeah, I am a hardcore biologist, but I love this, so exploring my own so suggestions are always welcome. Happy New Year...

• 0 views • link
written by padwalmk padwalmk

Log in to answer this question.

More posts like this