Results of two sequential bash pipes into a variable instead of a file?

Hi!

I would like to put the result of a pipe into a variable instead of a file. But it does not work as intended.

The well working script with the pipe resulting in a file is like that:

for sample in *.csv;
do
pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'| cat >Ldha/${input}.tsv
done

I tried to funnel it into a variable called "input" with these ways:

read input <(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
input=`pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'`
input=$(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
pr -m -t -s\  $sample  Ldha.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g' | read input

Nothing worked. If i run it with saving to files, it works. But saving well 50 000 files for each run... In the end, it should look like that (using a perl script for pearson correlations from the web):

for sample in *.csv;
do
input=$(pr -m -t -s\  $sample  Ets1.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g')
perl corr.pl -i $input  -col 1 2 | cat >>correl_Ets1.csv
done

But i fail to get the output from the pipe into a variable. Any ideas?

Best wishes,

Norman

bash pipe variable shell script

• 1,630 views • link

updated 6.7 years ago by Biostar

• written 6.9 years ago by normanreppingen •

Some comments:

You should provide small example files to allow us reproduce the issue
What "nothing works" means? What is stored in $input with each command? Did you check with echo "$input"?
It should be awk, not Awk

The following simple example shows no problems assigning piped stdout to a bash variable:

$ awk '{print $2"\t"$13}' relAbun.tsv | head
U2  U13
0   0
0.1 0
0   0
0   0
0   0
0.3 0.2
0   0
0   0
0   0

Now put the result in a variable:

$ input=$(cat relative_abundance_raw.tsv | awk '{print $2"\t"$13}' | head}' relAbun.tsv | head)
$ echo "$input"
U2  U13
0   0
0.1 0
0   0
0   0
0   0
0.3 0.2
0   0
0   0
0   0

• 0 views • link

written 6.9 years ago by h.mon

Dear h.mon,

thank you very much for your reply and the brillant idea with echo $input. Echo input looks good. But now what happens is the following: The perl script is taking the first value of the first file as a filename. Then it takes the first value from the second file as a filename. So, the input is transferred indeed to the variable input. But the filename, which would be taken from $sample is now lost and not fed to the perl script. I am not sure if this can be fixed.... Maybe i need to feed this as an array? But still, how would i pass the filename....

Best regards and thank you very much again,

Norman

• 0 views • link

written 6.9 years ago by normanreppingen •

Without knowing the internals of the perl script, I don't even know if this can be "fixed". But my gut feeling is the perl scripts expects the data to be stored in a file, and -i something is interpreted as "read the data from file something" - thus, save the results to a file and subsequently feed this file to the perl script in the same loop.

• 0 views • link

written 6.9 years ago by h.mon

Hi!

Your gut feeling is very reliable. I implemented your suggestion to just open the file. I almost accepted to produce 50k nonsensical files, but then it came to me that i also could delete them in that loop. 😂

So, this thing is now using two files to compute the pearson correlation, using the script from Didier Gonze. http://homepages.ulb.ac.be/~dgonze/SCRIPTS/PERL/correlation.pl

for sample in *.csv;
do
comparator=Csf3
pr -m -t -s\  $sample $comparator.csv | Awk '{print $6"\t"$13}' | sed 's/\./,/g'| cat >${sample}.tsv
perl corr.pl -i "$sample".tsv -col 1 2 | cat >>results$comparator.tsv
rm "$sample".tsv
done

Thank you very much for your help! :-)

Best regards,

Norman

• 0 views • link

written 6.9 years ago by normanreppingen •

0 answers

No answers yet.