Hi
I wrote a bash script with awk and sed commands in it to parse text files in a directory. This script runs without giving any output. What is wrong with it?
#!/bin/bash
for i in *.txt; do
awk -F'\t' '{print $16}' $i | sed '1d' | tr -d '\r' >$i_get_signals.sh
done
•
written
by
bk11 •
2 answers
try this:
for i in *.txt; do awk -F'\t' '{print $16}' $i | sed '1d' | tr -d '\r' > $i"_get_signals.sh" ; done
But whatever you are doing, this is long way of doing it. OP code is simply extracting 16th column, removing the header and removing carriage returns and writing it to a text file. This can be done better way.
Change the code to remove old file extension in new files:
for i in *.txt; do awk -F'\t' '{print $16}' $i | sed '1d' | tr -d '\r' > ${i%%.*}"_get_signals.sh" ; done
Here's another way to do things that might be a little simpler and help with debugging:
#!/bin/bash
for i in `ls *.txt`; do
echo $i
tail -n+2 $i | cut -f16 | tr -d '\r' > $i_get_signals.sh
done
I try to use echo and tail/head to debug pipelines before running them on the full datasets.
I try to minimize use of sed in scripts that I share with others, unless I know exactly what platform I'm doing things on.
If you want to make sure that all your inputs have 16 or more columns:
#!/bin/bash
for i in `ls *.txt`; do
echo $i
awk '{ if (NF < 16) { exit -1; } }' $i
done
This script will exit with a non-zero error code, if any file in the output of ls *.txt has 15 or fewer columns.
As noted in comments, avoid using .sh as a file extension unless you're writing output to a text file that you will run as a shell script. Use .txt or other extensions that are more appropriate. It may seem like nitpicking but details can matter when debugging things.
I am not sure if tr -d '\r' > $i_get_signals.sh works. This should be tr -d '\r' > $i"_get_signals.sh" . Shell looks for $i_get_signals.sh which doesn't exist. @ Alex Reynolds .
I usually use ${i}_get_signals.sh, don't know if that has any functional differences.
my understanding is that {} expands a variable, when they enclose a variable. Here, ${i} expands i (variable) so that ${i}_get_signals.sh becomes test.txt_get_signals.sh. If curly braces are absent, it becomes $i_get_signals.sh and bash would consider the entire string as variable (IMHO). WouterDeCoster. Based on this notation only, variety of string manipulation functions (on variable name) work in bash.
In this article https://www.linuxjournal.com/article/8919, they explained why ${foo}bar works and $foobar doesn't work with small examples.
Log in to answer this question.
-
Filter out low coverage contigs
written by melissachua90 •
-
BEDOPS bedmap confusing file for option
written by N •
-
Why my files ended with ^M ?? (shell/bash)
written by nlt_220593 •
-
awk column number is a variable
written by DBScan •
-
How can I use getopts in a script that appends lines from files in a separate directory to a new fi…
written by Digsby •
-
awk, copy and paste single line command
written by 3335098459 •
-
Bash for loop help please
written by Tark •
-
Bash script error on reading coordinates in a file
written by Ali HEBRA •
-
pick the longer contig splice variant if more than one splice variant exist
written by anahita •
-
Nucleotide Match With Unix Tools
written by Michael Schubert
We need more information. What does your input data look like? What output do you expect? Are you running the script while you're in the directory that the
.txtfiles are in? How are you invoking the script?You will also need to tell us why this is a bioinformatics question and not just a programming question else we may close it as off topic.
Yes I am running the script in the same directory that has the text files. This is related to bioinformatics as it is used to get URL to download histone proteins binding signals from different cell types.
It is related to bioinformatics, but only borderline. It is essentially just unix text processing, and there are more specialised forums for this. We usually tolerate posts like this as we understand that these are common problems in bioinformatics, but it's not pure bioinformatics.
Add
teestatements between stages to inspect output. Do your text files have 16 columns?Yes the text files have 16 columns:
Why do you send the result to a
.shfile?I tried sending the result to .txt file but also it did not work.
It shouldn't make a difference, but with the extension
.shyou suggest this is a shell script.That is because OP is using *.txt in loop and doesn't want output with .txt for obvious reasons. Instead OP could have used any other extension instead of sh (.out, .text, .file etc or without extension). WouterDeCoster
Yeah, of course, and while file extensions have almost no meaning in unix it is quite confusing to use something unexpected.