Hi I'm using SUN Grid Engine to submit jobs, written by shell script like this:
#!/bin/bash
#$ -pe single 24
#$ -V
#$ -cwd
#$ -o $HOME/sge_jobs_output/sge_job.$JOB_ID.out -j y
#$ -S /bin/bash
#$ -l mem_free=8G
cd $HOME/scratch/bwa2/
job_number=$SGE_TASK_ID
SAM_NAME_1=`head -"$job_number" master_list_1|tail -1`
/share/bin/bwa aln hg19_index "$SAM_NAME_1".recal.fastq.gz > "$SAM_NAME_1".sai
#Do sth. here to save the output as text file. Here by output I don't mean
# .sai file, but the error report like "10000000 sequences have been processed",
#And text file name is SAM_NAME_1, which is a variable
Maybe I didn't make quite clear. As shown as the end of the script, I want to put some command so that the error report can be saved as .txt file. How can I do that? (I know we can use > .txt at prompt; but here the .txt file name is a variable, so I must put command within script)
WHat I want is similar to $HOME/sgejobsoutput/sgejob.$JOBID.out -j y; but I want name as SAMNAME instead of JOBID. Thanks!
SO actually this turned out as a bioinformatic problem. :)
Edit:
I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt
But it still doesn't work..$SAMNAME1 part doesn't act as variable..
Is there any other way, which does NOT rely on this SGE flag? thx
•
written
by
Bioscientist
2 answers
The way to redirect STDERR to a file in SGE is by using the -e flag. The trick for your program is to substitute the SAM file name for the job name in the argument for -e. For you, all you should have to add is
#! -e "${HOME}/sge_jobs_output/sge_job.${SAM_NAME}.txt"
I haven't tested this out, but I know for certain that redirecting error output from SGE is with -e. Naming the file should be the easy part (I hope...).
Hope that's useful!
hi, seems it still doesn't work. It's showing 'sge_job.$SAM_NAME.txt', seems SAM_NAME doesn't work as variable...
I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable..
When I set error output using the -e command, I set the filename at the command line (e.g. qsub ... -e [error file name] ...), and I know it will work there, not sure if it will work in the script, but I think it should. Do you know if $SAM_NAME_1 has the the value you want it to? That would be the first place I would check. Unfortunately, I'm away from my grid right now so I can't check it for you. Glad to help you trouble shoot though!
hi, if I do in the command line as: "qsub XXX.sh -e $SAM_NAME_1" ,but here SAM_NAME_1 is a variable. Does that work?
Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e $sge_job.{SAM_NAME_1}.txt. You probably had that in mind, but just in case...
Hi bioscientist, that should work (it always does for me). Although be careful to name it right. In your case, it should be qsub XXX.sh -e "$sge_job.{SAM_NAME_1}.txt". You probably had that in mind, but just in case...
As brentp notes, you just need to change "$SAM_NAME_1" to ${SAM_NAME_1}. You'll also need to continue the script with:
bwa samse ${SAM_NAME_1} ${SAM_NAME_1}.sai ${SAM_NAME_1}.recal.fastq.gz
I didn't test this, but you should get the idea. The important thing when using a batch system such as SGE is to make sure that your script runs outside of the batch system before trying to use the batch system.
HI Sean, sorry I don't make clear what I want to do. I don't mean to make .sai file as .txt; but I want the error report like "how many sequences have been processed" saved as .txt. How can I do that?
See the -e flag for SGE.
I tried: #$ -e $HOME/sge_jobs_output/sge_job.$SAM_NAME_1.txt But it still doesn't work..$SAM_NAME_1 part doesn't act as variable...
Yes. You'll need to specify the name directly in the shell script. There is no variable interpolation in the SGE directives.
Yeah, but how can I redirect the error output in the shell script? ANy clues?
Log in to answer this question.
-
How to use cd-hit-para.pl in SGE?
written by FelipeMSD •
-
Error while running All-against-All alignments in OMA
written by pragyadeeproy •
-
Can't open file 'features': [Errno 2] No such file or directory
written by David_emir
-
defuse running error message
written by MS •
-
Running Interproscan on the cluster
written by seta
-
How to set-up mutiple SPAdes runs for various genomes
written by queenofcold •
-
Create Files From Double For Cycle
written by filipzembol
-
Root File Output Problem
written by Bioscientist
-
Sequence Number Count In Fastq.Gz File
written by Bioscientist
-
Use Python To List Filename With Specific Extensions.
written by Bioscientist
I think this isn't really bioinformatics related. but you likely want to use ${SAM_NAME_1}.recal.fastq.gz and ${SAM_NAME_1}.sai
Yeah, I know it's NOT bioinformatics-related; I post on stack overflow, but got no reply. So how should I do to save output as text file here? thanks.
Here's a fishing rod: the Advanced Bash-Scripting Guide http://www.tldp.org/LDP/abs/html/ . See Chapter 4. Introduction to Variables and Parameters.