Hi guys, This might a ridiculously simple question on makefiles so bear with me :)
Im looking at implementing makefiles for my next-gen sequencing processing instead of tying everything up in bash scripts. The Makefile below uses fasta files and runs them using the BLASTX alignment program PAUDA). Im able to run the Makefile using individual files, but when I run this on multiple files (using the % pattern syntax), I just end up making one file called %.blastx. Ive had problems with this for a while, so Im really keen to find a solution asap
all: %.blastx
clean:
rm -f *.sam *.pna
BLASTSH='/opt/local/pauda/lib'
DB='database_file.db'
%.pna: %.fasta
sh $(PDA_PATH)/z3_dna2pna.sh $< $@
%.sam: %.pna
sh $(PDA_PATH)/z4_bowtie-on-pna.sh $< $@ $(DB)
%.blastx: %.fasta %.sam
$(PDA_PATH)/z5_sam2blastx.sh $^ $@ $(DB)
Cheers
Okay you have a set of target rules (see 10.5.2 Pattern Rule Examples):
# Rule for .fasta to .pna
%.pna: %.fasta
sh $(PDA_PATH)/z3_dna2pna.sh $< $@
# Rule for .pna to .sam
%.sam: %.pna
sh $(PDA_PATH)/z4_bowtie-on-pna.sh $< $@ $(DB)
# Rule for .fasta + .sam to .blastx
%.blastx: %.fasta %.sam
$(PDA_PATH)/z5_sam2blastx.sh $^ $@ $(DB)
However you do not have the targets to trigger the rule chain, for that you need to use wildcard processing and a little pattern substitution:
# Set of fasta files to process.
fasta_files := $(wildcard *.fasta)
# For each .fasta generate a .blastx
blastx : $(fasta_files:.fasta=.blastx)
Then all you need is the primary target:
# Collation target
all : blastx
Then test with a small set in single process mode. If that works then up the number of processes with '-j' (assuming GNU make).
Awesome thanks for that explanation. What happens with multiple wildcards? For example, say I have Illumina paired end sequencing files (sample1_R1.fastq.gz and sample1_R2.fastq.gz), can you easily run both paired files using the same pattern substitution rules as you showed above?
# Set of fastq files to process.
fastqR1_files := $(wildcard *R1.fastq.gz)
fastqR2_files := $(wildcard *R2.fastq.gz)
# For each paired fastq.gz generate a sam file
fastqpaired : $(fasta_files:.fastq.gz=.sam)
Assuming you have a rule something like:
%.sam : %_R1.fasta.gz %_R2.fasta.gz
genSam.sh $^
Then you could use something like:
# Set of fastq files to process (since they occur in pairs use one to avoid duplicates)
fastqR1_files := $(wildcard *_R1.fastq.gz)
# For each pair of fastq files generate a sam file.
fastqpaired : $(fastqR1_files:_R1.fastq.gz=.sam)
That expands to a target like:
fastqpaired : sample1.sam sample2.sam sample3.sam
And thus invokes the appropriate rule chain.
For more details of how to write makefiles for GNU make see the manual and the book "Managing Projects with GNU Make".
I assume it is GNU make? See the fine manual: https://www.gnu.org/software/make/manual/html_node/Wildcards.html#Wildcards and https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html#Multiple-Targets
Short % is not a wildcard but e.g. * is. Could you try something like:
*.blastx: *.fasta *.sam
$(PDA_PATH)/z5_sam2blastx.sh $^ $@ $(DB) # assuming this is correct
If blastx is the "final" target, then I think you can just write it as:
blastx: *.fasta *.sam
•
written
by
Michael Dondrup
While the use of wildcard target names works for simple cases, this is not one of those cases. This fails due to the way the wildcards are resolved:
Log in to answer this question.
Btw, is this question borderline off-topic?