I've ran into a problem trying to update my snakemake workflow to accept input from multiple directories.

Originally I had used glob_wildcards() _within_ my Snakefile like this:

# Full path to raw files 
SAMPLES, = glob_wildcards("{sample}_R1.fastq.gz")

rule all:
    '''
        The target rule for the workflow.
    '''
    input:
        expand('fastQC/before_trim/{sample}_R1_fastqc.html', sample=SAMPLES),
        expand('fastQC/before_trim/{sample}_R1_fastqc.zip', sample=SAMPLES),

This is not ideal for the obvious reasons that it is 1) Contained within the Snakefile and 2) will only work on files from one directory (in this case the $PWD)

Rather than a user messing around with an optimized Snakefile I would like them to use a samples.txt file that contains information for the file name and file location.

For example a samples.txt file that looks like this:

sample  location
BC1217  /foo/bar/pow/fastq
470     /wam/bam/crash/data
MTG109  /penguin/fire/station/

In addition, to the samples.txt there is also a config.yaml which contains information on the reference files/locations and will _eventually_ contain choices for different rules (e.g. variant calling with Freebayes or DeepVariant) controlled with IF/ELSE statements - but that's a whole'nother post. It looks like this:

# Files
REF_GENOME: "WS265_wormbase/c_elegans.PRJNA13758.WS265.genomic.fa.gz"
GENOME_ANNOTATION: "WS265_wormbase/c_elegans.PRJNA13758.WS265.annotations.gff3"

# Tools
QC_TOOL: "fastQC"
TRIM_TOOL: "trimmomatic"
ALIGN_TOOL: "bwa"
MARKDUP_TOOL: "picard"
CALLING_TOOL: "varscan"
ANNOT_TOOL: "coovar"

I actually based this workflow on an example I found here but it doesn't work for me.

I'll post the most pertinent parts of the code here but the full Snakefile can be found at the gist

# Load Samples ----------------------------------------------------------------

import pandas as pd
# getting the samples information (names, path to r1 & r2) from samples.txt
samples_information = pd.read_csv("samples.txt", sep='\t', index_col=False)
# get a list of the sample names
sample_names = list(samples_information['sample'])
sample_locations = list(samples_information['location'])

rule qc_before_trim:
    input:
        r1 = expand('{FASTQ_DIR}/{sample}_R1.fastq.gz', FASTQ_DIR=sample_locations, sample=sample_names),
        r2 = expand('{FASTQ_DIR}/{sample}_R2.fastq.gz', FASTQ_DIR=sample_locations, sample=sample_names)
    output:
        expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_R1_fastqc.html', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names),
    expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_R1_fastqc.zip', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names),
    expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_R2_fastqc.html', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names),
        expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_R2_fastqc.zip', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names)
    log: expand('{LOG_DIR}/{sample}-{QC_TOOL}-qc-before-trim.log', QC_TOOL=config["QC_TOOL"], LOG_DIR=dirs_dict["LOG_DIR"], sample=sample_names)
    resources:
        mem = 1000,
        time = 30
    threads: 1
    params:
        dir = expand('{QC_DIR}/{QC_TOOL}/before_trim/', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"])
    message: """--- Quality check of raw data with FastQC before trimming."""
    shell:
        'module load fastqc/0.11.5;'
    'fastqc -o {params.dir} -f fastq {input.r1} &'
    'fastqc -o {params.dir} -f fastq {input.r2}'

I guess the problem is that expand() generates all of the combinations versus what I want is iteration over two lists. Jeremy Leipzig was kind enough to suggest the use of wildcards on Twitter.

Prior to trying this I had tried something I found from another source but it didn't work. It was a method that defined wildcards as input for each rule. Unfortunately, I cannot find the source at the moment so I'll try to take a better look in my bookmarks in a few minutes. However, I did find some code-snips to show that it looked something like this:

def get_raw_fastq(wildcards):
    ''' return a dict with the path to the raw fastq files'''
    r1 = samples_information[samples_information["sample"] == wildcards.sample]['r1']
    r2 = samples_information[samples_information["sample"] == wildcards.sample]['r2']
    return {'r1': r1, 'r2': r2}

def get_fastq(wildcards):
    ''' return the pair of compressed fastq files for a sample.

        This helper function returns the appropriate paths according to the
        config file.
    '''
    d = {}
        d['r1'] = list(samples_information[samples_information["sample"] == wildcards.sample]['r1'])
        d['r2'] = list(samples_information[samples_information["sample"] == wildcards.sample]['r2'])
    return d

I had a bit of trouble trying to grok how to change this code to make it work for my situation probably from my lack of familiarity with Python. I would appreciate if someone could help me solve this, thank you.