Managing Projects & FastQ files (Getting Organized)

Question

[jnowacki]

Tools for managing projects & FastQ files?

I'm about to be flooded with data and I need a system to organize everything. What tools are out there that will allow you to:

1) Organize fastq files by project
2) Allow you to search for files / projects that were used several years ago
3) Annotate projects so you know what is what?

Basically if my boss asks me "Where is our data & results from project XXXXXXX from April 2014?" I want to be able to quickly and easily dig it up.

[Pierre Lindenbaum]

see How Do You Manage Your Files & Directories For Your Projects ? (OMG that was 7 years ago !)

[GenoMax]

If you need formal software solution: https://omictools.com/lims-category At least some are bound to be free.

Answer 1

[Pierre Lindenbaum]

A simple solution (I should do this...)

1) Organize fastq files by project

group all your FASTQ into a UNSORTED bam : https://blastedbio.blogspot.fr/2011/10/fastq-must-die-long-live-sambam.html

the BAM will contain the sample name.

3) Annotate projects so you know what is what?

put some metadata into your BAM header: (like a code for your project)

2) Allow you to search for files

find the bam files extract the header and grep for the project. Something like:

find . -name "*.bam" | while read B ; do samtools view -H ${B} | grep PROJECT_NAME && echo ${B} ; done