This is a test version of Biostars. For the public version, visit https://www.biostars.org.
script to organize files

Hello,

I have thousands of files that I would like to organize into project directories automatically. I have a "file metadata table" as a tabulated text file with two columns: (1) file name, (2) project the file belongs to. How can I easily execute:

  • make a directory per project name
  • mv files to corresponding directory according to metadata

I believe this is something super easy to script for people that do this routinely. I've tried writing something in bash and perl but without success.

Following this operation my goal is to have another script navigate each directory and run a bioinformatic pipeline.

Thank you very much!

bash
• 1,518 views • link
updated by Ram Ram • written by Galileo • Galileo

1 answer

A bash example

while read file project
do
  # check if project dir exists, if not create it
  if [[ ! -d $project ]]
  then
    mkdir $project
  fi
  # move file to project dir
  mv $file $project
done < metadata.txt

Revise accordingly for your file and project paths

• 0 views • link
written by jv jv

Thanks jv, this seems to work but read added '$'\r' at the end of filenames of the second column and thus mv cannot find the file. Any ideas how to remove '$'\r' so the file name is recognized?

• 0 views • link
written by Galileo • Galileo

I found a solution: used dos2unix on the files that have '$'\r' at the end :)

• 0 views • link
written by Galileo • Galileo

Log in to answer this question.

More posts like this