This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Merging many files with having same columns with bash script

Hi Please I would like to merge a big number of .txt files in one. They all have the same columns , how to merge and keep only the header from the first ? It's a big number of files so I think I will need to use xargs ?! These are examples of what I have in my files : 

CH        SNP   A1  A2  N   P
20        SNP2  A   G   2   3.70E-03
19        SNP3  C   T   2   6.47E-03




CH       SNP    A1  A2  N   P
10       SNP1   G   C   2   3.11E-05

In total I have about 100000 files to merge with one header. What I ant to have is : 

    CH        SNP   A1  A2  N   P
    20        SNP2  A   G   2   3.70E-03
    19        SNP3  C   T   2   6.47E-03
    10      SNP1    G   C   2   3.11E-05

Thanks !

bash
• 2,075 views • link
updated by Pierre Lindenbaum Pierre Lindenbaum • written by mel22 mel22

Not really a bioinfo question I would say.

Can you add why it might relate to bioinformatics anyway?

And how would you like to see those files merged? It's always helpful to add a small example of what you try to achieve

• 0 views • link
written by lieven.sterck lieven.sterck

Ok I updated my post . Thanks

• 0 views • link
written by mel22 mel22

1 answer 

rm -f out.txt
find dir1/dir2 -type f -name "*blabla.tsv" | while read F
do
  if [ -f out.txt ] ; then
      tail +2 $F >> out.txt 
  else
     cp $F out.txt
  fi
done
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Thank you very much

• 0 views • link
written by mel22 mel22

I tried this solution but I still have a header line between merged files. My files are .txt space separated

• 0 views • link
written by mel22 mel22

ah, looking at your example ONE file contains the header line more that one time ? isn't it ?

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

They are all outputs from PLINK, so they have all same structure

• 0 views • link
written by mel22 mel22

I mean they are exactly same and no one has the header line more that one time.

• 0 views • link
written by mel22 mel22

I just tested it:

$  seq 1 10 | while read F ; do echo -e "CHR\n$F" > $F.tmp.txt; done 
$  rm -f out.txt && find . -type f -name "*.tmp.txt" | while read F; do   if [ -f out.txt ] ; then       tail +2 $F >> out.txt ;   else      cp $F out.txt;   fi; done
$ cat out.txt 
CHR
10
4
5
9
3
1
2
6
8
7
• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

Log in to answer this question.

More posts like this