I would start by checking how your library and the RepeatMasker library differ. In the util directory of RepeatMasker there is a script called queryRepeatDatabase.pl and this will tell you all the repeats in the database for a given species. You can obviously skip that step if you know your library is different. When you run RepeatMasker, just use the -lib option to specify your own file. I would not try to concatenate the files because they are different formats. If you want to use the full RepeatMasker library with your repeats, you could convert the RepeatMasker library to fasta, then concatenate the files and run the program with the -lib option. There is a script called buildRMLibFromEMBL.pl in the util directory of RepeatMasker for converting their version of the EMBL format to fasta.