I want to use a blast-like local aligner to find common substring between common human readible texts. e.g. social media data. A normal plagiarism finder won't do because the intention is to align GB and TB of data.
Hence, does anybody know NGS local aligner which work on unicode?
1 answer
I think your are looking at the wrong tools/domain. Text similarity search is a subdomain of its own - with countless tools and techniques that work far better than trying to align sentences. If you have not found a tool that works on terrabytes of data it just means you have not looked exhaustively enough.
You should be looking for a tool that was specifically designed to do this, blast will likely not produce results that are meaningful or easy to interpret. Sukhdeep's link already has some pointers.
Try Citeseer as a search engine to find what you are looking for: http://citeseerx.ist.psu.edu/
Log in to answer this question.
might be a help http://stackoverflow.com/questions/3406428/fast-sequence-alignment-on-unicode-strings