Duplication is expected in ChIP-Seq, but 96% duplication is not unless 1) you depth of coverage is massively excessive, or 2) your binding factor interacts with very few sites. A much more common explanation is that your IP failed and/or the amount of IPed chromatin was too low for efficient library construction, which results in a huge amount of PCR duplication. You can discriminate via genome browser of your non-deduplicated data. Bona fide peaks will have multiple overlapping reads with offsets, while samples with only PCR duplicates will stack up perfectly without offsets.

You would expect to find duplicated sequences in Chip-Seq data, because you are only sequencing the parts of the genome pulled down by the IP procedure. Your data is probably fine, so don't remove the duplicates.