I mainly kept them because of other biostars questions that seemed to suggest that you should keep them if you can, though given the added complexity it might just be better to discard them like you said. The biostars questions that I referenced are below:
Should I use unpaired reads from trimmomatic When is recommended to remove or keep Unpaired "Orphan" reads in downstream analysis?
Thank you for your suggestion