I tend to agree with you that when there is very little adapter contamination, trimming them is not a critical step. However, the question is what's the cost of doing that anyway, and not worry about it anymore? The cost is usually disk space and compute time. If you are short on these resources, or have lots and lots of data to process, then maybe skipping the adapter trimming step is not a bad idea. If it won't be too much of a task, then just get it over with. Also keep in mind that adapter trimming is not the only recommended preprocessing step. So if for example you run a tool like Trimmomatic to trim low quality bases/reads, then why not also tell it to trim adapters?