Unlikely I guess, adapters/primers bind to the flow cell and reduce the usable amount of reads but that is pretty much it. It is somewhat beyond me though why you would sequence a probably very expensive experiment without checking libraries on a Bioanalyzer yourself and then repeat bead cleanup until the adapters are removed. Usually one extra round is sufficient. Anyway, it seems your library is the problem, not the sequencing. People often do a shallow sequencing of their single-cell libraries to infer number of cells (and by this approximate quality) before going deep, maybe this is something you should consider in the future.