Exactly -- this is my concern. One of the negative controls I'm using is IgG and another is a protein that should not bind a positive control transcript.
In both cases, the negatives pull down very little RNA (undetectable even by Qubit RNA High-sensitivity assay).
Even though the qRTPCR looks good, I'm worried the library will have a high representation of "sticky" RNAs. That because there is so little RNA, there will be spurious amplification and perhaps make it seem that RNAs are highly abundant when it is just an artifact of there being so little RNA to start with.