The issue with clustering and PCA plots is that by themselves they do not provide any way to distinguish between technical and biological variation between samples.
They can be very useful to provide hints and clues to distinguish between technical and biological variation, but they cannot distinguish between that on themselves.
Take that example you gave where the labels were possibly swapped. If there is no additional evidence from the data that they were swapped (such as specific genes being expressed in a condition where they have been knocked out), I think it would not fair to declare that they were indeed swapped based on the clustering.
The reasons are that
Based on the clustering alone, it's impossible to know whether there was really a sample swap or an unusual biological phenomenon has occured.
Ultimately, one can't distinguish between declaring that the samples were probably swapped because the data makes more sense when this is assumed, and declaring that the samplse were probably swapped because the data fits better the hypothesis of the researchers.
Well yes, generally speaking, removing an outlier should always be justified ; and cherry picking data that best fit an hypothesis should be absolutely avoided. That being said, I think there is room for discussion for what is a valid justification for removing outlier. Potential technical issues are not always easily identified. In many cases, one can have reasons to suspect that something went wrong, without knowing what exactly. Perhaps if you could provide a more specific example, we could try to provide more specific answers.
Thanks, I want to understand the issue conceptually first; then I will know how to apply it by myself to different cases. The point I was arguing for is that the only way how one can distinguish betwee a justified removal of an outlier, and cherry-picking of the data is if there is some technical problem with the outlier. How to identify the specific technical problem is another issue, though a challenge in itself of course.
Is that wrong? Is there another criteria to distinguish conceptually between justified removal and cherry-picking, that you can suggest?
I agree that the identification of a technical problem is the only way to know for sure that removing an outlier is fully justified. However, I think that suspicion of technical problem based on PCA for instance (to take Istvan example) can also provide sufficient justification for removing outlier. If 9 out of 10 replicates perfectly cluster together while the last one appears completely different, then it most likely reflects the fact that something went wrong with that replicate. After all, it is very common to have samples or whole experiments to fail ; ask a wet lab biologist if you don't believe me.
Some level of judgment is needed of course. To give you a counter-example, if in a PCA two out of three replicates cluster together, then I would not find it justified to remove the third one. The type of data can also affect your interpretation ; it is why I asked for a specific example. ChIP experiments for instance, are infamous for being technically hard to make it work as planned.