If you still have the option to optimize your experimental set-up, I would definitely encourage you to do that.

There are ways to account for the batch effects, even in unbalanced samples, but it'd be a pity if you'd have to waste your 10 replicates on accounting for effects that you may not be interested in -- assuming you don't care for the changes introduced by the difference in behavioral state. If you're mostly interested in the effect of your treatment, it'd be best to figure out which of the two behavioral states is the one you're actually interested in and focus on that one, especially since you already mentioned that the treatment effects are probably going to be more subtle than the changes induced by the behavioral switch.

Best of luck!

Dear Friederike,

Thanks for your helpful input on this. I am still very much in doubt about what is the best way of moving forward. I think that even though the direct effect of the behavioural state is not what I am interested in, I would be interested in knowing the extent to which the treatment effects are general across the two states, or whether there would be an interaction between treatment and behavioural group. So I guess in that case the best way to go would be to have the samples balanced across behaviours and treatments, right? On the other hand it would be great if one would be able to calculate how much I would lose in terms of statistical power to detect DEGs based on the treatments if I include samples from both behavioural groups. I would then have a factor with two levels for behaviour with n=20 each and a factor with 4 levels for treatment with n=10 each, divided into 5 and 5 for each behaviour. Is there any way to estimate how strong the treatment effect would have to be if the behaviour would affect about 40% of the genes, to detect a reasonable amount of the genes affected by the treatment?

Thanks again.

Best, Andrea