I concur with everything ATpoint has mentioned.
I was wondering if the in-group variance would be too big for this to work.
You will not know until you've looked at the data and tested it. That's exactly what the stats implemented in the usual differential expression tools are there to help you decide: whether the factor of interest (in your case: age) has a systematic effect on the expression pattern of a given gene. If the variability within your two groups is large, the p-values will indicate that, i.e. you will probably not find any genes with p/q-values that'll meet the usual criteria for statistical significance.