In one of my gene sets, I am interested in selecting gene subsets such that no two genes occur in the same set more than "t" times. Mathematically, given n objects and selecting k objects without replacement, how many combinations exist such that no two objects are in the same group more than "t" times?

E.g, If we have five genes A, B, C, D, E and pick 5 choose 3 = 10 combinations with t =2: But gene combination A B is present three times and hence out of three possibilities

A B C

A B D

A B E

I will select only 2 (=t) combinations randomly.

Is there a closed form solution for finding out how many combinations exist and listing them out for say n = 40, k=5, and t=3?

Thanks, but I would also appreciate some explanation of your solution. How did you arrive at this?

Sorry. I thought a bit more and realized that it is not that simple. Do you really need a closed form?

Maybe it is easier to simulate and count.

My previous formula is correct only if you have a fixed pair.