This isn't directly supported by plink 1.9. However, if M is large enough that it isn't reasonable to just perform the entire computation and then filter for the lines of interest, the following hack will help:

1. Create a file (I'll call this id_order.txt) which has the M sample IDs of interest on the bottom, with the other (N-M) on top.
2. Use "plink --bfile ... --indiv-sort f id_order.txt --make-bed reordered" to create a new fileset with the desired sample order.
3. Run "plink --bfile reordered --genome --parallel k k --out ...", where k is the largest integer which isn't greater than N/(2M).
4. The resulting .genome.[k] file will still have a few extra lines, so you may want to use e.g. a Python script to filter them out.