The invariant points can be any points that you choose. In protein superposition you read in the two structures and extract xyz coordinates (typically those of the alpha-carbon atoms). Then, you use some iterative procedure along these lines:

1. Select points from each structure that you don't want to move (termed fixed or invariant)
2. Superpose those points
3. Allow other points to move around (within limits imposed by bonds and steric clashes)
4. Calculate some "goodness of fit" score (typically RMSD)
5. Repeat until score is minimised

Try a web search using the terms in your question, such as protein structure superposition invariant point. You will find lots of useful information on the first page of hits.