I'm not familiar with that tool specifically, but the reason you get different numbers each time is likely because the software is modeling some stochastic fluctuations and it isnt an entirely deterministic process. If the software has the option to provide a starting random seed, try making this the same each time you run it.
Ligand docking is a fairly 'wooly' process though, so that all sounds pretty normal to me (in my albeit very limited experience).
To answer your terminology questions:
If this is what I think it is (based on how ITASSER uses this terminology), this is likely how frequently occupied a givel voxel is by the model. I.e. if you run the same model 100 times, and the ligand/protein tend to occupy the same 3D space each time, that's more likely to be the real shape/conformation.
RMSD is a way of measuring the Euclidean (straight line) distance between two points, usually atoms (often alpha-carbons) in this context, in 3D space.
Its simply the x, y and z coordinates squared (to remove any sign), added together, and then all square rooted to return the magnitude to the right level. Its defined as:
This can be done for a single atom, or across all elements, (which is where the Sigma comes in).
Max RMSD and average RMSD will be exactly what they sound like.