Unfortunately, there is no literature from Intel that says the board is mSATA compatible though it does clearly have a slot for a miniPCIe card. Assuming it is compatible, the controller is compliant with the latest SATA standards so you should be fine with whatever you throw in there if it does in fact support mSATA.
Block diagram on page 16 http://downloadmirror.intel.com/20950/eng/DH77DF_TechProdSpec01.pdf
mSATA connector connected (1) SATA 3.0Gb/s
eSATA connector connected (1) SATA 3.0Gb/s
(2) SATA 3.0Gb/s connectors
(2) SATA 6.0Gb/s connectors
That fills out the 4 SATA II and 2 SATA III available on the H77 chip. There doesn't appear to be any shared connectors.
I have this board, in one of my htpcs paired with an i3 2120t. love the board, hate the pcie/msata port. I could not get anything pcie to recognize (2 wireless cards and a tv tuner tested) in it. The port in the bios is listed as msata and shows in with the rest of the sata ports. Hopefully your results with msata are better then mine.