Mercury has little chance to have a moon mainly because of its small gravity and orbit. A key piece of the dynamics is the notion of "Hill sphere." Well inside the Hill sphere the orbits of moons are bound to the planet, while near and outside the Hill sphere any orbiting bodies are more closely bound to the Sun. The radius of the Hill sphere is proportional to a
, where a is the semi-major axis of the planet, Mp
its mass, and Ms
the mass of the sun. For Mercury, a
are small, so the Hill sphere is small.
Another piece of the puzzle is the stability of lunar orbits over the lifetime of the solar system. This is complicated. There can be subtle dynamical interactions between the lunar orbits and the orbits of other planets, as well as interactions between the lunar orbits and variations in the tug of the sun due to Mercury's eccentricity. Because of the comparatively small size of the Hill sphere, it is likely that there is a relatively small set of stable orbits (if any!) and that these just happen not to have been populated while the solar system was forming.
The scarcity of material around Mercury's orbit may be explained by solar winds and condensation radii for solids. During the formation of the solar system, only a few substances such as iron and nickel could remain in solid and form terrestrial planets (the inner four, including Mercury). A vast amount of substances such as hydrogen and methane were in gas and were often swept away by T Tauri winds (similar to but much stronger than present solar wind).
A lot of factors have contributed to the formation of the present solar system. Those mentioned above may have played major roles, but they certainly do not explain the whole story.
Thanks to Professor Gammie for helping with the answer!
(published on 01/19/2012)