Good question! I'll assume you are referring to the magnetic poles of the earth, which are in different locations from the geographic poles. The geographic poles are defined to be the places where the earth's rotation axis crosses the surface of the earth, and the magnetic poles are the places where the magnetic field points straight into or out of the earth's surface. The magnetic field everywhere is what we call a "vector" quantity. It has a strength and a direction -- and this direction may point anywhere in three-dimensional space, including up and down, not just north and south. Just about anywhere on earth (except near the equator) the field will have some vertical part to it -- it will be at an angle to the ground.
If you are holding the compass horizontally at one of the magnetic poles, so that the magnetic field is perpendicular to the plane of the compass, then there will be no preferred direction for the needle to point. It may point in the last direction it was pointing as you walked up to the pole. It may turn until friction slows it down if the needle was spinning (you can get it spinning by using another magnet or by turning the compass and letting friction spin the needle up). If you are carrying anything magnetic at the same time, then the compass needle's direction may point according to this other object's magnetic field.
If you hold the compass sideways, one end of the compass's needle will point down into the earth, and the other will point up. If you hold the compass at some angle between horizontal and sideways (which is most ways of holding the compass, and it may be hard to get it exactly horizontal), then the needle will point along the direction which is lowest. Some compasses (the cheap ones) are built such that the needle is supported on a sharp pin; the compass needle has a hemispherical cup in the middle and the pin rests in the cup. The compass needle then may wobble around and dip up and down even if the compass itself is held flat. It may then dip and point down until the needle scrapes on the compass face or the needle rubs on the edge of the cup. More expensive water-filled compasses with special bearings won't wobble like this and will behave as described above.
Here's a fun experiment to try. See if you can find, with a compass, the angle the magnetic field makes with respect to the ground in your location. To do this, you would orient the compass in a variety of different angles and find the one where the needle does not prefer any direction -- it will stay anywhere you put it. It might be best to use one of the fancy water-filled compasses for this, otherwise the pin and cup may rub and scrape in some orientations more than others and spoil the result. Then the plane of the compass will be perpendicular to the field. It will be hard to get it exactly right because if the angles are close but not quite right, then the magnetic forces on the compass needle will also be very small and it may take a long time for the needle to swing over to point in the right direction, and gravity may spoil it if the two ends of the needle weight different amounts.
p.s. Just in case you meant what would happen if you stand at the actual pole (not the magnetic one) the behavior is of course different. At the north pole, for example, if you hold the compass horizontally the needle which is supposed to point north will point south, toward the north magnetic pole. That's not saying much, since at the north pole ANY horizontal direction is south. The north magnetic pole is somewhere around the islands in far northwestern Canada, and that's where the needle will point.
(republished on 07/22/06)