How high a ball bounces depends on two things -- 1) how fast the ball hits the ground or whatever surface it is bouncing from, and 2) how much energy is lost in the collision between the ball and the surface. Provided that you have chosen the surface and the impact speed carefully so that the glass ball does not shatter, the glass ball ought to bounce quite well.
Ordinary glass is very much stiffer than ordinary rubber, in that it takes quite a lot more force to bend it out of shape by the same amount as one would bend a rubber object of the same shape. The real question here is not the stiffness, but rather it is how much of the energy is gotten back after the force is gone and the glass and rubber are allowed to return to their original shapes. Some glasses have very low losses when deformed. You can test this by tapping the glass ball with a metal object (such as a spoon) and listening to how long it stays ringing. If you get a dull "thud" which dies away right away, then, the glass is "lossy". If it rings a long time, then the vibrating glass does not dissipate energy very much when strained in alternating directions.
Some different kinds of rubber have more energy loss than others. Whatever it is they make squash balls out of bounces very poorly. You can also make a hollow rubber ball bounce very badly by letting all the air out (a flat basketball will bounce badly if at all), or worse yet, by cutting big irregular holes in it. Then, when the ball hits the ground, the different parts of the ball will bend and collapse and absorb the mechanical energy and heat up. Rubber bells do not ring very long (as described above), and so glass balls will probably bounce better than rubber ones if they are given the chance, that is, if they do not break.
More on the subject of breaking: For bounces of noticeable and measurable heights, the surface used for bouncing should not be very hard -- it should bend a little bit when something hits it. If it is hard, then the glass ball will shatter. I lived in an apartment with a ceramic tile floor in the kitchen, and every time I dropped anything made of glass, it the glass shattered. Now my kitchen has a vinyl covering on the floor, and glass jars bounce when I drop them.
Energy can be lost in the floor in this case, and for a surface which will not break the glass, this is a big concern. One could make the surface out of rubber, but then you might expect just about any object to bounce as high as the rubber ball, because it is the rubber surface that is deforming in the collision and not the ball. You could put a spring on the surface, and that would be good to give you almost all of your energy back, and it would also matter very little what the ball was made out of in this case, provided that the spring compresses very much more than the ball (we're also neglecting air resistance -- the flat basketball may have a problem with air resistance too).
For bounces of very small heights, the glass probably will not break. You can test this by dropping the glass ball from a height of about a millimeter or two above a ceramic tile floor, and it should make plenty of noise to tell you when it has stopped bouncing. This may be longer than the time you may observe with a rubber ball dropped from the same very small height.
Warning: if you do this experiment, even from a millimeter or two, you may make very small chips of glass which are hard to clean up, and they may cut you. It is very important to wear eye protection as well -- you don't want one of these chips getting in your eyes. Please be careful with bouncing glass on any surface.
(published on 10/22/2007)