You're right that this would be a serious problem if it were what cosmologists are saying, but it isn't.
If I understand correctly, cosmologists do not say that the
universe was ever a single point. Nor do they say that it is currently
infinite. It is very close to flat, which does mean that it must extend
mathematically well beyond the observable universe. However, current
theory is consistent either with an infinite (open) or a finite
(closed) universe. That property would not change, so only if it
happens to be closed would there even be a possibility of it having
ever been a point. It is generally assumed that when quantum mechanics
is somehow integrated with General Relativity, the extrapolation back
to a point will break down even for a closed universe. The object that
was about grapefruit size after inflation is the currently visible
universe, not the whole possibly infinite mathematical extension of it.
It does seem, at least within the context of current theory, that the
universe will expand forever, but that turns out to be a separate
question.
Mike W.
(published on 10/22/2007)