Although the rules that the Romans played by are not recorded, there is certainly not much leeway for variation. They most probably played to a score of 21. The dimensions of the court could be expected to be about the same as modern courts. The ball was larger than the one used today, and likely did not bounce quite as well, but the court dimensions could not suffer much shrinkage as a result. But no markings indicating the exact dimensions have survived the ages.
The ball-courts (sphaerista) served multiple purposes, and handball was definitely one of them. It is less certain that they would have used the playing fields (palaestra) for handball. In the diagram at right, of the baths at Herculaneum, the area labeled palaestra served the purpose of a sphaerista; it has a hard court and is too small for games like harpastum. The palaestra usually had, or was defined by, surrounding walls, but the playing field itself was dirt, and unless hard-packed, would have made a poor surface for handball.
The playing of racketball is highly unlikely. Although there is some suggestion in the literature that the Romans had a racket for some kind of tennis game, no evidence for this has been found. They did, however, have a kind of hard glove for a version of this game. The larger, heavier, hand-made leather balls would not be likely to survive the punishment of a racketball style game.
Alexander Adam provides this translation
from (Lucan and Pison):
Those who played at ball were said ludere
raptim, vel pilam revocare cadentem, when they struck it rebounding from
And also this passage from Plautus: