When Reflector Mage enters the battlefield, return target creature an opponent controls to its owner's hand. That creature's owner can't cast spells with the same name as that creature until your next turn.
Reflector Mage's ability doesn't stop any player from playing lands (in the case that the returned creature was also a land).
In several cases, the name of the creature that was returned won't match the name of that card once it's back in its owner's hand. For example, if the card was copying another creature, it will probably have a different name in its owner's hand and can be cast again before your next turn. The same is true if a double-faced card with its back face up is returned to its owner's hand this way.
If the returned creature had no name while it was on the battlefield (likely because the creature was face down), its owner can still cast face-down creature spells with morph or megamorph, or cast the same card face up, before your next turn.