Only the players who haven't slept should be able to see, hear and be attacked by the phantoms. Other players should just see you getting attacked by nothing.
In my opinion, phantoms should be a hallucination. Sleep-deprived players would be the only ones seeing and being hurt by the monsters rather than a player who just slept. Well-rested players will see the sleep deprived players being attacked by an invisible force.
I posted this on Reddit a few weeks ago here: