Many players find the Phantoms very annoying. Personally, I don't mind them, but even I must admit that I think they come on a little quickly.
To fix this, I would change Phantoms from being hostile to semi-neutral. When neutral, all they do is fly around in large, vaguely menacing circles, centred on the player, screeching infrequently. They fly away and de-spawn shortly before the Sun rises. When hostile, Phantoms act just as they do now.
Phantoms still spawn from night 3 onwards, but are now only hostile to players in their 5th night onwards.
This achieves two things:
We now have a little longer before we actually need to sleep when we're avoiding Phantoms.
We now have a 2 night warning when we get close to the limit, instead of immediately having a horde of angry nightmare bats clawing at your back.