According to the Minecraft Wiki, pillagers can sometimes drop swords and armor during raids. My suggestion is for them to use these. When they are within ~6-7 blocks of the player, they would switch to their sword. Some pillagers would spawn with armor and some of their equipment would be enchanted. This would make combat with these mobs more fair, yet you could still just block with your shield and never take a point of damage (which would make them "very weak in one way and strong in another," which I'm assuming is what you want).
Please sign in to leave a comment.