Villagers need to have a shorter range to hire themselves from the block

I have a Minecraft world and I live in a big castle and the basement has a trading post but some villagers seem to hire themselves from a grindstone on the other side of the castle, and then it seems that sometimes when I have a villager I haven't traded with and destroy it's workplace it still remains its job for a long time or sometimes permanently.