Followers are automatically designated to the same server, so even if you change servers through the Global Settings, it will go back to the server designated for followers.