You could table rule this out, but given that your Empire is effectively a nexus city state it could be wiped out fairly quick if the party managed to gain control of said tower or built one of their own. Heck an Illusionist with Mislead and a little luck could probably wipe out the city by impersonating the Emperor.