We find out in the comics that a direct attack on the Tower is possible, that was what the Great Old Ones pulled But there is probably different ways of doing it. One of which was owning the land...