Doesn't it seem odd that Walter pretty much, and without much resistance, broke Steven's control over Gilead?

So first we learn he
Spoiler:
had it away with Gabrielle, he was Steven's court advisor/sorcerer or whatever, then that Steven had to send Roland away because Walter, somehow, could have killed him in his own home
, in what is essentially, an empires fortress, did he have so much inside treachery and control over Steven's people that Steven couldn't do nothing about protecting Roland in Gilead ?

I don't understand this, there's no hint at Steven being under some kind of mind control, being the
Spoiler:
Dinh of Gilead
, therefor having a vast amount of power, why couldn't he have banished Walter into exile or just have him killed?
Spoiler:
Steven was aware that Walter was working for Farson and has known for 2 years about the affair with his wife
, so what, he just lets it all happen?

Walter can't possibly be THAT powerful that he couldn't have been dealt with by a group of Gunslingers, has anyone have ideas about this?