Before Negan was revealed, whenever they encountered the Saviors, they came across as arrogant Negan wannabes, demanding tribute right away.
But those always around him are cowering in his presence, like the fat guy who brought him the bat. Negan talks about breaking balls, like making fun of Dwight. Yet they don't project that same kind of arrogance in his presence, only to the outsiders.
So he rules mostly with fear? He doesn't have lieutenants who are almost as ruthless, to spread fear into all the followers?
If a couple of them takes him out, you don't get the sense that someone else will take over and then punish the perpetrators. They may try but what's to prevent the new guy from being taken out again?
How much of an empire can there be without palace guards and loyal generals who command big parts of the troops?
Maybe they haven't shown it yet but there doesn't seem to be much of an organizational structure to The Saviors, beyond Negan and everyone else, some of whom have more points than others, as doled out by Negan.
You would think he'd get some loyal lieutenants to always watch his back.