The fallacy lies in the wording.
I can only let what is in my power.
Other people are generally not in my power.
What I really mean by letting someone do something is refrain from complaining after they've done it.
Certain joint courses of action can be negotiated, and Tradition IV can occasionally be invoked (informally, of course) to politely bring to someone else's attention some major effect on others of some trivial action of theirs, and, naturally, reasonable people will accommodate this.
But in almost every situation where I'm using language like "letting him get away with it" I'm really dealing with a runaway train.
It does me no good to either lie down on the track in front of one or attempt to board one.
No more train-top shoot-outs for me.