How to restrict access of toolbar buttons?

I have some private windows where observers can’t touch, but they can still click the toolbar buttons if they are visible to them.

How can I restrict that behavior?