I fully agree on the safety issue. The way I handle this is... if I have a pitcher in a legal windup or set position, and the batter is not yet focused on the pitcher, I'm holding up a hand to F1 directing him not to pitch. If he disregards this and delivers anyway, I'm coming out from behind the catcher, away from the batter and loudly calling and signalling "No pitch!" I am then expressing to F1 my deepest displeasure at not heeding my time signal to allow the batter to get set in the box.
My point is that I'll allow the battery to get/give their signs all they want, batter in or out of the box, but I will not let F1 deliver a pitch until the batter is ready.