Its not the pitchers you have to worry about at the lower levels as much as it is the catchers.
All it takes is one stray pitch/foul ball and you have a broken toe and you are out of commission for several weeks. That results in lost games--> lost game fees -> hampers your ability to spring for the plate shoes you really want.
Just like any "workout," the more you work games in the plate shoes, the easier it will become. If you go from a 9U (sneakers) to a 13/14U (plate shoes) the plate shoes will feel that much heavier.
My $0.02? Don't stop wearing them. Your muscles will adapt.