If a fan becomes so obnoxious as to where I can no longer concentrate, I feel it is necessary to stop the game and resolve the issue.
I umpired a small tournament championship game and a fan was shouting at me the entire game. When his son came to the plate and he started to shout curse words. I immeadiately confronted the fan and told him I would gladly give him my umpires gear and he could umpire the rest of the game for me. Like magic he changed from a rude fan to complimenting me on what a wonderful job I was doing.
I suspect alcohol was also involved.
Normally if there are rude fans, my league president is around to take care of the issue.
Lesson: Unless fans are threatening you, shouting rude or obscene things, or alcohol is involved, try to ignore them.