Sorry, late to the party!!
What you did right? You gave him a chance.
What you did wrong? I would have done something like this (which I've done before) STOP SIGN - "No, no no, Jim, ... WE ARE NOT COUNTING today, I'm not having it, STOP! If he continues, BOOM. I've found that they totally change their tone with the above. (mind you, ...you didn't necessarily do anything wrong, but the "what did you say" could have made things worse,... IMHO)