you haven't understood" is literally correct but it is a little bit impolite and it is against what the talker implies. If you pay attention s/he tries to be polite. It is talking like s/he is asking for an excuse.. so that is why I used on purpose "You have misunderstood me"