>But doesn't a defeated stag lose his horns? I seem to recall a broken antler is a sign of defeat.
They can, and if their antlers break, I'm sure that does count as a loss, but I don't know that a loss necessarily accompanies loss of antlers for that season.
Ultimately, you probably are not going to find a concrete, logical explanation for it. There was some reasoning at some point, but it's essentially an old-timey meme that caught on, and from there it takes a life of its own.