I guess it's because I don't find Tom Felton at all attractive and have no wish to see him in leather pants.

Well, that's a fairly insulting depiction of fellow fans, imho.
Everyone likes different characters for different reasons, and if you don't warm to some, cool, but perhaps if you're that 'baffled' by fandom interest in any, you might want to look into reading some essays/meta on the subject(s), rather than writing off others as 'insane' or shallow.
It's the equivalent of saying: 'Ah, you're a Snape fan. You must be in love with Alan Rickman and think that he's going to make out with Harry in the next book.'