You know, I'm going to have to agree with you.
A minor gets too close to C major, which is too happy, if not happy, not sad. Also, the Dorian mode is sad, solemn, and in all natural notes, the dorian is very close to D harmonic minor.
But then, just last week I learnt that there are no absolute tones. Depending on which temperament you use (there are a lot) the pitch of a note changes. Because of this, the sound of a key will change in different temperaments too. In ages past, some keys were completely disregarded because they just sounded shit. By that I mean the intervals in the scales just didn't match up.
Today, I think it's 'equal' temperament which is being popularly used, since it makes all keys playable (if imperfect).
Perhaps the question should read: The saddest key of all keys using equal temperament?
