There's another reason why the popularity of "Take Five" is remarkable: It's performed in a musical structure that people in the Western world often show cognitive resistance to.
Most Western music is dependent on a structure with two, three, or four beats in a measure—or some multiple of those—with even spaces between the emphasized beats.
Time signatures like these are often known as "irregular," "complex," or "asymmetrical" time signatures.
Here's that very number, performed in 1966, seven years after it's initial release. Brubeck had a relatively early racially integrated combo as well. Right on, Dave: