Self-taught on drums, he began performing in Los Angeles in the mid-1940s. His drumming is recorded in The Hunt, the recording of a famous Los Angeles jam session featuring the dueling tenors of Dexter Gordon and Wardell Gray on July 6, 1947. He recorded with Lester Young‘s quintet from 1949 to 1955 and with Stan Getz, Coleman Hawkins, Charlie Parker, and Miles Davis.
Kay joined the Modern Jazz Quartet in 1955, replacing original drummer Kenny Clarke. He remained through the group’s dissolution in 1974 and occasional reunions into the 1990s. In addition to his MJQ compatriots, he had an enduring partnership with cool jazz altoist Paul Desmond through the first half of the 1960s. He played drums on several of Irish singer-songwriter Van Morrison‘s albums: Astral Weeks, one song on Saint Dominic’s Preview, and four songs on Tupelo Honey.
Kay was known for incorporating percussion instruments alongside his drum kit, such as timpani, small cymbals, triangle, bell tree, and darbukas, the latter referred to as “exotic-looking” drums in a 2006 article.