Clark was born and raised in Herminie, Pennsylvania, a coal mining town east of Pittsburgh. His parents were originally from Stone Mountain, Georgia. His miner father, Emery Clark, died of a lung disease two weeks after Sonny was born. Sonny was the youngest of eight children. At age 12, he moved to Pittsburgh.
When visiting an aunt in California at age 20, Clark decided to stay and began working with saxophonist Wardell Gray. Clark went to San Franciscowith Oscar Pettiford and after a couple months, was working with clarinetist Buddy DeFranco in 1953. Clark toured the United States and Europe with DeFranco until January 1956, when he joined The Lighthouse All-Stars, led by bassist Howard Rumsey.
Wishing to return to the east coast, Clark served as accompanist for singer Dinah Washington in February 1957 in order to relocate to New York City. In New York, Clark was often requested as a sideman by many musicians, partly because of his rhythmic comping. He frequently recorded for Blue Note Records, playing as a sideman with many hard bop players, including Kenny Burrell, Donald Byrd, Paul Chambers, John Coltrane, Dexter Gordon, Art Farmer, Curtis Fuller, Grant Green, Philly Joe Jones, Clifford Jordan, Jackie McLean, Hank Mobley, Art Taylor, and Wilbur Ware. He also recorded sessions with Charles Mingus, Sonny Rollins, Billie Holiday, Stanley Turrentine, and Lee Morgan.