Coles spent his early career playing with R&B groups, including those of Eddie Vinson (1948–1951), Bull Moose Jackson (1952), and Earl Bostic (1955–1956). He was with James Moody from 1956 to 1958, and played with Gil Evans‘s orchestra between 1958 and 1964, including on the Miles Davis album Sketches of Spain. After this he spent time with Charles Mingus in his sextet which also included Eric Dolphy, Clifford Jordan, Jaki Byard, and Dannie Richmond. Following this he played with Herbie Hancock (1968–1969), Ray Charles (1969–1971), Duke Ellington (1971–1974), Art Blakey (1976), Dameronia, Mingus Dynasty, and the Count Basie Orchestra under the direction of Thad Jones (1985–1986).
Coles, nicknamed “Little Johnny C”, recorded as a leader several times over the course of his career. He died of cancer in 1997.