He was born in Lackawanna, New York, into a family with a vocal group and radio program. Smith says that his mother was a major influence on him musically, as she introduced him to gospel, classical, and jazz music. He was part of several vocal ensembles in the 1950s, including the Teen Kings which included Grover Washington Jr., on sax and his brother Daryl on drums.
After recording several albums with Benson, Smith became a solo recording artist and has since recorded over 30 albums under his own name. Numerous prominent jazz artists have joined Smith on his albums and in his live performances, including Lee Morgan, David “Fathead” Newman, King Curtis, Terry Bradds, Blue Mitchell, Joey DeFrancesco and Joe Lovano.
In 1967, Smith met Lou Donaldson, who put him in contact with Blue Note Records. Donaldson asked the quartet to record an album for Blue Note, Alligator Bogaloo. Blue Note signed Smith for the next four albums, all in the soul jazz style, including Think (with Lee Morgan, David Newman, Melvin Sparks and Marion Booker) and Turning Point (with Lee Morgan, Bennie Maupin, Melvin Sparks and Idris Muhammad).