Grimes was born in Newport News, Virginia, United States, and began his musical career playing drums and one-fingered piano. In 1938 he took up the electric four-string tenor guitar. In 1940 he joined the Cats and the Fiddle as guitarist and singer. In 1943 he joined the Art Tatum Trio as guitarist and made a number of recordings with Tatum.

After leaving Tatum, Grimes recorded with his own groups in New York and with a long list of leading musicians, including vocalist Billie Holiday. He made four recordings with his own group, augmented with Charlie Parker: “Tiny’s Tempo”, “Red Cross”, “Romance Without Finance”, and “I’ll Always Love You Just the Same”, the latter two featuring Grimes’ singing.
In the late 1940s, he had a hit on a jazzed-up version of “Loch Lomond”, with the band billed as Tiny “Mac” Grimes and the Rocking Highlanders and appearing in kilts. This group included tenor sax man Red Prysock and singer Screaming Jay Hawkins. Grimes continued to lead his own groups into the later 1970s and he recorded on Prestige Records in a series of strong blues-based performances with Coleman Hawkins, Illinois Jacquet, Pepper Adams, Roy Eldridge and other noted players including, in 1977, Earl Hines.

With Paul Williams, he co-headlined the first Moondog Coronation Ball, promoted by Alan Freed in Cleveland, Ohio, on March 21, 1952, often claimed as the first rock and roll concert. In 1953 he may have played on the Crows one-hit wonder, “Gee“that has been called the first original rock and roll record by an R&B group.