He became proficient on the instrument by the time he entered North Texas State University. Ellis majored in music, but because they did not yet have a guitar program at that time, he studied the string bass. Unfortunately, due to a lack of funds, his college days were short-lived. In 1941, Ellis dropped out of college and toured for six months with a band from the University of Kansas.

In 1943, he joined Glen Gray and the Casa Loma Orchestra and it was with Gray’s band that he got his first recognition in the jazz magazines. After Gray’s band, Ellis joined the Jimmy Dorsey band where he played some of his first recorded solos. Ellis remained with Dorsey through 1947, traveling and recording extensively, and playing in dance halls and movie palaces. Then came a turnabout that would change Ellis’s career forever.

As pianist Lou Carter told journalist Robert Dupuis in a 1996 interview, “The Dorsey band had a six-week hole in the schedule. The three of us had played together some with the big band. John Frigo, who had already left the band, knew the owner of the Peter Stuyvesant Hotel in Buffalo. We went in there and stayed for six months. And that’s how the group the Soft Winds were born”. Together with Frigo and Lou Carter, Ellis wrote the classic jazz standard “Detour Ahead”.