“I left Columbia in the mid-'60s. I had a guaranteed contract for, like, $100,000 a year. And I just let it go. And I wasn't a rich man. There were a lot of bad vibes around the whole thing.”