Episode 78: “What’d I Say” by Ray Charles

A History of Rock Music in 500 Songs
A History of Rock Music in 500 Songs
Episode 78: "What'd I Say" by Ray Charles

Ray Charles

Episode seventy-eight of A History of Rock Music in Five Hundred Songs looks at “What’d I Say” by Ray Charles, and at Charles’ career in jazz, soul, and country. Click the full post to read liner notes, links to more information, and a transcript of the episode.

Patreon backers also have a ten-minute bonus episode available, on “Sea of Love” by Phil Phillips.

Tilt Araiza has assisted invaluably by doing a first-pass edit, and will hopefully be doing so from now on. Check out Tilt’s irregular podcasts at http://www.podnose.com/jaffa-cakes-for-proust and http://sitcomclub.com/


I refer to the original lineup of the Raelettes as a trio, and as a result miss out Darlene McRae, the fourth member of the group.


There’s no Mixcloud this week, as twelve of the fourteen tracks here are by Ray Charles, and Mixcloud has limits on how many songs by one artist one can include.

I’ve used two sources for the information here —  Charles’ autobiography, Brother Ray, which gives a very clear view of his character, possibly not always in the ways he intended, and Ray Charles: The Birth of Soul by Mike Evans.

There are three collections of Ray Charles’ work that everyone should own, and which cover the music in this podcast. The Complete Swing Time and Atlantic Recordings is a seven-CD set which contains everything up to 1959, The Complete ABC Years 1959-1961 is a three-CD set covering the next phase of his career, and Modern Sounds in Country & Western Vols 1& 2 is a single CD with those two albums on.


This podcast is brought to you by the generosity of my backers on Patreon. Why not join them?


When we last left Ray Charles, he had just had a run of hits at Atlantic records, including several of the songs that became the foundation of soul music. But as I mentioned at the end of that episode, after that run of hits, he hit a dry spell, and for a few years he was releasing records like “Swanee River Rock”, which were hardly up to the standards of his best work. After his first single of 1957, “Ain’t That Love”, which made the top ten on the R&B chart, most of his singles didn’t chart at all for the next two years, with some bobbling around at the bottom of the R&B top twenty.

He was having a tough time in his life, too. He was addicted to heroin, he had a small child, and he was playing night after night in third-class venues. At one point, several members of the band, including Charles himself, had been arrested for heroin use, and Charles had had to pay a bribe of six thousand dollars to get the charges dropped — he’d been let out of jail before the rest of the band, and had to record his hit “Hallelujah, I Love Her So” with session musicians rather than his regular band:

[Excerpt: Ray Charles, “Hallelujah, I Love Her So”]

Most of the places he was playing were bad in other ways. Many were filthy — he sometimes had to rent hotel rooms to get changed in because the dressing rooms were unusably dirty — and in those days before portable keyboard instruments became commonplace, he had to make do with whatever pianos were at the venues. He would talk later about how some were so badly out of tune that he’d have to play in C sharp while the rest of the band were in C, just so he could be something like in the same key as them.

This did improve his musicianship, though — he had to learn to play in keys that most musicians would normally avoid, and he became a much more fluent pianist. But he ended up taking an electric piano with him on the road, so he could be sure it would always be in tune. Other musicians would make fun of him for this, as the electric piano was regarded at the time as a novelty instrument, not something a serious musician would use, but Charles knew it had possibilities.

So, by the late 1950s, Charles seemed to be trying to go more in the direction of becoming a jazz musician, rather than an R&B one, in an attempt to play more upmarket gigs. He kept releasing R&B singles, but he was increasingly moving in a jazz direction both in his albums and in his live performances. In 1957 he played Carnegie Hall for the first time, on a bill which also included Billie Holiday, Zoot Sims, Dizzy Gillespie, Mose Allison, and Chet Baker, along with a performance by Thelonius Monk and John Coltrane which has itself become legendary:

[Excerpt: Thelonius Monk and John Coltrane, “Blue Monk”]

He was encouraged in his turn towards jazz by the Ertegun brothers, who had founded Atlantic as a jazz label, and who were still very much jazz lovers first and foremost, even as their label was increasingly an R&B one — and they were encouraged by people like Miles Davis, who kept telling them that Charles was something special, and should be allowed to become one of the jazz greats.

At the time there was a keen interest among many jazz musicians in making a new form of jazz that was more influenced by the other musics that black people played — the blues and gospel, in particular. People like Art Blakey and Horace Silver were trying to incorporate these musics into their own, partly because they loved them, and partly because they felt that black people had invented jazz, but it was becoming an increasingly white music. By incorporating a gospel or blues feel into their music, they could create something based on their own heritage, something which it would be impossible for white people who hadn’t grown up in those traditions to copy successfully.

Many of these musicians had started using the terms “soul” and “funk” about their music — and it’s particularly notable that someone like James Brown, for example, did not come from a blues background, but from a jazz one. Brown always talked about his influences being people like Lionel Hampton and Miles Davis, and you can definitely tell if you study Brown’s records that he was passionate about fifties jazz.

Ray Charles in the late fifties was arguably the person who was mixing these styles most fluently, doing things like recording an album with Milt Jackson, the vibraphonist with the Modern Jazz Quartet:

[Excerpt: Ray Charles and Milt Jackson, “How Long How Long Blues”]

But of course, even in the jazz shows, he was still playing his R&B hits — he was a musician who was blurring boundaries, not one who was moving all the way from one genre to another.

On several of his records in the mid-fifties, he had been backed by a group of backing vocalists known as the Cookies, who were the go-to backing vocalists for Atlantic Records, and had also sung with people like Chuck Willis and Big Joe Turner. Charles invited two of the girls to become two-thirds of a new vocal trio, the Raelettes, and back him on the road — the Cookies continued with a new lineup and were to have a few hits of their own in the early sixties.

The Raelettes would go through several lineup changes, largely because they fell in and out of favour with Charles over their personal relationships — there was a rather unpleasant, but not totally unfounded, joke that went around that if you wanted to be a Raelette you had to let Ray — but the original lineup was Margie Hendrix, Pat Lyles, and Gwen Berry, and Charles would always say that Margie Hendrix, the trio’s leader, was at least the equal of Aretha Franklin or Etta James:

[Excerpt: Ray Charles/Margie Hendrix “The Right Time”]

But the Raelettes caused a lot of controversy. A lot of people had already been upset by the way that Ray Charles appropriated gospel music and turned it into pop songs — they felt that that style of music was sacred, and he was defiling it, though his own argument was that he never sang about God, and if he had a bit of gospel feeling in his voice, that was just how he sang. And after all, while he had been one of the first to do this, he wasn’t the only one — there was Clyde McPhatter, and Sam Cooke, and James Brown, all doing the same kind of thing.

But adding the Raelettes made his performances seem, to many critics, like he was copying the call-and-response vocals in Pentecostal services. And certainly a record like “Yes Indeed” seems deliberately to be invoking the church at points:

[Excerpt: Ray Charles, “Yes Indeed”]

But the record that would see him turn away from jazz, and start his period of greatest commercial success, was one which took that church music and turned it to extremely earthy concerns. And it came about at what he intended to be one of his last low-class dance gigs playing for the R&B audience.

Charles’ normal method of working was to record songs before going out on the road with them, and to have tight arrangements written for the recordings by people like Jesse Stone or Quincy Jones. He always knew what he was going to do on stage, and didn’t like to mess around with new things in performance. But one night he had mistimed the length of the show he was playing, and found himself on stage at one o’clock in the morning, with another half hour to go of the night’s show, and no more songs rehearsed.

So he said to the group, “Listen, I’m going to fool around and y’all just follow me.”

He started playing some riffs on the electric piano, following a standard twelve-bar blues structure, and improvised a few lyrics, mostly about dancing. Just as he had on his first Atlantic hit, “Mess Around”, he took inspiration from an old boogie woogie classic, this time Pinetop Smith’s “Pinetop’s Boogie Woogie”:

[Excerpt: Pinetop Smith, “Pinetop’s Boogie Woogie”]

After a few minutes of this, he told the Raelettes to just copy him, and started a call-and-response section, where he’d make moaning noises and then they’d repeat them. The song went on for several minutes, and he thought of it as just a relatively successful way to fill out an underrunning show — so he was quite surprised when audience members came up to him afterwards and asked him where they could get a copy of the record.

So over the next few nights, he played the riffs he’d improvised, and did the call and response section, and slowly built it up into a properly structured song. And every time, the audience went wild.

He called Jerry Wexler at Atlantic and told him he was coming to him with a new song to record, “and it’s pretty nice”. Normally he didn’t like to build up the songs he was going to record in advance — he thought that it was better for him to let the music speak for itself. But this time he thought it was worth giving it a bit of a build up, so — “It’s pretty nice”.

“What’d I Say” was a far more technically innovative record than it’s normally given credit for, and one that was largely built in the studio. As well as being Charles’ first record to really show off his electric piano playing, it was also one of his first to be recorded on an eight-track machine.

Tom Dowd, Atlantic’s engineer, is someone who is acknowledged by almost everyone he worked with as one of the great recording engineers of all time. He’d actually started as a physicist, working on a cyclotron at Columbia University, which he attended from the age of sixteen, and he had been working towards his degree when he was called up for World War II, and put to work on the Manhattan Project. When the war was over, he’d planned to continue in physics, but his war work was top secret and so not useful for getting a qualification, and after working on the development of the atom bomb there was nothing he could learn in an undergraduate physics degree, so he’d switched careers and become a recording engineer instead.

In the early years at Atlantic he’d worked miracles with terrible equipment and recording spaces — in those early years, Atlantic’s recording studio had also been its offices, with the desks and chairs cleared out of the way when it came time to record. He’d constantly pushed Atlantic forward, insisting on recording on tape when everyone else was recording on acetates, and recording in stereo when people were still only buying records in mono. And he’d recently got hold of an eight-track recorder — the second one in existence, after the one that Les Paul had built for himself.

The song that Ray Charles brought into the studio, “What’d I Say”, was clearly something with commercial potential. But it was a song designed to stretch out for a long time, to fill up time in a show. There was no way it could be a single at the length that Charles played it. It was also so obscene at points that it was very unlikely to get played on the radio.

But Dowd put together several different edits of the track, recorded on his new eight-track machine in what was for the time astounding sound quality. He took the seven-and-a-half-minute track that they recorded, cut it down to five minutes and seven seconds, and then split it into two so it could go over both sides of a single. The result is one of the great masterpieces of dynamics in pop music.

It starts with Charles playing, solo, just with his left hand, a simple twelve-bar riff on the electric piano:

[Excerpt: Ray Charles, “What’d I Say”]

Then his right hand comes in, while the drummer plays a Latin rhythm, mostly on the hi-hat:

[Excerpt: Ray Charles, “What’d I Say”]

Then, for the bulk of what became part one of the song, Charles sings disconnected verses over this backing:

[Excerpt: Ray Charles, “What’d I Say”]

Then at the end of the first part, the horns come in, answering Charles as he sings the song’s title:

[Excerpt: Ray Charles, “What’d I Say”]

The song comes to a sudden stop, and the band pretend to complain about the unexpected ending of the song:

[Excerpt: Ray Charles, “What’d I Say”]

And then, for part two, we get a call-and-response, with the Raelettes answering him along with the horns on the choruses:

[Excerpt: Ray Charles, “What’d I Say”]

But also the bit of the track that caused the most controversy — several breakdowns with Charles and the Raelettes sighing and moaning in an almost pornographic way:

[Excerpt: Ray Charles, “What’d I Say”]

As Charles himself said in his autobiography, “I’m not one to interpret my own songs, but if you can’t figure out “What I Say,” then something’s wrong. Either that, or you’re not accustomed to the sweet sounds of love.”

Many radio stations banned the song — although Charles noted that when later white artists, like Bobby Darin, Jerry Lee Lewis, and Elvis Presley recorded cover versions of it, those versions weren’t banned, and said “That seemed strange to me, as though white sex was cleaner than black sex”.

None of that stopped it becoming a hit, though. It went to number one on the R&B charts, and number six on the Hot One Hundred, becoming his biggest hit up to that point. But even more than its chart success, it was a record that had influence in all sorts of places. Many people consider it the first soul record, though we’ve already looked at several songs in this series which I would consider soul. But it was certainly one of the ones that defined the genre. And it was the record that single-handedly turned the electric piano from a joke instrument into one that was as respectable as any other.

Atlantic followed up the single with an album he’d recorded earlier, a big band record made with various members of the Count Basie and Duke Ellington bands — great players like Zoot Sims, Paul Gonsalves, Clark Terry, and Fathead Newman, playing arrangements written by Quincy Jones. The lead-off track of that album, a cover version of Louis Jordan’s “Let The Good Times Roll”, was only a minor hit, but it’s now one of the records most identified with Charles:

[Excerpt: Ray Charles, “Let The Good Times Roll”]

That album introduced another aspect of Ray Charles’ legend — the record was called “The Genius of Ray Charles”. Charles himself always disliked the term “genius” being applied to him, but Jerry Wexler thought it appropriate, and it stuck — over the next few years there would be albums like The Genius Sings the Blues, The Genius After Hours, The Genius Hits The Road, and Genius + Soul = Jazz.

As it turned out, it was also the last album Charles recorded for Atlantic — though he’d recorded enough of a backlog that they could release four more albums over the next couple of years. His contract with them was up for renewal in October 1959, and while it seemed at first as if that was a pure formality, he ended up going to another label.

ABC Paramount wanted to expand into the R&B market, and came to him with an offer that no other artist had ever had from a label. He’d get complete artistic control over his recordings, he’d get seventy-five percent of all profits made once the label had recouped their costs, he’d get a guarantee of fifty thousand dollars per year against his royalties — and remember that three years earlier, when RCA had paid thirty-five thousand dollars for Elvis’ contract, that had been the most any label had ever paid. And, best of all, after five years the ownership of the masters would revert to him — he’d own his own work, rather than the label owning it.

Charles took the offer to Atlantic and gave them the chance to match it, because he did like recording for them, but they said there was simply no way that they could come close to it, so he moved to ABC Paramount. And a strange thing happened — for a few years he flourished artistically as he never had before, but as a performer, and not as a songwriter. In fact from that point on he almost completely stopped writing songs, and concentrated on other people’s material.

The first album he did for ABC Paramount, The Genius Hits the Road, is a patchy affair, a collection of old songs about places in America, like “Mississippi Mud”, “Alabamy Bound”, and “New York’s My Home”, but it had one standout hit, a version of “Georgia on My Mind” that sixty years on is still considered the definitive version of that song:

[Excerpt: Ray Charles, “Georgia on My Mind”]

That went to number one on the charts, and earned him four Grammy awards.

It looked like the move to ABC was a successful one, but the next album, Dedicated to You, was a bit of a misfire. It was a similar themed collection, this time of songs based on women’s names, but it didn’t have anything like the highs of “Georgia On My Mind”. Luckily, the album after that, Genius + Soul = Jazz, was a return to form — another album of funky jazz with the Basie band and Quincy Jones, which gave him another top ten hit with the instrumental “One Mint Julep”:

[Excerpt: Ray Charles, “One Mint Julep”]

To follow that album, he put together a big band of his own for the first time — a seventeen-piece group that could play the Quincy Jones charts the way they sounded on the records. His career seemed to be going from strength to strength. He recorded “Hit the Road Jack”, which became his second number one on the pop chart:

[Excerpt: Ray Charles, “Hit the Road Jack”]

An album of duets with Betty Carter, which included their version of “Baby It’s Cold Outside”, now generally considered the definitive version of the song:

[Excerpt: Ray Charles and Betty Carter, “Baby It’s Cold Outside”]

And then he made another move which seemed bizarre to everyone, recording an album of country songs, Modern Sounds In Country & Western Music. The idea of Ray Charles doing an album of songs by people like Don Gibson and Hank Williams made no sense to anyone except Ray Charles, but he had the right to make whatever records he felt like.

And as it turned out, that became his greatest album, and possibly the peak of his career. While the songs were all country songs, Charles did them all in his own style, with either orchestral or big band backing, and with no concession either in his vocals or the instrumental backing to country genre elements. The lead single from it, Don Gibson’s “I Can’t Stop Loving You”, went to number one on the Hot One Hundred, the R&B charts, and the Adult Contemporary chart, but my favourite track from the album is the second single, Eddy Arnold’s “You Don’t Know Me”, which may be Charles’ greatest performance ever, and went to number two on the charts:

[Excerpt: Ray Charles, “You Don’t Know Me”]

That album confirmed that Ray Charles could make any kind of music he wanted — rock and roll, soul, big band, jazz, or country — and have it sound like no-one else. It solidified him as the most important musician of his generation, a link between all the disparate threads of American music. He had nothing left to prove — and so from that point on his artistic development stalled.

His next albums were Modern Sounds in Country and Western Volume 2 — a good album, but not as essential as the first; Ingredients In A Recipe For Soul, an album mostly consisting of old standards, and Sweet and Sour Tears, a collection of songs themed around crying. All are thoroughly enjoyable albums, but none of them reach the peaks of his very best work. And after one further album, Have a Smile On Me — a weak album of alleged comedy songs — he was arrested again for his heroin use, and spent a year on probation, unable to work.

He got clean, and never used heroin again, but on his return to music, while he made many fine records, a spark was lost. Now, I want to be very clear here that I am *not* saying that stopping using heroin made him a less interesting musician or any of that nonsense. I have no interest in romanticising addiction. It’s just that in the first thirteen years of his career, he was constantly finding new things he could do, pushing his music in different directions, and discovering what “Ray Charles music” really was. For the last forty years, he was working within the boundaries he had set in those initial years. But then, it can be argued that in that time, entire genres of music were also contained entirely within the boundaries he had set.

He kept working almost up until his death. His final album, Genius Loves Company, was recorded when he knew he had terminal liver cancer. It’s an album of duets with singers who had been influenced by him, like Van Morrison and Elton John, plus his contemporary Willie Nelson:

[Excerpt: Ray Charles and Willie Nelson, “It Was A Very Good Year”]

The album was released two months after his death in 2004, and became his first number one album since Modern Sounds in Country and Western Music, more than forty years earlier. It went triple platinum, and earned nine Grammy awards — as much as a recognition of the esteem in which Ray Charles was held as of the quality of the album itself. It’s safe to say that at the time he died there wasn’t a musician alive in the fields of rock, R&B, soul, and country music whose career hadn’t in some way been influenced by his, and as long as recorded music exists, people will still listen to Ray Charles.

4 thoughts on “Episode 78: “What’d I Say” by Ray Charles

    1. Andrew Hickey

      No, I’m saying “What’d I Say?”, which is the song’s title as it appears on all the records and sheet music and so on. Some sources when they’re transcribing Charles’ speech, notably the ghosted autobiography David Ritz did with him, use “What I Say” instead, but the official title of the song has always been “What’d”. Which is what I say.

  1. Shan Jayaweera

    In the Peter Jackson observational documentary Get Back there’s moment where George Harrison talks about being wowed by seeing Ray Charles and his band which included Billy Preston who they knew from their Little Richard encounters. Ray Charles is one of the great voices so glad you’ve got him in this story

Leave a Reply