Lionel Richie Worried His Grandma Would ‘Actually Disown’ Him Over 1 Song

Lionel Richie opened up about the one song of his that his grandmother did not approve of.

The 76-year-old four-time Grammy winner grew up in a musical family with his grandma Adelaide Mary Brown being a classically trained pianist.

However, during an interview with Robin Roberts that aired on the Thursday, Sept. 25 episode of Good Morning America, Richie revealed that his grandmother did take issue with some of his musical releases.

When asked to name his favorite song that he released as a member of the legendary funk band The Commodores, Richie easily decided on their 1977 hit “Brick House” as the answer.

“‘Brick House’ will always be… I mean forget about it,” he said.

However, the “Dancing on the Ceiling” hitmaker told Roberts that Brown was not a fan of the song, which compared a woman’s figure to a “brick house.”

“This was the only song where I thought my grandmother was actually going to disown me,” he admitted. “I tried to explain this to my grandmother: ‘This is a group song. I didn’t write all the lyrics,’ you know?”

In fact, Richie said that he actually was in agreement with Brown at one point, explaining, “I thought it was kind of disgraceful myself, until it became a hit.”

This is not the first time that Richie expressed concerns about “Brick House.”

In 2016, Richie told GQ that the song — which is remains popular at events like weddings — was one that he never expected to work.

At the time, he also reiterated that his grandma was not so proud of it at first.

“I don’t know what was wrong with me, but all I kept thinking about… I think it was just the fear factor of ‘I have to play this song for my grandmother,'” he admitted.

Continuing, he shared, “The song came out, and she didn’t go to church for about two or three weeks because it was just absolutely disgusting, you know what I’m saying? And then, of course, as the song became a hit, she kind of got over it because the check came in. Everybody’s happy.”

Elsewhere in his discussion with Roberts, the American Idol judge added that there was one thing he did not share in common with Brown. While his grandma could read sheet music, he still cannot.

“I don’t know [why],” he said. “I’ve asked God that several times, and it was said to me by Marvin Gaye, ‘There is an answer to this. Don’t go looking for it. Just accept it.’ ”

Richie was a founding member of The Commodores in 1968, but he left the group in 1982.

“Thank God for The Commodores because I never would have discovered Lionel Richie,” he told Roberts. “I found something in my life that was never present until I joined the band.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

As for his departure to pursue a solo career and to write more, the music icon described the decision as “painful.”

“A lot of people thought, ‘This is a band. This is a group.’ This is the closest thing to brothers I’ve ever had in my life. And they pushed me. We went through life together, and all of a sudden there’s a moment in time when I was having such a moment of discovery that I could write that I didn’t realize, ‘ Okay; that’s a hit, and then that’s a hit.’ ”

Richie’s interview with Roberts was in support of his forthcoming memoir Truly, which arrives on Tuesday, Sept. 30.