Kody Brown Says ‘Sister Wives’ Is ‘an Experience of Criticism and Backbiting’ (Exclusive)

Kody Brown is getting honest about how his experience on Sister Wives has changed over the years.

As the long-running TLC series heads into its 20th season, the family patriarch, 56, is branching out into another realm of reality television with Special Forces: World’s Toughest Test. He tells PEOPLE that the two shows “are very different,” and that is something he is grateful for.

“I really appreciated the difficulty of Special Forces because it was very life-affirming,” he explains. “You accomplish a task, very difficult, with other people, and there’s a camaraderie you develop for it. Sadly, Sister Wives has really become an experience of criticism and backbiting. The experience with Special Forces was life-affirming in the sense that I got so much out of it, almost like a pivot away from my life, my real life.”

The popular competition series follows a group of celebrities as they fight to survive military-style challenges in a test of both mental and physical strength. Kody says his biggest takeaway from his time on Special Forces was a piece of advice from veteran Rudy Reyes.

“He said, ‘What are you doing that you should not be doing, and what should you be doing that you are not doing?’” Kody shares. “That’s a lesson I’ve started applying in my life. I could have used this 10 years ago.”

Sister Wives has documented the ups and downs of the Brown family — which includes 18 children in total — since 2010. In recent years, viewers have watched as Kody’s plural marriage with Meri, Janelle, Christine and Robyn Brown fell apart.

After more than 25 years of marriage, Kody and Christine were the first to announce their split in 2021. The following year, he and Janelle confirmed they were separated, and days later, PEOPLE confirmed that his 32-year marriage with first wife Meri was also over. He and Robyn are still together, now in a monogamous relationship.

“Really just the only part of our lives that are connected are through the show,” he admits of his dynamic with his three former wives.

When he was first asked to participate in Special Forces, Kody reveals Robyn wasn’t entirely on board. However, she quickly realized that he was going to take the opportunity whether she approved or not.

“When the show was introduced to us, she was like, ‘I don’t think I want you to do that and we’ve got to have a discussion,’” he recalls. “There ended up not being a discussion, it just became an assumption from there, then on. I think when she saw — when I showed her or told her what it would be similar to or what I thought it would be like — she was like, ‘Oh, this is one of those things that you are going to do and I’m not going to be able to stop you.’”

Robyn joined the family and “spiritually married” Kody in 2010, while Kody was legally married to Meri and “spiritually married” to Janelle and Christine. In 2014, Kody and Meri divorced and he and Robyn legally married, although he remained romantically involved with the four for several more years.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Special Forces: World’s Toughest Test airs Thursdays at 9 p.m. ET/PT on Fox, and Sister Wives season 20 premieres Sunday, Sept. 28 at 10 p.m. ET on TLC.