Nothing, Forever, an AI-powered Seinfeld spoof show on Twitch, was quickly becoming the next big thing on the platform. During the always-on stream, a cast of Seinfeld-adjacent characters had befuddling conversations, made weird jokes, and moved through a world of crude, blocky graphics, all backed by a laugh track and directed by AI.

But then it was suspended for two weeks after the Jerry Seinfeld-like character made transphobic remarks. That suspension is set to lift on Monday, and while its creators at Mismatch Media have been working to make sure transphobic comments won’t happen again, they can’t guarantee it.

The transphobic remarks happened after Mismatch changed the AI models underpinning the stream. “We started having an outage using OpenAI’s GPT-3 Davinci model, which caused the show to exhibit errant behaviors,” according to an announcement in the Nothing, Forever Discord. “OpenAI has a less sophisticated model, Curie, that was the predecessor to Davinci. When Davinci started failing, we switched over to Curie to try to keep the show running without any downtime. The switch to Curie was what resulted in the inappropriate text being generated.”

In a later post, Mismatch added that “we mistakenly believed that we were leveraging OpenAI’s content moderation system for their text generation models” and that it would be working to implement OpenAI’s content moderation API before the show went live again.

Since then, Mismatch has been doing stability testing against that implementation and making sure there aren’t any false negatives, Mismatch co-founder Skyler Hartle said in an interview with The Verge. “So far, it looks really good,” he said. But then, he hedged, saying that “of course, with software, there’s always variability.” I asked Hartle how Mismatch makes sure its guardrails work. “I think that in the space of generative AI and generative media, there is an inherent uncertainty.”

He referenced many of the wild things people have already been able to get ChatGPT and the new Bing AI chatbot to say: “I think everybody in this space needs to be worried and thinking about this.” Mismatch Media is tackling this by making an AI “safety council slash team,” Hartle said, which is trying to figure out mitigation strategies so that AI safety measures can evolve alongside generative AI pieces. “We feel very strongly that it’s our duty as people in the generative space to do this as safely as possible.”

In addition to leveraging the official OpenAI content moderation API, Mismatch also wants to use OpenAI to assist in the moderation process. “We are working to create guardrails that actually leverage OpenAI to pass our content to them and ask a series of questions and prompts,” Hartle said. Mismatch is “figuring out the right ways to have OpenAI and these large language models help moderate this process. These models are the best thing at parsing natural language right now, so it makes a lot of sense to also try to use them as a secondary system.”

Hartle doesn’t expect the tone of Nothing, Forever to change with the additional content moderation systems. That likely means we’ll continue to see the show keep creating more bizarre and irreverent moments, but hopefully this time, there’s no transphobia.

Hartle also said that Mismatch wants to introduce an audience interaction system that it had previously built but decided not to launch with Nothing, Forever. The system “allows fans to safely interact with the show and potentially massage the direction that the show heads in while still retaining its generative spirit,” according to Hartle. Mismatch hopes to launch the system alongside the lift of the Twitch suspension, but he doesn’t “want to promise anything at this point.”

I’m personally skeptical of an audience interaction tool in an application like this. While it could be used in a grand moment of internet unity like the heyday of TwitchPlaysPokemon, I’m worried that it will turn into something like the Tay fiasco. 

Beyond Nothing, Forever, Mismatch Media wants to build a platform for creators to make shows of their own. “There’s a lot that goes into that, and a lot that we’re figuring out and iterating on, but the plan is to empower, like the next generation of people to do these kinds of things,” Hartle said. The goal is to get this platform up and running within the next six to 12 months. 

I also asked Hartle something I’ve been wondering since I first watched the show: is the plan actually to make it run forever? “Our hope is to run the show for as long as we can as long as it economically makes sense to do so because it is very expensive,” Hartle said. “But that said, we are building this out as a technology platform. And we want to make more of the shows. If we’re successful at that, and we’re able to build a business up around us, I see no reason why Nothing, Forever shouldn’t run as long as the fans and community want it to run.”