Criminal Minds: Thomas Gibson discusses getting fired

Thomas Gibson is finally speaking out about the physical altercation that got him fired off Criminal Minds.

“I feel like it took years to make a good reputation and a minute to damage it,” the TV veteran, 54, says in the current issue of PEOPLE.

Gibson says he was filming the second episode of the upcoming twelfth season of his hit CBS drama on July 26 when tensions flared between him and writer-producer Virgil Williams.

“We were shooting a scene late one night when I went to Virgil and told him there was a line that I thought contradicted an earlier line,” says Gibson. “He said, ‘Sorry, it’s necessary, and I absolutely have to have it.'”

Frustrated, Gibson says he returned back to set and was sharing Williams’ response with some of his castmates when Williams entered the room.

“He came into that room and started coming towards me. As he brushed past me, my foot came up and tapped him on the leg,” Gibson recalls. “If I hadn’t moved, he would have run into me. We had some choice words, for which I apologized the next day, and that was it. It was over. We shot the scene, I went home – and I never got to go back.”

After an initial two-week suspension, Gibson was informed by production that he was “dismissed” from the show.

Gibson will appear in the first two episodes of season 12 but “creative details for how the character’s exit will be addressed in the show will be announced at a later date,” reads a statement from ABC Studios and CBS Television Studios that was released to PEOPLE Aug. 12.

“It seems like they’re trying to erase me from the show. That hurts. But I’m using this time to be with my kids, and I look forward to what’s next,” says the father of three. “Maybe I’ll do a comedy, or a play, or direct, or produce.”

“My pride and reputation are hurt, but in the end I know the good work is what people will remember,” he continues. “I just need more opportunities to do good work and be a good guy.”

Season 12 of Criminal Minds premieres Sept. 28 on CBS.