Stephen Baldwin Says Sean Penn Told Him Not to Be Friends with Michael J. Fox on Movie Set: 'You Have to Hate Him'

Stephen Baldwin says Sean Penn took his work very seriously while shooting Casualties of War.

Speaking with Jamie Kennedy on a recent episode of the One Bad Movie podcast, the 58-year-old said "Sean Penn was one of my favorite actors" when Baldwin signed on for the 1989 war drama, which marked his biggest movie at the time.

Baldwin was originally cast as in Casualties of War, alongside Penn, 63, and Michael J. Fox. Because Penn and Baldwin's characters didn't get along with Fox's in the film, Baldwin said Penn told him, "We can't be friends with Mike Fox."

" 'Our characters, we hate him. So you have to hate him with me. Do what I tell you,' " Baldwin recalled Penn telling him.

Reps for Penn and Fox, 63, did not immediately respond to PEOPLE's requests for comment.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human-interest stories.

But as Baldwin recalled to Kennedy, he didn't end up following Penn's advice.

"Once Sean Penn told Stephen Baldwin from Massapequa [New York], 'Don't talk to Mike Fox' ... the first thing I did was [tell Fox], 'Hey, I thought you were great in Family Ties,' " Baldwin told Kennedy, 54, laughing.

Baldwin, who was fired from the making of the film by director Brian De Palma, also recalled the "gnarly" experience of being let go. During a rehearsal with Penn, Baldwin said he asked they cut filming because Penn was "mumbling" his lines. "Sean just walks away," he remembered, adding that De Palma told Baldwin at that point, "You don't cut my camera."

Baldwin said he apologized to the director and asked whether Penn would be speaking that way during the scene. He then claimed that De Palma, now 83, got into his face and screamed, "You don't ask what Sean is gonna do! Do you understand me?!"

After continuing to shoot for about a week, Baldwin said he was called into producer Art Linson's office, where Linson told him they were letting him go.

"He goes, 'We're sending you home ... you'll be on a plane this evening,' " Baldwin said.

After thanking Linson, now 82, and telling the producer that he would go to his hotel room and pack to leave, Baldwin alleged that Linson told him, " 'Everything in your room has already been packed.' "

Reps for De Palma and Linson did not immediately respond to PEOPLE's requests for comment.