Steve Guttenberg Jokes About Fickle Hollywood: 'You're a Racehorse and When You Stop Winning, They Send You to the Glue Factory' (Exclusive)

Actor Steve Guttenberg, 65, is no stranger to the highs and lows of having a Hollywood career.

"There was love if you succeeded, and no love if you didn't succeed," he tells PEOPLE, with a laugh. "My publicist at the time said, "You're a beautiful racehorse, and when you stop winning, they send you to the glue factory."

But he insists that none of it bothers him — and he was fully aware of what he was getting into when he signed up to be an actor. In fact, he says that when he was just starting out, he actually quit because he didn't like the competitive culture.

"I figured I had nothing to lose," he says of moving to Los Angeles from Long Island after high school graduation in 1976 and taking a shot at acting. "My mom and dad both gave me a devil-may-care attitude. If it didn't work, big deal, I'd come home. I had nothing to lose."

He landed a commercial within his first two weeks in Hollywood and then did two movies, but decided to move back East anyway, and go to school at SUNY Albany.

"I didn't like the culture," he says of the movie industry. "It got to me. Young actors are very competitive and I didn't make many friends, so I decided to go back to school."

Five months into his first semester, he got the audition for The Boys From Brazil, landed the part, and that was that. Soon he was a bonafide star, cast as the lead in the Police Academy franchise, Cocoon and Three Men and a Baby, and he had both money and fame.

"I was guarded about it," he says, of not letting it all go to his head. "I still am. It's a capricious business, and when you're hot you're hot, but you're not always hot ... It was the fashion industry — you go in and out of fashion. So I always kept my mind realistic about it."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

Guttenberg, who also just released a memoir Time to Thank, about caring for his ailing father, says as his career continued, there were certainly parts he wanted and didn't get.

"You have to pull yourself out of it and realize if they didn't choose you, it doesn't mean you're not great, just like if they do choose you, it doesn't mean that you're great," he says. "It's a tough life and it's a weird life for people. You have to be very, very careful. That's why people lose their minds in show business — it's easy to buy what they're selling."

He adds, "You have to be careful not to buy into the success. You have to remember the stars are in the heavens, and you're just a human being who got lucky."

Time to Thank is available now, wherever books are sold.

For more on Steve Guttenberg, pick up the latest issue of PEOPLE, on newsstands Friday, or subscribe here.