Luke Grimes Says Not Having Kevin Costner on ‘Yellowstone’ Set ‘Felt Different’ (Exclusive)

For Luke Grimes, the seriousness of Kevin Costner’s departure from his starring role in Yellowstone was not lost while filming the series’ final episodes without him.

The actor and musician, 40, tells PEOPLE that Costner’s decision not to return after nearly eight years of working together instilled “a finality to it.”

“Him not coming back felt like, ‘Okay, we’re going to have to land this plane now for real,'” Grimes says. “I think the patriarch that left was always going to be part of the story. It was always there, it was going, what do these kids do? What does this family do when their rock is gone?”

Although he acknowledges the “drama” that preceded it Yellowstone‘s return to filming — including plotting conflicts between the Costners Horizon: An American Saga and Yellowstone it led to the Oscar winner bowing out of his role on the show in June — Grimes adding that he felt “it was always going to go that way.”

“Whatever the circumstances, (it) definitely felt like, ‘Okay, we’re coming in for a landing,'” he says of Costner’s absence. “It definitely felt different.”

Never miss a story – sign up PEOPLE’s free daily newsletter to keep you updated on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

“He’s a big presence, man, guy is an icon,” adds the actor. “So it felt a little different not having him around. But as always, I mean, we’re one big family now. It was just as special, and hopefully the fans will be able to connect with the story just as much. . . I think they will.”

Regarding family, Grimes points out that although “Kayce as a character feels a bit like an outsider in his own family … I think the thing that put him between a rock and a hard place was so deep down he felt the loyalty, even though he knew it might not be the best for him.”

He notes that the dynamic was “kind of fun to play with” over the years.

For more on Yellowstone, pick up the latest issue of PEOPLE.

With Costner’s departure, fans are wondering what’s next for America’s favorite Montana ranchers. Grimes suggests “there’s always the rumble.”

“I think when something is that successful, there’s a pressure to keep it going because they know they have an audience and that’s what they’re looking for,” he says. “But it all comes down to Taylor (Sheridan’s) vision. He has to see it to write it, and I don’t think he’s going to write anything he doesn’t want to. Basically, we’re just going to see if Taylor has an idea that he gets excited about, and that’s going to make all the difference here.”

Yellowstone returns on Sunday, November 10 at 8 PM ET/PT on the Paramount Network. All five seasons are streaming on Peacock and Paramount+.