With three kids and a new dog at home,Scott Foley is barely getting any sleep at night.

But the Scandal star says he wouldn’t want it any other way.

“Fatherhood is insane,” Foley, 42, told PEOPLE at the 46th NAACP Awards on Friday in Pasadena, California. “I’m lucky enough to have three healthy children. And I’m lucky enough to be woken up every day between 3 a.m. and 5 a.m.”

Surprising, the culprit isn’t 3-month-old son Konrad: It’s his big brother, Keller.

“My 2½-year-old, if he doesn’t ‘drop’ anything at 3 a.m., it’s a good night,” Foley said, laughing.

So, how does the actor make up for those sleepless nights? Quick naps onthe Scandal set — and white lies to his wife, Marika Dominczyk, of course.

“I nap at lunch. But if I take up any longer than 10 minutes, I’m a wreck for the day,” he said. “And I make up call times. If I have to be [on set] at 9 a.m., I’ll say, ‘Sorry, babe, I got to be there at 7.’ Then, I sleep in my trailer for a couple hours.”

Foley, who plays Captain Jake Ballard on the show, says the key to his relationship with his wife (they also have a daughter, Malina, 5) is simplicity.

“We keep it together because we’re pretty simple,” he said. “This [Hollywood glamour] is all fake. We make dinner at home. We sit with the kids. We do crossword puzzles at night between watching The Bachelor and Scandal.”

– Matthew Cole Weiss