"My dad, he was really charismatic and funny, but he was also, like, kind of sinister," the actor told PEOPLE ...