The King's Speech star was almost an X-Foe