Carrying on an affair while in office - very poor judgment, but let he who is without sin cast the first stone.
Leaving for a roll in the hay with mistress over Father's Day - judgment now at slimeball level. Give me the stone, I'll throw it.
The governor goes off the radar and is uncontactable, leaving the state without its Executive and without turning power over to the Lt. Governor - its time to resign with whatever grace he can muster.
The governor gets caught and makes a tearful mea culpa - cry me a river. Actions have consequences, and while I may feel some remorse for the governor, saying "sorry" does not begin to fix the problem of a governor whose lack of judgment in his personal life held out the potential for serious damage to his state.
Making the trip on the public's dime - you're out of there, Bozo. And if the state wants to file criminal charges over misuse of funds, they are completely warranted.