Am I right to think my husband wouldn't have done this if it wasn't serious?

It's obviously not just sex, he values her as a person and wants her near and available to him. Just move on, what's the point if his heart doesn't lie solely with you. And even if it was 'just' sex, does that make it more okay?