Is there any similarity between “having an actual affair” and having sex with a prostitute while you're married? I think most people would answer yes. Then consider: Is there any similarity between having sex with a prostitute while you're married and paying to watch a prostitute perform sexual acts for your voyeuristic gratification? Again, I think a lot of people would say yes: There's a distinction, obviously, but I don't think all that many spouses would be inclined to forgive their husbands (or wives) if they explained that they only liked to watch the prostitute they'd hired. And hard-core porn, in turn, is nothing more than an indirect way of paying someone to fulfill the same sort of voyeuristic fantasies: It's prostitution in all but name, filtered through middlemen, magazine editors, and high-speed internet connections. Is it as grave a betrayal as cheating on your spouse with a co-worker? Not at all. But is it on a moral continuum with adultery? I don't think it's insane to say yes.
That's Ross Douthat. I think this only makes sense from the perspective of a once almost universal, now simply common, and in any case very silly assumption that sex with a spouse is the one permissible form of sex. So, any deviation from that one case of good sex is some grade of bad sex. This reminds me of Jimmy Carter's admission that he had “committed adultery in his heart many times.” Simply thinking about touching yourself while thinking of someone not your wife is somewhere on the Douthat-Carter moral continuum. However, for most people whose minds have not been addled by religious dogma, the distinction between touching yourself and touching someone not your spouse or committed monogamous partner is well nigh categorical. One's just wrong, one's just not.
Also, pause to consider that similarity is not a transitive relationship. I am similar to a kitten in that we are both mammals. Kittens are similar to cotton balls in that they are both fluffy. I am not therefore similar to, or on some continuum with, a cotton ball.