I don't know that women look down on card games any more than men do. TCGs/CCGs just are not a mainstream thing amongst adults.
I find it more likely that women are deterred not by how society will look down on them, but how other people in the same hobby will. All varieties of gaming...