A subculture was a subdivision of a culture where people practiced their own customs and traditions. Melders were considered an undesirable subculture by 22nd centuryVulcans. (ENT: "Stigma")
While not specifically referred to as such, other subcultures could be seen in Star Trek, most notably the Hippie and punk subcultures on Earth. Sevrin's movement, and Gaia Klingons might also be an example. An alien example would be the gender-preferring faction of J'naii.