A colony was the term for a group of ants that lived together, whether natural or in captivity.
In 2269, Spock surmised that the Lactrans were as advanced a civilization compared to Humans as Humans were to a colony of ants. (TAS: "The Eye of the Beholder")
In 2376, Mezoti's science project for the Annual Voyager Science Fair was a Teirenian ant colony. She had infused the soil with a blue ion dye to make the ants more visible. (VOY: "Child's Play")