A protectorate was a planet or region which existed under the protection (or control, if the term was used euphemistically) of another more powerful group.
The Evora homeworld became a protectorate of the United Federation of Planets in 2375. (Star Trek: Insurrection)
Raatooras, an Arin'Sen colony, was a protectorate of the Klingon Empire before the Klingons abandoned the planet. (ENT: "Judgment")
On Kelemane's planet, the Central Protectorate was located on a subcontinent in the southern hemisphere. (VOY: "Blink of an Eye")
Haven, seemingly not a Federation member world, had a treaty with the Federation which required the Federation to protect it, and thus may also have had protectorate status.