Space madness was a term for space-related insanity. There was more than one type. It was sometimes caused by contamination or exposure to various elements.
In 2266, Spock posited that the insanity that had apparently led to the deaths of the Psi 2000 personnel could be a type of space madness heretofore unknown to them. (TOS: "The Naked Time")