Space psychology was a branch of psychology that studied the patterns which developed in the close quarters of a starship during long voyages in deep space.
Starfleet took the subject of space psychology quite seriously. For instance, the determination of the state of mind of a captain was important in court martial proceedings, and could be facilitated by consultation with an expert in space psychology, such as DoctorLeonard McCoy. McCoy, for his part, didn't consider himself an expert, when asked by Areel Shaw. (TOS: "Court Martial")
In 2369, several members of the USS Enterprise-D crew attended a three day conference on the psychological effects of long-term deep space assignments. (TNG: "Timescape")
Although the connection has not been established, this discipline was likely connected with exopsychology.