Red line was a limit marker for a machine. If a machine's operation passed this point, there was no guarantee of safety.
In the mid-2270s, the transporter aboard the USS Enterprise exceeded this line when an attempted beaming into the ship was made from Earth. The resulting failure of the system, caused by an inactive transporter sensor, resulted in the death of two individuals. (Star Trek: The Motion Picture)
In Federation starships, if a ship exceeded the red line, which was not recommended in any circumstance, there was a possibility that the ship's space frame might become overly stressed and fail. In 2364, when pursued by Q, Lieutenant Worf informed Captain Jean-Luc Picard that the engines were operating at warp 9.3, the red line for Galaxy-class starships. The crew was able to reach warp 9.6, at which point the captain ordered a reduction in velocity. (TNG: "Encounter at Farpoint")