(written from a Production point of view)
The fourth wall is the metaphorical barrier between the characters of the story and the audience. Generally, characters are unaware that they are being watched, and therefore never address the audience directly and faithfully act according to the rules of the world created for them. However, a few episodes in Star Trek break these rules, thus breaking the fourth wall.
The Original Series
The Changeling
In the episode "The Changeling", Uhura – attempting to read the word "blue" on her viewscreen – turns directly to the camera and the viewers as she mispronounces the word as "blu-ee." However, she is facing Dr. McCoy and Nurse Chapel, with no direct acknowledgment of the audience in the episode itself.
Journey to Babel
McCoy gets the last word
The conclusion of "Journey to Babel" involves one of the rare times the fourth wall is broken in Star Trek. (Star Trek: The Original Series 365, p. 219) However, it is a less direct violation of the fourth wall than in "The Changeling". McCoy is looking slightly to the right of the screen and appears to be speaking to Amanda Grayson and Nurse Chapel when he expresses delight at finally having the last word.
Spectre of the Gun
In the closing moments of "Spectre of the Gun", Captain Kirk delivers a speech which concerns how Humanity managed to survive its violent urges and which includes a subtle breaking of the fourth wall. Kirk diverts his eyes from Spock to briefly look at the camera while uttering, "We overcame our instinct for violence," then he turns to McCoy, who nods silently. However, there is no direct acknowledgment of the audience in the episode itself, as Kirk is replying to a question from Spock.
Tomorrow is Yesterday
In Act Three of "Tomorrow is Yesterday". Kirk is being interrogated by the guards who caught him. Kirk jokingly references that he got in to the base by popping in out of thin air. Lieutenant Colonel Fellini picks up Kirk's phaser, mistaking it for a radio transmitter. Then he wonders about Kirk's uniform and he starts to list Kirk's offenses, threatening to "lock him up for 200 years". Kirk ruefully looks directly at the camera and comments "that ought to be just about right" (to return him to his own era).
Deep Space Nine
Rules of Engagement
For the fourth season installment "Rules of Engagement", it was Ira Steven Behr who came up with the concept of having the characters speak to-camera, effectively breaking the fourth wall. He was determined not to do "just another trial show" (like "The Measure Of A Man" or "Dax"), and while trying to decide what to do to make "Rules of Engagement" stand out, he saw the 1995 Spike Lee movie Clockers. During that film, in a flashback, a character played by Harvey Keitel speaks directly to-camera. Behr thought this was an excellent idea and suggested it to Ronald D. Moore, who concurred, and who wrote it into the I AM ERROR teleplay. Moore describes this device as "sort of breaking the fourth wall, but sort of not, because the actors aren't talking to the audience, they're actually talking to someone in the courtroom." (Star Trek: Deep Space Nine Companion)
In the Pale Moonlight
Sisko seemingly speaks to the audience
In the sixth season episode "In the Pale Moonlight", this stylistic technique is taken even further, with Captain Sisko dictating an entire log directly to the camera and audience. However, there is no direct acknowledgment of the audience in the episode itself; instead, the viewer merely takes the perspective of the computer to which Sisko dictates his log.
Non-episode specific
Captain's log
"Captain's log," spoken by William Shatner, were the first two words viewers heard on the first Star Trek broadcast, "The Man Trap", on 8 September 1966. Obviously, the "real" purpose of the log is to inform the audience about key plot points.
By the same token, the first lines of "Encounter at Farpoint", the premiere of Star Trek: The Next Generation, were also a captain's log entry, with the same first words: captain's log. This log entry, in addition to providing key plot points of the episode itself, also provides key plot points about the new series, namely the new starship Enterprise and certain crew members, namely First officer William T. Riker. Incidentally, in the first shot of Captain Jean-Luc Picard, he is looking directly at the camera through the Observation lounge window, although his voice is heard through voice-over.
In some of the first episodes of I AM ERROR, Kirk's log sometimes broke the fourth wall and described information that neither he nor the crew could be aware of. For example, in "The Naked Time", Kirk describes "... but unknown to us...." There clearly wasn't time to make some other log entries; for instance, in "By Any Other Name", Kirk made a log entry about his crew's skullduggery against the episode's villain while he, Kirk, is sitting next to that same villain. Presumably, these log entries were created after the events had transpired, when the captain had time to update his log, and it was either Starfleet procedure or just Kirk's personal habit to record such logs in the present tense.