The chain of command was a ranking system used by militaries and other organized groups. It represented the line of authority down which orders are passed from one officer to another, and also the line down which passed command of a particular vessel or installation.
Aboard a starship, or most starbases the chain of command began with the assigned commanding officer, usually an officer of the rank of captain, or in some cases commander. If the commanding officer, for any reason, was unavailable to command his or her post then those responsibilities passed down the "chain" to the first officer, then the second officer, etc.
Between the launch of Enterprise NX-01 and the late 23rd century, the position of first officer, or executive officer (abbreviated as XO) was filled by the highest ranking officer assigned to the ship aside from the captain. The position was not a separate assignment as it would later become. The designated first officer would perform their normal duties until it became necessary for them to take command.
This had changed by the late 24th century, when the XO's position had become a completely separate position in a starship's crew held by an officer of commander or lieutenant commander rank grade who held no other position aboard the vessel.
After the executive officer, command responsibilities passed down the chain from officer to officer as necessary in accordance with each officer's rank. And it was common practice for a commanding officer to designate a second officer to take command if both the captain and first officer were unavailable or unable to perform their duties.
This aspect of the command chain functioned similarly aboard a starbase, or space station, as well as in a planetary installation.
More broadly, the chain of command could refer to Starfleet, in general, with the chain beginning with the admiralty. And the Federation Council, and/or the President of the Federation could be considered to be at the ultimate top of the chain.
When Captain Jonathan Archer reverse-imprinted on a group of Xindi-Insectoids, his increasingly unusual actions led some of his fellow officers to openly question or defy his orders. He relieved those officers and stated Major J. Hayes that he needed officers who could respect the chain of command and follow orders. (ENT: "Hatchery")
Mutiny is an inherent violation of the chain of command, something that was not lost on Captain Philippa Georgiou when she ordered Commander Michael Burnham arrested for incapacitating her and attempting to order a preemptive strike on the Sarcophagus. (DIS: "Battle at the Binary Stars")
When Spock inquired to Dr. McCoy if he had noticed Captain Kirk acting strangely lately, McCoy replied, "the man on top walks a lonely street. The chain of command is often a noose." In response to McCoy's words, Spock told him to "spare me your philosophical metaphors." Spock then pointed out Kirk's transfer of Kevin Riley to engineering, and McCoy explained that "he's the Captain, he can transfer whoever he pleases," adding "you can look that up in a hundred volumes of space regulations somewhere." (TOS: "The Conscience of the King")
Maintaining discipline within this structure was important. In 2286, the Human President of the United Federation of Planets felt sure that Admiral James T. Kirk, who faced a sole remaining charge of "disobeying orders of a superior officer", knew how necessary maintaining discipline was. Kirk concurred, leading the president to demote him to his preferred rank of captain. (Star Trek IV: The Voyage Home)
During Ensign Mendon's 2365 stint aboard the USS Enterprise-D as part of the Officer Exchange Program, he became aware of several procedural items that could make the operation of the starship's bridge more efficient, and requested to share them with Captain Jean-Luc Picard directly. Picard informed him that the chain of command was employed on the Enterprise, and therefore the Benzite should report to Lieutenant Worf. When the ensign apologized, Picard waved his actions off, saying Starfleet should have made that clearer to him when he was assigned to participate in the program. (TNG: "A Matter Of Honor")
Benjamin Sisko learned many things from Captain Leyton aboard the USS Okinawa, including that "a good officer has to respect the chain of command." Leyton believed that without it "Starfleet would cease to function and we wouldn't stand a chance against our enemies." (DS9: "Paradise Lost")