Umi Matsuzaki is a 16-year-old student at Isogo High School living in Coquelicot Manor, a boarding house overlooking the Port of Yokohama in Japan. Her mother Ryoko is a medical professor studying abroad in the United States. Umi runs the house and looks after her younger siblings Sora and Riku and her grandmother, Hana. College student Sachiko Hirokouji and doctor-in-training Miki Hokuto also live there. Each morning, Umi raises a set of signal flags with the message "I pray for safe voyages".
One day, a poem about the flags being raised is published in the school newspaper. Shun Kazama, the poem's author and a member of the journalism club, witnesses the flags from sea as he rides a tugboat to school. At school, Umi meets Shun when he participates in a daredevil stunt for the newspaper, leaving Umi with a negative first impression. Umi later accompanies Sora to obtain Shun's autograph at the Quartier Latin, an old and dilapidated building housing the high school's clubs. Umi learns that Shun publishes the school newspaper, along with Shirō Mizunuma, the school's student government president. She ends up helping on the newspaper. Later, Shun convinces the other students to renovate the building after a debate on the future of the Quartier Latin, which may be demolished. At Umi's suggestion, the female student body cooperates with the other students.
Back at Coquelicot Manor, Umi shows Shun a photograph of three young naval men. One of these men is her deceased father Yūichirō Sawamura, who was killed in the Korean War. Shun is stunned to see the photograph. At home that night, he opens a notebook to reveal he has a duplicate of Umi's photograph. He questions his father, who reveals that Yūichirō arrived at their house one evening with a young Shun, shortly after the end of World War II. The Kazamas had recently lost their infant, so they adopted Shun. Upon discovering that he is avoiding her, Umi eventually confronts Shun, who reveals they may be siblings. Having checked the city records, Shun discovers that both their names are in the Sawamura family registry. Umi is able to hide her feelings from Shun, and they continue as friends.
When the renovation of the Quartier Latin is complete, the Kanagawa Prefectural Board of Education decides to proceed with the building's demolition. The students nominate Shun, Umi, and Shirō to visit Tokyo and meet with Tokumaru, a businessman and the school board's chairman. The trio travels through the city, which is in preparation for the 1964 Summer Olympics, and successfully convince Tokumaru to visit the Quartier Latin. As Shirō leaves the group, Umi professes her love to Shun, and he reciprocates her feelings in spite of their situation.
After she returns home, Umi discovers that her mother, Ryoko has returned. Ryoko reveals that Shun's father was Hiroshi Tachibana — the second man in the photograph. In 1945, Tachibana was killed in an accident on a repatriation ship. Shun's mother died in childbirth, and his other relatives were killed in the atomic bombing of Nagasaki. However, Ryoko was unable to raise Shun, as she was pregnant with Umi during her time as a medical student. Yūichirō registered the child as his own to avoid leaving Shun as an orphan in the confused postwar years, but Shun was eventually given to a local couple, the Kazamas. Umi, however, still has concerns.
Tokumaru later visits the Quartier Latin and, impressed by the students' efforts in renovating the building, he decides to abandon the demolition. Umi and Shun are soon summoned to the harbor. They meet Yoshio Onodera, now a ship's captain, who was the third man in the photograph, and the sole survivor of the three. Confirming that Umi and Shun are not related by blood, Onodera tells the full story of the three naval men in the previous era. With everything resolved, Umi returns to Coquelicot Manor and resumes her daily routine of raising the flags.