The Locust Valley Library was founded by Frank N. Doubleday, Frederick A. Horsey, Edward Latting, James T. Mitchell, Jr., Dr. Edward Seaman, Charles F. Williams, and Edward M. Ward on October 17, 1909, with a $300 contribution from the year-old Matinecock Neighborhood Association. Its first home was a two-room cottage at One Bayville Road, near the corner of Bayville and Buckram Roads. The first Chairman of the Locust Valley Library Association was Neltje de Graff Doubleday (Mrs. Frank N.), who wrote nature and gardening books under the pen name "Neltje Blanchan." The first librarian was Miss Bertha Whiting.

The building that now houses the library was built in 1914 as the Matinecock Neighborhood House through gifts of money from local residents and donated services of carpenters, plumbers, masons, and others. There was a kitchen and bowling alley in the basement. The basement also housed the Locust Valley Fire Department. With the help of Frank N. Doubleday a library wing was added 1923, and in 1936 the Matinecock Neighborhood Association turned over the building and grounds to the Locust Valley Library.

In the early 1980's the building was renovated to expand library services. Throughout the years the building has continued to serve as a community center. In the late 1990's the building was again renovated to bring library services into the new millennium.