Sag Harbor became a United States port of entry in 1789, with a growing population involved in servicing whalers, coasters, and West Indian trade ships. The Custom House was owned by Sag Harbor's first United States Custom Master, Henry Packer Dering. The daily activities of Dering, his wife, and nine children are vividly portrayed in the room settings of a formal dinner, office, children's room, kitchen, pantry, and laundry.

The house offers tours and educational programs.