The current Richmond House is a Grade II* listed building built in the 1980s, incorporating the 18th century Richmond Terrace.
Richmond Terrace is built on the site of Henry VIII's White Hall Palace bowling green looking north towards the Privy Gardens.
The first Richmond House on this site was built around 1660 and occupied by Charles Stuart, 3rd Duke of Richmond from 1668. After the Duke's death in 1672 his wife continued to live in it until her death in 1702. As there were no heirs it was taken over as an official building first by the Secretary of State and then by the Comptroller of Army Accounts.
In 1710 another Richmond House was built adjoining the first, to a design by Lord Burlington for the 1st Duke of Richmond, the title having been recycled for Charles Lennox, an illegitimate son of Charles II. In 1738 the first Richmond House was granted to the 2nd Duke, so that it could be demolished in order to give him unhindered views. In 1758 the 3rd Duke took over the house and used it as a sort of studio come gallery for aspiring artists and sculptors following his return from his Grand Tour. The house was altered again in 1782 to designs by James Wyatt, but was destroyed in a fire in 1791.
The site remained vacant until 1822 when the Crown granted a lease to George Harrison who built eight houses to Thomas Chawner's design, between 1822 and 1825. With the exception of No 8, which was used as offices from the outset, the houses were lived in until the early 20th century. In the 1920s the leases on the houses expired and they were taken back by the Crown and used as Government offices.
In 1975 William Whitfield was appointed to produce a masterplan for the site. His design retained the front of the terrace whilst demolishing the rear parts and the mews to make way for a new office building. Starting in 1982, the refurbishment and remodelling of Richmond Terrace was undertaken by Building Design Partnership (BDP) while Whitfield Associates were responsible for the new building. On completion in 1986 the building was occupied by the Department for Health and Social Security (DHSS) and named Richmond House.
The Department of Health relocated to a site on Victoria Street in late 2017, with Parliament taking possession of Richmond House in early 2018. After reconfiguration work, Richmond House will provide additional office space for Parliament. Richmond House has also been identified as a possible location for a contingency Chamber.