The first recorded settlement on the site was by Ella, King of the South Saxons c.AD490. Subsequently the medieval castle which occupied the entire island was built by Lord Reginald Cobham c.1342, who also built the medieval parish church of Lingfield.
In 1649 during the civil war, the castle was occupied by Oliver Cromwells parliamentary forces and demolished. After which the site remained derelict for more than a century. Around 1754 Sir James Burrow raised the level of the central island and landscaped the grounds build the present gothic style ornamental pavilion re-using medeival material from the site.
The castle is a grade II Listed building, an ancient Monument of the county of Surrey and a scheduled National Monument.