On the hilltop above the very picturesque village of Castle Combe there was a Roman villa vacated in the 5th century A.D. Little happened until Reginald de Dunstanville built a Norman Castle on Castle Hill during the English Civil War of 1135-1154, so putting the "castle" into Castle Combe. By the 14th century this Norman castle had fallen into a state beyond repair and a new manor house was built in the shelter of the valley below.
http://www.thecastleinn.co.uk/castle-combe/a-brief-history/