The college was founded in 1542 by Lord Audley of Audley End on the site of a Benedictine college. Behind the Second Court is the Pepys Building, which contains Samuel Pepys' own library, bequeathed by him to the college in 1703 and containing the cipher manuscript of his Diary.