The castle was founded in 1204 by Duke Ludwig I.

Sections remaining from this initial building period include the Keep, the Hall (Alte Dürnitz") and the Castel Chapel dedicated to St George, a double chapel with its original statuary (ca. 1230/35) and winged altarpieces from the 15th century.

Under Wilhelm V and his consort Renata von Lothringen the complex was transformed into a Renaissance palace: the Italian annex, the arcades and the famous wall paintings for the Fool`s Staircase ("Narrentreppe") featuring scenes from the Italian Commedia dell`arte date from his period.

A further highlight is the series of tapestries depicting events in the life of Otto I of Wittelsbach..