The Castle located on Kamplin Square was erected by the Frankopan noble family over several centuries and generations. The oldest part of the fortress is a square shaped tower leaning on the diocese building where once a courtroom stood. An inscription situated on the lunette above the entrance says that the building was erected in 1191, in the period of the Bishop Ivan and knights Bartul and Vid and with the support of the whole municipality.

The round tower on the northern part of the castle was built after the square one, probably in the 13^th century. It is not known exactly when, but the entire complex was completed before 1348. It was reconstructed twice around 1480 and 1600. It has a slab built into it with the lion of St. Mark and the inscription “Aureae Venetorum libertati” (Holy Venetian Freedom) dating back to 1500. There is another square tower in the eastern part of the castle, facing the sea, whilst the western part had a guardhouse. The complex also used to have areas where the soldiers were housed too.