The first residential and defensive stronghold was supposed to be built in Baranów as early as the 12th century from the Jaksa foundation, who in 1135 was to erect an unknown timber stronghold on the order of Bolesław III Wrymouth. It was located in the vicinity of the ferry on the Vistula River, several hundred meters away from the Baranów. This is evidenced by the names of the neighboring villages: Przewóz and Przewoźnik, located on both sides of the river. The ferry near Baranów was particularly important because it connected trade routes running from the west from Kraków to the east towards Przemyśl with roads leading from the north from Sandomierz to the south, towards the Hungarian border. The name Baranów probably derives from the breeding of rams, which was then carried out by the inhabitants of this settlement, using the lush Vistula meadows for grazing.
The village of Baranów appears several times in written documents from the 13th and 14th centuries. Among others in 1266, the widow of Baran from Pełczyska sold Baranów for 15 fines of silver to Salomea, sister of Bolesław the Chaste, and in 1354 King Casimir the Great donated Baranów goods to a certain Pietrasz from Małachów and established a fair in the settlement. In 1376, the castle, or rather a wooden stronghold, was to burn down during the Lithuanian raid. The construction of a stone residential and defensive tower probably took place in the 15th century. The chronicler Jan Długosz mentioned that it belonged to the Baranowski family of the Grzymała coat of arms.
Around 1569 medieval building was pulled down by one of the subsequent owners – Rafał Leszczyński. It erected a renaissance defensive court, now embedded in the west wing of the palace, which has survived to our times. The extension of the court took place in the years 1591-1606 according to the design of Santi Gucci, at the time when the owner was the voivode of Brzeg-Kujawy, Andrzej Leszczyński. In this form, the palace happily survived the period of the Swedish invasion in 1655. Another expansion was carried out around 1695 according to the design of Tylman of Gameren, replacing the interior design, crowning the corner towers and introducing bastion fortifications, which replaced the older curtain walls. However, at the end of the 18th century, the Baranów fortifications lost importance, and were demolished in the first half of the 19th century.