The founders of Gorlice and the first owners of the manor were a family of Kraków burghers named Karwacjan, who were mentioned in the town books in 1324. The town was founded by Dersław I Karwacjan, owner of, among other things, several tenement houses in Kraków, lead and silver mines and the castle in Janowice, who bought Gorlice from King Casimir the Great around the middle of the 14th century. He was succeeded by his son Dersław II, who inherited the town.

The dynamic development of Gorlice at the beginning of the 15th century necessitated the planning of a new market square and building plots. Then, the oldest part of the manor house, the manor tower, was built. It was a building with a long cellar and possibly a wine storehouse on a slope above a navigable river. In the 15th century, a square tower was added to it, with a basement, a first floor with lofts, and a wooden attic.  The stone mansion was a residential and defence tower, the seat of the nobility and knights, and complemented the town's defence system.  In the mid-16th century, after the death of the last of the Karwacjan family, Gorlice was inherited by the Strasz family. During the war in 1915, the bombed-out mansion burnt down, and by 1928, its walls were being dismantled to rebuild the surrounding houses. In 1938, work began to secure the remains of the manor house, which was only partially rebuilt in the 1970s.

The complete reconstruction, based on archive photographs, took place between 1982 and 1992, designed by architect Krzysztof Filar. The architectural decoration and landscaping of the surroundings was performed by Zdzisław Tohl. Today, the manor house is home to an art gallery. On the ground floor, you can see the remains of an early 15th-century and 16th-century manor house.  The stone cellars house a café, while the upper floors house exhibition halls, a large one named after Father Bronisław Świeykowski, the town's commissary mayor from 1914 to 1915, and a smaller one named after Professor Włodzimierz Kunz, rector of the Academy of Fine Arts in Kraków and Honorary Citizen of Gorlice.

The Art Gallery Karwacjan Manor House is the oldest establishment of the Museum of the Karwacjan and Gładysz Manor Houses, which, in addition to the exhibition halls, also houses a small hotel and administrative and storage rooms. The gallery has an educational profile, with lessons and meetings between artists and young people, films on art are screened, and workshops on various art disciplines and techniques are organised. Educationally sound and spiritually developing journeys into the ancient world await young people.  In more than thirty years of activity, the Gallery has presented more than 800 exhibitions of painting, printmaking, sculpture, fine art photography and other contemporary art forms.

In 2005, on the 650th anniversary of the town's foundation, a statue of Dersław I Karwacjan was unveiled in front of the building.