Neighbor Demanding We Pay 4000 For Building A Fence After They Built One. Can They Force Us To Pay?

I am located in Kansas. A little over a year ago new neighbors moved in next door and immediately built a fence. I thought this was great because I have been wanting to fence the yard and because I rent I asked the landlord for approval and they had no problem with it. When they told us about their plans, we told them that we just rent and didn’t know where the property line was and that was the end of that. They also never informed our landlord. A month after their fence was completed and we started planning our own fence. Because one side of the yard was already fenced, we only had to fence the back and a small portion of the front that connected to their side. With approval from the landlord and permits from the city we built the fence but made sure to not actually connect to theirs.

As we were cementing in posts the neighbor came over screaming at us because we were building a fence and were going to be using their fence. They actually demanded we make a second fence along theirs so that we could not use it. Even mentioned how they spent 12,000 dollars on it and it wasn’t fair. At this point I was getting pissed and asked them for their property survey since according to the city property line maps show they were clearly on our side of the property line. Of course, they did not bother getting one and neither did we. After they left we finished up and the next day his wife comes over as we are finishing up and lets us know that they would allow us to use their fence until they decide to move at which point they would get a survey done. Ok cool, now we are done with all this BS nonsense. During the next year they let their fence go to crap and didn’t weatherproof or stain it and it looked like crap compared to ours.

Last week we get a call from our land lord that they wrote a letter demanding we give them 4,000 dollars plus damages because we built a fence without notifying them. On top of that they had someone come and powerwash and put stain on all the sides but left the side we face alone. Seemed to go out of their way to not even ask enter our yard to do that side. (edit) In my opinion they did this to claim damages for not having access to our side of the yard.

Can they actually force us to pay this? The landlord even said they are acting like idiots but were going to check with the city and let us know what to do.