Contractor Bid Going Up After Closing Our Mortgage

My husband and I are currently in the process of building a manufactured home on land we were given from his family.

However, every step has been an absolute nightmare. We are FINALLY closed on the mortgage... nearly six months after applying for it. Our mortgage company has been ridiculously incompetent and unhelpful so that's why it took so long.

Well, because they took so long, our contractor ghosted us. Can't get a hold of him and when we do, he wants us to file the building permits, despite being licensed and bonded. So we found a new contractor, who then pointed out that our original contractor left out some pretty big stuff on the original bid.

The mortgage company is saying they cannot adjust the amount, nor will they answer any of our questions regarding this. They said if we want to adjust the amount, we'll have to basically start all over again, paying a bunch of out of pocket fees accrued in closing.

What do we even do in this case?