The penalt would cap weeks of heated negotiating and underscore the extent of the bank’s legal woes, people briefed on the talks said.