If you close the contract, you will be required to leave feedback for the client and the client will be notified of the closure and invited to leave feedback for you. Unless you've communicated with them fairly recently, it is unlikely they will bother and it will be a 'no-feedback' contract. That has a neutral impact on your JSS unless you develop a "pattern" of no-fb contracts which can harm your score.
Also worth remembering is that contracts are now weighted by dollar value in the JSS calculation. And as always, each one potentially weighs more when you don't have very many.
Personally, I would let it sit. I never see a reason to change a neutral factor into a potentially negative factor