One of my clients while awarding the project had set the expectation that his project will require 40+ hours per week, But has not been able to deliver more than 1-2 hours of work every week since the contract started.
Is it possible to terminate the contract without the client affecting my score/rating/jss?
It doesn't sound like a "bad" contract.
It sounds like a contract in which the client does not need as much work as you hoped.
Lots of freelancers would LOVE to work for a client who gives them 1 to 2 hours of work per week.
If this isn't a contract that you like, then simply tell the client:
Thank you for the opportunity to work on your project. I have enjoyed the work, but things have changed for me and I will no longer be available to work on this after next week. If you want my help documenting anything or training a new freelancer, just let me know. I will be happy to help."