🐈
» Forums » Coffee Break » Re: OFFICIAL PROPOSAL TO UPWORK: Failure in J...
Page options
arthur-culman
Community Member

OFFICIAL PROPOSAL TO UPWORK: Failure in Job Success Score algorithm

Hello fellow freelancers,

 

Before saying that "this topic should be messaged to admins/devs", let me say that I want to make this public to raise some discussion about it: Job Success Score and how it's calculated.

 

Let me get this out of the way: I sincerely believe that Job Success Score is a needed feature. This post is not against it.

However, there is one thing that I find very especially problematic in it. The lack of feedback can count negatively towards your JSS.

 

I have several completed jobs (usually small scale, one small job), where the client has disappeared or does not return my messages. Asking for a review is something that I might do once, but will not keep spamming an unresponsive client.

 

My arguments on why this policy is faulted:

1. On certain fields, there are a considerable amount of clients who use UpWork only for one project or have a minimal amount of projects on a large timespan. I cannot speak for other fields, but mine (copywriting/translation) I feel like it's an issue. They receive the work, release the payment and leave never to return again.

 

2. The sole fact that the payment was released with no disputes, speaks for the fact that job was completed succesfully.

 

3. The clients are not forced nor penalized for not leaving feedback, so if someone doesn't care enough to dedicate a small time, they can do that.

 

The lack of review does not directly equal that Job Success was bad. Thus, it makes no sense to count it negatively towards it.

 

 

My proposed solution to this problem:

 

Forcing the reviews for the clients. I propose an update in a following manner: when client has an option to release a payment, a simple window could pop up, kindly asking to leave an honest review to release the payment. The "Release payment" button cannot be pressed before at least the star-based reviews are left, text review can still be left optional. This will not take more than 20 seconds of clients time.

 

Same system can be applied to freelancers, too. Before withdrawing payments, system makes sure that all clients have received review from the freelancer. Does not take much time either and if freelancer has already left reviews to all clients, no problem the money can be withdrawn without any stops.

 

Job Success Score is great, but it really is obscure om how it's calculated. I have made deduction on why mine has gotten bumped down consistently by reading all the documentation and comparing it to my profile and work history.

 

As I said before, I chose to post about this because I want to see am I alone with this or is it a more widespread problem. If it's indeed a problem, maybe community engagement will encourage UpWork to review this policy, which does not directly reflect the input and quality of work.

 

Please let me know do you agree or disagree with this, and why?

Are you satisfied with JSS calculation system?

 

All the best to all of you

 

ACCEPTED SOLUTION
wlyonsatl
Community Member

Arto,

 

If the purpose of the JSS system were to be to provide true feedback for freelancers it would be transparent and only be calculated using elements the freelancer can affect. Of course, a freelancer cannot force a client to provide feedback, so we're told it's not a big deal for the JSS if the freelancer doesn't have a lot of "no feedback" completed projects. There is no known precise definition of "a lot," so non-feedback clients are just a fact of life for many freelancers.

 

But the purpose of the JSS system is to a) make it easier for clients to wade through the sometimes overwhelming proposals they receive and b) to make clients feel they are choosing "the best of the best" when they select a freelancer for their project. This is why the system clients use to post new projects defaults to a minimum JSS requirement of 90, despite the fact there is no evidence there is any real difference between a freelancer with a JSS of 89 and another freelancer with a JSS of 90.

 

There are other ways to deal with a), but they would require a little more work by the client. (We can't have that.) Upwork has provided no information that addresses point b).

 

Upwork needs a lot more clients spending a lot more money in order to eventually turn a profit. That is particularly important now that the company's shares are publicly traded. The big drop in the company's share price last week was an unavoidable reminder of this for management.

 

So, transparency is not a primary goal of the JSS system. (And if it were, there is no way Upwork could prevent freelancers unhappy with a client's less-than-stellar feedback from hastling the client.)

 

Upwork has made no apparent significant change to the JSS system during the few years I have used Upwork. Management has much bigger fish to fry and there is no reason for them to listen to freelancers' complaints about the JSS's weaknesses and inconsistencies as long as Upwork has far, far more freelancers than are needed to serve the far fewer clients who use Upwork. There is no reason to expect any change to the system is likely for the foreseeable future.

View solution in original post

20 REPLIES 20
flrgx
Community Member

I think this is a really good suggestion.

 

Job success score has been a point of contention for some freelancers and it can be a barrier for some people who are still starting out. Your suggestion legitimately helps people to improve their score.

petra_r
Community Member


Arto K wrote:

 

However, there is one thing that I find very especially problematic in it. The lack of feedback can count negatively towards your JSS.


"No feedback" contracts, provided money was paid and you don't have a really high percentage of such contracts, have absolutely no effect  on your JSS at all.

Nor do idle contracts (provided money was paid and you don't have a really high percentage of such contracts.)


You are asking to eliminate something that isn't even happening. Huge waste of time.

 

Before you say "But why is my JSS 83% when I have only 5 star feedback?" - go to your "My Stats" page and look at the private feedback those clients who did leave feedback (only) left for you (under "clients who would recommend me.")

 

JSS is mainly calculated from private feedback, because nearly everyone leaves 5 star public feedback.

 

 

versailles
Community Member


Arto K wrote:

The lack of feedback can count negatively towards your JSS.


 

Your Job Success Score on Upwork

 

(...)

A pattern of contracts that end without feedback can reduce your score


When it happens occasionally, your score is not impacted by contracts ending without feedback. But when this occurs in excess for a freelancer, it is flagged and counted against your score. Requesting that the client close the contract and leave feedback can prevent this situation. Contracts paused due to poor client behavior are not considered in your score.

(...)

 

also:

 

(...)

What else do I need to know about the Job Success score?


It’s intentionally different from your 5-Star rating
The scores differ because Job Success contains more than just public feedback. It also includes private feedback, long-term contracts, and repeat contracts. The collection of these inputs helps freelancers to better differentiate themselves on Upwork.

(...)

-----------
"Where darkness shines like dazzling light"   —William Ashbless
wlyonsatl
Community Member

Arto,

 

If the purpose of the JSS system were to be to provide true feedback for freelancers it would be transparent and only be calculated using elements the freelancer can affect. Of course, a freelancer cannot force a client to provide feedback, so we're told it's not a big deal for the JSS if the freelancer doesn't have a lot of "no feedback" completed projects. There is no known precise definition of "a lot," so non-feedback clients are just a fact of life for many freelancers.

 

But the purpose of the JSS system is to a) make it easier for clients to wade through the sometimes overwhelming proposals they receive and b) to make clients feel they are choosing "the best of the best" when they select a freelancer for their project. This is why the system clients use to post new projects defaults to a minimum JSS requirement of 90, despite the fact there is no evidence there is any real difference between a freelancer with a JSS of 89 and another freelancer with a JSS of 90.

 

There are other ways to deal with a), but they would require a little more work by the client. (We can't have that.) Upwork has provided no information that addresses point b).

 

Upwork needs a lot more clients spending a lot more money in order to eventually turn a profit. That is particularly important now that the company's shares are publicly traded. The big drop in the company's share price last week was an unavoidable reminder of this for management.

 

So, transparency is not a primary goal of the JSS system. (And if it were, there is no way Upwork could prevent freelancers unhappy with a client's less-than-stellar feedback from hastling the client.)

 

Upwork has made no apparent significant change to the JSS system during the few years I have used Upwork. Management has much bigger fish to fry and there is no reason for them to listen to freelancers' complaints about the JSS's weaknesses and inconsistencies as long as Upwork has far, far more freelancers than are needed to serve the far fewer clients who use Upwork. There is no reason to expect any change to the system is likely for the foreseeable future.

Hello Will,

 

Thank you for your in-depth answer, I truly appreciated it.

 

I understand what you said, to be honest I knew it already before posting, but I did not wan't to seem hostile towards UpWork as a company, as it is not my objective.

 

Obviously, as a for-profit company, UpWork will keep concentrating to the client side, as that is where the money is coming from. With the rise of remote worker/digital nomad movement, there will always be another freelancer, but clients are harder to obtain and keep.

 

This is the "not-really-my-problem" part, but with rise of new competitors on the market, it should be pretty important to listen to people who work for you. UpWork is to the date market leader and the best platform with large client base. However in my opinion they would need to look further into future, because if enough freelancers are gruntled enough, and better (or equivalent) option comes along, UpWork might experience migration, which will defintely be bad for stockholders.

 

What comes to the possibility of "what if the forced reviews are bad", then freelancer has not met the client satisfaction. To my experience, pretty much all clients first send a PM and ask a correction for bits their not satisfied with. Most of the clients know how important a good review is and will try the win-win way first, by asking the freelancer to correct their work.

 

Bad reviews happen when freelancers refuse to cooperate or are just not up to clients standards, which in my mind means that they have lied/boasted on their profile or application (the amount of clients that are really just "mean" is fractional).

 

In conclusion I agree with you that there is probably not going to be a change for this in foreseeable future. But if it's never discussed about there will never by a change and that's the reason why I started on this topic.

"gruntled"? I have always wondered what the opposite of disgruntled would
be. 😉

This is the time when I play the "not-a-native-speaker" card. Smiley Very Happy

versailles
Community Member


Arto K wrote:

I propose an update in a following manner: when client has an option to release a payment, a simple window could pop up, kindly asking to leave an honest review to release the payment. The "Release payment" button cannot be pressed before at least the star-based reviews are left, text review can still be left optional. This will not take more than 20 seconds of clients time.


I love the idea. Maybe worth exploring.

 

 

-----------
"Where darkness shines like dazzling light"   —William Ashbless


Rene K wrote:

Arto K wrote:

I propose an update in a following manner: when client has an option to release a payment, a simple window could pop up, kindly asking to leave an honest review to release the payment. The "Release payment" button cannot be pressed before at least the star-based reviews are left, text review can still be left optional. This will not take more than 20 seconds of clients time.


I love the idea. Maybe worth exploring.


Pointless / unworkable / unfair:

  • Doesn't work with hourly or multi-milestone contracts.
  • It is only public feedback, which is just about meaningless as far as the JSS is concerned.
  • Leaves more feedback for contracts with many small milestones than big contracts, and none on hourly.
  • Clients can't always know how the contract will pan out after the first milestone
  • Often clients don't want to leave feedback. They are not happy, but don't want to ruin the freelancer's profile either. So they let the freelancer close the contract so they don't have to leave feedback. So forcing this does not do freelancers any favours...

Often clients don't want to leave feedback. They are not happy, but don't want to ruin the freelancer's profile either. So they let the freelancer close the contract so they don't have to leave feedback. So forcing this does not do freelancers any favours...

 

Well, that makes it even more important to have the feedback forced.

 

I'm not looking for 5-star reviews if I haven't earned them. But having forced feedback would give clearer picture of the true skills of the freelancer. UpWork often talks about updates that they are "good for the overall community of clients and freelancers". This would definitely be that kind of update, don't you agree?

 

Also the thought of "maybe they don't want to because I did something wrong" has crossed my mind. By then I definitely wan't to know how I did not meet the satisfaction of the clients to make sure that I improve my working methods accordingly. We learn from out mistakes.

 

>Clients can't always know how the contract will pan out after the first milestone

Feedback is left after the contract has ended. Otherwise it's shown as "job in progress". Correct me if I'm wrong, but review can be left only after the contract has closed?

Yes, that would be real feedback, which is not the purpose of the JSS system.


I'd prefer that Upwork tell the client the freelancer will automatically
receive "average" ratings across the board after a contract is completed
unless the client selects other ratings.


Only clients who were dissatisfied or very satisfied would be motivated to
leave more specific feedback and the idea of a freelancer's performance
being "average" would lose some of it's current stigma.


Right now, every freelancer wants their JSS to say they are above average,
which is not a possibility even the average high school graduate can
understand.




Arto K wrote:
Often clients don't want to leave feedback. They are not happy, but don't want to ruin the freelancer's profile either. So they let the freelancer close the contract so they don't have to leave feedback. So forcing this does not do freelancers any favours...

 

Well, that makes it even more important to have the feedback forced.

 

I'm not looking for 5-star reviews if I haven't earned them. But having forced feedback would give clearer picture of the true skills of the freelancer. UpWork often talks about updates that they are "good for the overall community of clients and freelancers". This would definitely be that kind of update, don't you agree?

 

Also the thought of "maybe they don't want to because I did something wrong" has crossed my mind. By then I definitely wan't to know how I did not meet the satisfaction of the clients to make sure that I improve my working methods accordingly. We learn from out mistakes.

 


I seriously don't think that FORCING clients to leave feedback is going to benefit the community in any way, and if anything, it will negatively impact JSS scores if you start annoying clients by giving them more hoops to jump through. You can talk about wanting honesty, but at the end of the day, a JSS score of 89% vs. 90%+ means the difference between getting jobs vs. not getting jobs, so if you ask a client for feedback and don't get a response, then you're better off not pestering them.

 

As for your desire to know whether your clients are satisfied... personally, I have a pretty good idea of what the client thinks of my work based on our communication on the message board, so I don't need little stars to tell me whether I've done a good job or not. But if you really want to know, why not just ask them? "Are you completely satisfied with my work, or is there anything else that I can do for you?" That would be better customer service than demanding that your clients fill out a feedback review.

 

Right, Christine.

 

Allowing their feedback on freelancers to default to "average" should make clients happy - they'd have to do even less than the current JSS system requires.


Will L wrote:

Right, Christine.

 

Allowing their feedback on freelancers to default to "average" should make clients happy - they'd have to do even less than the current JSS system requires.


That would be to the disadvantage of freelancers with an above-average JSS, so I'm against it. 😉

 

Currently, jobs with earnings but no feedback are excluded from the JSS, which is more or less equivalent to giving them a score equal to the freelancer's own average score. That seems better than your suggestion, under which a no-feedback job would drag an above-average JSS down and lift a below-average JSS up.

 

Freelancers are only penalised for no-feedback jobs in two special cases: (1) jobs with no earnings, and (2) freelancers with an "excessive" proportion of no-feedback jobs. Instead of your suggestion, you should be objecting to those special rules (assuming you don't like them). I would agree that the first case is in need of a change. The second case seems relatively unimportant, as I think it affects only a few freelancers.

>so if you ask a client for feedback and don't get a response, then you're better off not pestering them.

Is that not the point of this post?

 

>But if you really want to know, why not just ask them?

Surprisingly enough, I actually do that.

 

What comes to "hoops to go through" we're talking seconds here, not a 10-page detailed questionnaire.

Hi Arto. If I've counted correctly, you have 6 jobs with feedback, and your JSS of 83% equals 5/6 to the nearest percent. Coincidence? I think not. I would say that one of your clients left you not-so-good private feedback, perhaps the one who left five stars but no text. That would fully explain your JSS. As Petra said, jobs with no feedback don't affect your JSS unless you have an "excessive" proportion of them. We don't know what the critical proportion is, but I think it's safe to say that you don't have it.

 

There are flaws in the JSS system, but I can't see Upwork making the change you're asking for. Personally, I'm saving my petition ammunition for my own hobby horse: getting Upwork to fix the error in its time tracker algorithm which causes it to overcharge clients.

lysis10
Community Member

People are so freakin one dimensional. Sometimes, a client not leaving feedback is a GOOD thing when they're not happy. You don't want to force clients to leave feedback. When a client is kinda "meh" on the work, lots of them will tend to just disappear and don't want to leave feedback. They're not raging mad unhappy. They're just kinda "meh." This is when you don't want to be the "plz give me feedback sir" type of person and just let them disappear and close it yourself.

 

And yeah, this doesn't cover hourly where you have months and months of just automatic payments.

I think forcing feedback is not a good idea...For example if you catch me on the wrong day, I can be a grumpy sod and If on one of those days some website has an annoying pop-up forcing me to do something when I'm trying to do something else...I am not going to be a happy camper at all and you will receive my ire!

jill_whitehead
Community Member

I believe the JSS is highly flawed. 

 

I have had my account open since 2013. I have never had a poor review, tons of 5 star ratings and every client said they would recommend me. I took a year off Upwork came back, got a job right away, closed out another contract with a 5 star rating and took another test on Upwork. This was all in 7 weeks. Two 5 star ratings with raving reviews. 

 

I do not even have a JSS. I contacted chat support and they said it is because I have had 6 paid contracts in 24 months, because I took long term positions. So basically I have to get 2 more jobs and close both of those contracts before I can get my rating back?! 

 

You have got to be kidding me? They are costing me money. We all know the clients who are willing to pay the higher rates are the ones who look for rising talent and 90% JSS or higher! I feel this is very unjust. If you provide a reference for an inn-house job they don't ask for a time frame, so why is it different with Freelance. I performed the work and earned the rating. I should be able to maintain my rating, unless I have a negative review, even in which case the negative review should be up for review, as we all know some clients are impossible to work for. 

 

Anyone else have this experience? 


Jill W wrote:

 

Anyone else have this experience? 


Basically everyone on Upwork has had that experience. No one gets a JSS until they've completed enough jobs in the recent past, whether we are new or returning.

 

A JSS that's based on just two jobs could be terribly low. No JSS is probably better than a low JSS to some clients.

Latest Articles
Learning Paths