chadworth
Member

Reduced Client Happiness Score - So It Happened to Me

I've been watching my "job success" score with some interest.

 

I've noted Garnor saying that it's updated at least once every two weeks.

 

When it was first introduced I had 79% client happiness.  I understood how that could be.

 

I refunded a client when we couldn't come to terms and my client happiness increased to 83%.

 

I took on three jobs since then, 5 star feedback for all three.

 

All three of those clients ended contracts and rehired me. 

 

I was on tenterhooks waiting to be catapulted into the heady realms of the top rated freelancers.

 

My client happiness score dropped from 83% to 81%.

 

I checked with customer support and they said:

Oh yeah.  That's a known issue.

 

Can you remove it from my profile since you're acknowledging that you have an issue?

Nope can't do that.

Can you manually set it to what it should be?

Can't do that either. 

Can you remove it until such time as you work out your issue?

Nope.  Can't do that either.

 

Now I've been stuck in the queue with an open ticket for 8 days.  And the erroneous information is being displayed to who knows how many potential clients.

 

I know it's being shown to clients.  I am also a client.  Every once in a while oDesk sends me my profile, and suggests it would be great if I hire myself. 

I kid you not.

 

How is this ethical?

oDesk admits that they are displaying erroneous information on my public-facing profile--information that mischaracterizes my abilities and propensity to deliver high quality work and it's no biggie apparently that it's there and potential clients use this erroneous information to evaluate my suitability for work.

 

What is wrong with the people who administrate this site?

If you have something you acknowledge is broken, at least make it private, or move it to a development server until you get it working.

 

To intentionally display inaccurate information is crossing some kind of line now.

36 REPLIES 36
chadworth
Member

Update.

 

Now I'm seeing Good Fit badges again.

 

I wonder if this is related.

kugrin
Member

One word.. actually,  two.. totally ridiculous. 

 

 

lanwanman
Member

LQQKs like the accurancy of the My Stats Comunitation score isn't much better. See separate post "Inaccurate My Stats | Communication." Seems like the more active the freelancer gets, the more inaccurate My Stats reporting gets.

 

Hire yourself...lol...!?!?  "Client Happiness Score," another good one...lol.... According to OD, the Good Fit badge has been removed from all job descriptions.

Ron aka LanWanMan

 

Hire yourself.

Yeah it's no joke.

 

While I appreciate that oDesk apparently puts me in front of lots of people, it's no good that my profile is displaying incorrect information. 

 

My marketing effectiveness stats have suffered as a result.

 

I wonder if that factors into the calculation of my client happiness score any.

 

Am I stuck in a negative feedback loop?

Hi Ron,

Regarding the issue you're seeing with Responsiveness counts. I'm seeing your responsiveness numbers showing at 100%. I wonder if this is due to a correction when the client withdraws the invitation (and thus shouldn't get counted against your responsiveness score). We've also seen your Job Success score holding steady at 100% (great job).

Garnor, thank you for the reply. Less than 10 hours later, and I see My Stats | Communication (Responsiveness) back to 100% (aka Always). In the future if I see something that appears irregular, I'll wait 24 hours or more to see if the system corrects it on its own like it did this time. It did seem to happen when the client cancelled a job opening that I had bid on.

Ron aka LanWanMan
kugrin
Member


@Chad R wrote:

[...] oDesk admits that they are displaying erroneous information on my public-facing profile--information that mischaracterizes my abilities and propensity to deliver high quality work and it's no biggie apparently that it's there and potential clients use this erroneous information to evaluate my suitability for work.

 

What is wrong with the people who administrate this site?

If you have something you acknowledge is broken, at least make it private, or move it to a development server until you get it working.

 

To intentionally display inaccurate information is crossing some kind of line now.


Well said, and I agree. It's also not the first, second, or third time this happens. In fact, I am not even sure if a single feature has been rolled out without major issues in the last year or two, and I can't quite fathom why or how that is still acceptable. It's not even perfection that I expect because there are always going to be bugs, but some basic diligence and upfront testing would be nice. I don't see a pressing deadline for most of these features, so why not wait a week or two, really make sure it's done right, and then roll them out? Rolling out broken / dysfunctional features one after another robs oDesk of all credibility. It's quite sad to witness actually. I also imagine it to be expensive, because a heck of a lot of support resources are wasted on issues that could've been avoided. It would also benefit people with time sensitive issues like account holds that currently have to wait for months on end before they can expect any kind of resolution because it seems everybody at oDesk is busy logging bugs. I'd really love to hear oDesk's Head of Product take on this, because I hold him personally accountable for all of these fails. 

garnorm
Member

Hi Chad,

Thanks for the post. I checked into your Job Succes score and My Stats page and have some follow up. 

 

It appears you received an incorrect response from our Customer Support team (we're following up on that) as we do not see any bug affecting your Job Success score and instead it's behaving as we would expect given the ratings and other measurements we've seen from your job activity.

 

Regarding being shown your own freelancer profile when you're hiring as a client, we're digging into this one. It sounds like an error on our part that we need to fix. We have our Engineering team taking a look at it.

Garnor, how come the good fit badge is visible again? Has it been totally revamped and made to work properly ? if so, how come nobody announced it was being re-introduced. You at oDesk were rightly given genuine kudos for listening to legitimate complaints and removing the feature. But that kudos will surely melt into nothing if all that's happened is that the good fit thing has been tinkered with and sneaked back in again. You had the chance to mention it in the last couple of posts, so while you're hopefully still around, could I ask you to clarify whether the Good Fit feature has officially been re-introduced; and if so, (a) how come it wasn't announced?, and (b) how will it differ from the previous function?

Hi Stephen,

 

The Good Fit badges shouldn't be showing. We are checking with the Team to make sure the issue that is causing them to be displayed is sorted out.

 

Thank you.

~ Valeria
Upwork

"Every once in a while oDesk sends me my profile, and suggests it would be great if I hire myself."

 

Chad also mentioned the above, which I think is particularly worrying. Would a moderator be kind enough to comment on this?


@Garnor M wrote:

Hi Chad,

Thanks for the post. I checked into your Job Succes score and My Stats page and have some follow up. 

 

It appears you received an incorrect response from our Customer Support team (we're following up on that) as we do not see any bug affecting your Job Success score and instead it's behaving as we would expect given the ratings and other measurements we've seen from your job activity.

 

Regarding being shown your own freelancer profile when you're hiring as a client, we're digging into this one. It sounds like an error on our part that we need to fix. We have our Engineering team taking a look at it.


 So regardng "behaving as we would expect given the ratings and other measurements we've seen from your job activity".

 

Three clients who left 5 star ratings, and rehired me should lead to a decrease in the client happiness score?

 

My profile is not private.

 

Everyone can see this is the case.  How is this working as reasonable people should expect?

Chad

 

You have one no feedback at all and another 3.5, and even with that you don't know what other's said privately, which could change your score. Also what we can't see is anything you might have cancelled or not started or refunded. So you can ask but they are not going to tell you publically nor can we guess on those factors.

 

Your scores will change again when you complete the six jobs you have going right now. If one of those has completed it won't show for a week or two.

 

 


@Suzanne N wrote:

Chad

 

You have one no feedback at all and another 3.5, and even with that you don't know what other's said privately, which could change your score. Also what we can't see is anything you might have cancelled or not started or refunded. So you can ask but they are not going to tell you publically nor can we guess on those factors.

 

 


Thanks for responding Susan but you're missing the point entirely.  I had all of these (the 3.5 rating and none responsive client) before AND my client happiness score was higher than it is now.

 

In the OP I noted that I could understand how it could have been the 79% it was originally.

 

It increased to 83% afterwards.

 

Since the increase to 83% I have had 3 five star ratings and been rehired by those clients.

 

What part of those three 5 star ratings and rehires should cause my score to decrease from 83% to 81%?

 

I think if people weren't starting from the perspective that there isn't something wrong with the algorithm, or its implementation, we wouldn't have these sorts of misunderstandings.  The algorithm obviously makes a diffrerentiation between people with above 90% client happiness and those below.  We have the Top Rated badge as evidence.  It is quite likely the algorithm is skewed in such a way that those with higher ratings continue to be rated highly, while those with lower ratings have their scores incorrectly calculated.

 

I also note that despite Garnor's assertion that there isn't a known issue with the client happiness score, I have received no indication from customer support that there is no issue.

 


@Suzanne N wrote:

Your scores will change again when you complete the six jobs you have going right now. If one of those has completed it won't show for a week or two.


 

What is to say my score won't further decrease after I complete these 6 jobs, given that it decreased AFTER successfully completing 3 jobs?  That is my issue.

 

And I'm not the only person who has noted scores fluctuating for zero logical reason.  The forums are peppered with those complaints.


@Suzanne N wrote:

Chad

 

You have one no feedback at all and another 3.5,  


I would also note that there numerous instances of people who have less than a 4 star rating and 100% client happiness.

 

Others have posted links to those profiles here on these forums.  I'm not sure forum guidelines permit me to do so, so I'll refrain.  I hire people on oDesk ever so often and I'm always looking for freelancers. 

 

I know I spotted two yesterday. 

*UPDATE*

 

I just received a communication from Tech Support.

 

I asked them to clarify whether there was indeed a known issue with regards to the client happiness score.

 

Tech Support responded to indicate:

 

a) There is a known issue that is "complex"

b) Said issue is still being "worked on and monitored."

 

I believe posting the entire contents of the email would violate forum guidelines.  If I am given permission by Garnor or some other mod to redact parts that would offend the guidelines I am quite happy to post the redacted email.

 

 

Tech support doesn't always know either and it is easier I think for them to say yes or no then to find out. But if one of mods would answer then I guess you will get your answer.

 

I had a ticket closed with it was an issue on thier end and it had been fixed and it was not an issue at their end nor had it been resolved. It was a policy they don't have mentioned anywhere.

 

You ask what happens when you close your jobs. Well I guess it will depend on the clients private feedback and how they score you won't it.

 

I have seen a lot of clients post publically that they reserved five star ratings for those who were impeccible and never gave 5 stars. So perhaps some are giving 5 stars publically but are giving less privately.

 

If there is an issue with the system guess we wait until they fix it. I don't think any amount of complaining in the forum is going to make them fix it any faster.

 

And I wouldn't post an email between you and support on the forum unless you want to get banned from the forum or a warning.

Hi Chad,

 

Please, refer to the most recent message sent to you through the ticket. It contains more details about your Job Success score.

 

I would like to confirm once again that there are no bugs or technical issues affecting Job Success score.

~ Valeria
Upwork

*UPDATE*

 

This is interesting and troubling all at once.

 

Engineering department says: We definitely have a known issue with the client happiness scores.

 

Chad says, hey tech support ask Engineering if they're sure cause a forum moderator says there are no issues,

 

Tech support says, no no, we're very sure.  I checked and it's complex and being worked on as we speak.

 

Late last night I get an email from Tech Support.

 

Again, there are prohibitions on pasting the entire email.  One line in this email says: "it somehow is not related to any bugs or known issues".

 

Comedy or Tragedy?

 

The rest of the email is almost verbatim pasted from a moderator's comments on this very forum, the sort of mealy marketing speak some find objectionable and unsatisfying, particularly because it's very good at deflecting rather than giving answers to pointed questions.

 

Let's see how much further I can go into this rabbits' warren before I get ousted.

Now that I understand oDesk's position I would like to point out a few things.

 

oDesk is not claiming that I didn't get three 5 star ratings with stellar comments and my client happiness score decreased.

 

oDesk takes the position that the decrease is to be expected.

 

There are a variety of issues I could key on but lets stick with technical details.

 

The algorithm makes a computation that oDesk is heavily invested in keeping secret.  However, this algorithm has real world consequences, and its utility is predicated to an extensive degree on its intelligence or infallibility--you choose.

 

But if the algorithm (or its handlers) isn't smart enough to understand that three five star feedback ratings, accompanied by glowing, thoughtful comments that clients took the time to think about and set down, which lead to all three of these clients rehiring the contractor is MORE IMPORTANT than whether the clients may or may not have not checked the "I will recommend this freelancer to my friends" button then we have a very large problem.

 

From a different angle:  The algorithm should be smart enough to understand that the very act of giving five star feedback and rehiring a contractor is a better indicator of client happiness than giving the freelancer a high rating on the recommendation metric.

 

Last angle:  After all, what speaks loudest?  Actions or words?  I know from experience, that I don't work for free.  For a client to say this freelancer was great, and then turn around and pay me more money should count for more than whether they are willing to say they would tell their friends about me.

 


Krisztina U wrote:

Rolling out broken / dysfunctional features one after another robs oDesk of all credibility. It's quite sad to witness actually. I also imagine it to be expensive, because a heck of a lot of support resources are wasted on issues that could've been avoided.

 


 

 

I don't have the attention span to go into too much detail right now, but I think the culprit here really is the recommended score (aka Net Promoter score or NPS). I found the concept interesting and read up on it, and while I am by no means an expert on the subject matter, I think the Net Promoter score is misused and used to measure something it's not intended to measure and therefor gives false results. The Net Promoter score is used to measure direct customer satisfaction, so if oDesk used it to ask the question "how likely would you be to recommend oDesk", or if we asked our client "how likely would you be to recommend me to a colleague", the result might differ. I'd suspect in some cases quite drastically actually. There are a lot of pros and cons, it was originally invented by GE, but I haven't read one recommended usecase that even comes close to oDesk's business model or where it's used to measure an indirect relationship. oDesk seems to weigh the ratings after NPS criteria, so a 9 and 10 equals loyalty and satisfaction (promoter), 7 and 8 are counted as passive satisfaction (neutral) and excluded (!) from the result, and everything less than 7 is bad. Which means that someone that gets a 6 is treated the same way as someone that gets a 0, both fall into the group of "detractors". 

Net Promoter Score.

 

Haven't had a chance to look into this but I will now that you brought it to my attention.

 

Thank you Krisztina.

 

As an update my score has fluctuated back up to 83% without any change in job standings.  Have closed no contracts, have opened no new ones. 

 

 

Chad, keep in mind that the score looks at last 12 or 24 months, so the cause for the fluctuation might be something that happened a year or two ago. It's a little bizarre.

I've only been on oDesk 3 months give or take so. . .

 

I doubt that.

 

So another update.

 

A client closed a contract last night.  5 star rating.  We'll see what this does to my client happiness score.

Another Update.

 

Client happiness increased by 1 measly percentage point.

 

I suppose this is better than decreasing so there's that to give thanks for.

 

Thank you upwork.

Update

 

Yesterday my client happiness score floated up to 88%.

 

I took on one other client last week, and an old client gave me two new jobs.

 

I have not applied to as many jobs as I normally would. 

 

I do note that a client that payed me a significant amount of money recently ended a contract.  She recommended me highly both publicly and privately (I think).

 

I also happened to note browsing through the forums yesterday that greater weight is given to ratings that come from larger projects.  This was in the context of the star rating, but I would not be surprised to learn that the client happiness score is weighted this way as well.

 

Stuff to consider.

 

Chad, keep in mind that anything less than a 9 for the "would you recommend this freelancer" question is going to hurt you to some degree. I do think it makes sense to have the star ratings dollar weighted. Imagine you get a 5 star on a $5k project, and a 1 star from a crazy client on a $25 project, why should your overall rating suffer? I wish the JS was dollar weighted as well, and while we haven't received confirmation by the mods, evidence seems to suggest that it isn't. Which means you could have 100s of successful jobs and make tens of thousands of 5 star dollars, and one unreasonable client can seriously hurt the score.

It happened to me too--but it only dropped 1% so no complaints. Still, wish I knew how Job Success is computed. In my case, *perhaps* it's because the 'Long-term clients' percentage declined, since I've had nothing but 5.0 feedback and 'Clients who would recommend you' hasn't changed at 93%. Cat Frustrated

__________________________________________________
"No good deed goes unpunished." -- Clare Boothe Luce

It's the secret sauce John.

 

Pawns like you and I aren't worthy to behold its glory.

 

I'm tired.  That's a poor attempt at sarcasm.

 

Forgive me.

I only have anecdotal evidence, but I'm beginning to suspect that the client happiness score is weighted as well.

 

Take a look at Setu's thread here:

https://community.upwork.com/t5/Freelancers/3-Inherent-Biases-of-Upwork-s-Platform/m-p/81392#U81392

So got an email this morning--oDesk telling me that I'm Top Rated.

 

What this means is that I love oDesk now and will not complain about the Client Happiness score again.

 

Well, at least not until I dip below 90% client happiness again.

 

Before people start with the pitchforks--that was a joke directed at a friend on these forums who brought to my attention that some of our fellow freelancers don't see an issue when upWork does something that negatively affects all freelancers until they personally experience the consequences.  Smiley Wink

 

I note that I have applied only twice to jobs  during the last calculation period (and won neither of those btw); have closed no jobs; and gotten repeat work from only one long standing client, while continuing to work on a long-term contract for another client.

 

 

 

 

Congrats Chad, keep up the good work 🙂

You had a job that fell out the calculation period; that must have raised your score.
---- easy like Sunday morning ----

MY ONLY SLIGHTLY MORE USEFUL THAN BASELESS SPECULATIONS ABOUT THE CLIENT HAPPINESS SCORE CALCULATIONS.

 

Now that I’ve been investigating/following this issue for some time, I feel confident enough to make some observations.

 

It does not appear to me that most things take an immediate effect on your score. I think if you get a particularly egregious negative rating that might send the equivalent of a push notification to the client happiness robot and prod it to recalculate and display your score today, but apart from that, when you see a change in your score, what caused your score to change happened about two weeks ago-or more accurately, in the period between the last two calculations.

I suspect also that there is a subset of people at odesk authorized to cause an immediate recalculation of your score and there is probably a policy in place that governs that. Obvious examples would be if odesk contacted you directly and asked you to work with a client and that client left superlative feedback. It would be to their benefit, and yours, to force an immediate calculation if your client happiness score will make a significant upward bump--or they are likely to percieve it that way at any rate. I am sure most of us are sufficiently paranoid to be able to imagine scenarios where odesk employees would be authorized to force a recalc for negative reasons.

I also think the system is weighted towards more recent occurrences. So let's say you take a 90 percent hit to your score for a negative review today--public or private. 6 months from now that negative review will count negatively against you but not 90 percent negative. Say 30 percent for the sake of this purely intellectual exercise. Then past the two year cutoff, it doesn't count at all.

Again this is only marginally better than baseless speculation. While I do have data to support my conclusions, I have no way of knowing whether what I observed was not caused by engineer-tinkering with the algorithm, since I began collecting data.  See for example:

 

https://community.upwork.com/t5/Freelancers/Job-Success-score-sudden-drop/m-p/83624#U83624

 

and particularly:

 

https://community.upwork.com/t5/Freelancers/Job-Success-score-sudden-drop/m-p/83790#M37258

What do you mean you didn't write this with your main account?  If you created another account you can get suspended because it's agains ToS

"Fairness is giving all people the treatment they earn and deserve. It doesn't mean treating everyone alike-Coach John Wooden"

I posted a honest reply in a forum. You know very well that no freelancer would ever post something like this with their active profile. You don't even know if I'm still on Upwork or have since moved on (in fact, I have, for many reasons).

 

It is also telling that this is the first thing you focus on. I posted this to help the Upwork community to improve this platform, nothing more, nothing less. They can do what they want with that. And I hope they will use it to become better.

 

Maybe even excellent.

 

 

Okay was just letting you know. Have a great day.

"Fairness is giving all people the treatment they earn and deserve. It doesn't mean treating everyone alike-Coach John Wooden"