๐Ÿˆ Community
setumonroe
Member

Successful!? JS back-test and prediction :)

Over the past month I was able to successfully back-test 5 different profiles accurately, obtaining the correct JS score figures for up to 4 previous iterations. Of course, I have used what information was given by the platform as the components of the JS score, and deduced what I figured was relevant (all important) in making a job "Successful."

 

With the help of an un-named "accomplice" providing additional profile information, I was able to reproduce 3 past JS scores in said profile; but more importantly, correctly predict the current score updated overnight :). It is simple, and may not contain all the possibilities - but it worked. Undoubtedly this is not fool proof and may be incorrect, but worth a try.

 

I have no idea what the repercussions to this may be by Management, but knowledge should be free.

 

Once again I will say, knowing the formula for the JS score calculation, can in no way lead to the system being "gamed" - since the freelancer has no way of knowing the private feedback left, and can only "game" the score by actually doing better work. Which is ultimately what Upwork touts it wants.

 

Microsoft Excel - JS score.xlsx (07-12-2015 08.33.58).png

---- easy like Sunday morning ----
20 REPLIES 20
monstersong
Member

You're a GD champion, Setu.

Only drawback is that it will only work on relatively new accounts that came about right before or after JS introduction. Since those accounts can be checked against the changes in the score.

For older accounts Upwork would have to provide us with our data.
---- easy like Sunday morning ----

The JS score should be updated again in roughly 12 hours. I have been modifying and testing a basic formula to at least see if the primary components are correct. However posting after the fact cannot be a true test. A real test lies in making an actual public prediction based on past assumptions.

 

Therefore this is the predicted score using the same profile. Hopefully it will be successful; if not, no loss incurred.

 

Microsoft Excel - JS score.xlsx (07-25-2015 07.36.24).png

 

There can be three variants to the score:

Contract 15/07/2015 = 0.5, then JSS = 93%

Contract 15/07/2015 = 1.0, then JSS = 96%

Contract 15/07/2015 = 0, then JSS = 89%

---- easy like Sunday morning ----

Thanks for all the hard work to get this information for us.

Much appreciated Jean.
---- easy like Sunday morning ----

How did you do with your predictions?

Sorry to disappoint you Jean, but it was unsuccessful. The profile showed 95%.

 

I won't make any excuses. Using "jobs" as an item the formula worked on many profiles for many updates of the JSS. I believe this was changed (perhaps related to the massive glitch everyone encountered with the JSS last week), from "jobs" to "outcomes."

 

(successful contract outcomes โ€“ negative contract outcomes) / total outcomes.

This was not what was explained to me originally. And no, I am not pulling formulas out of my head.

 

There are two ways you can look at it:

[{(Job1 + Job2 +Job3) / Total} + feedback + disputes etc + longterm] = JSS

or

[(Job1 + feedback + disputes etc + longterm) + (Job2+ feedback + disputes etc + longterm) + (Job3 + feedback + disputes etc + longterm) / Total] = JSS

 

Either way, I have found some thing to be apparent:

 

Your Job Success Score on Upwork - Upwork Blog - Google Chrome (07-26-2015 10.40.08).png

Long term relationships don't really count in trying to understand the formula since you can have 100% JSS without it. Which raises another question, it poses a means of inflating the score.

 

Your Job Success Score on Upwork - Upwork Blog - Google Chrome (07-26-2015 10.42.09).png

Public feedback seems to contribute next to nothing and now I am worried about the actual role of private feedback. Consider these profiles:

 

Hire Data Entry Specialists - Upwork - Google Chrome (07-26-2015 11.01.41).pngHire Data Entry Specialists - Upwork - Google Chrome (07-26-2015 11.02.46).png4.png5.png

 

Remember there can be no private feedback without public feedback.

I was told however that feedback is collected throughout long term contracts - not just at the end. I don't know what form that takes, but I guess that is why we can have so many profiles like these.

 

Your Job Success Score on Upwork - Upwork Blog - Google Chrome (07-26-2015 10.41.31).png

We are told that these only affect your JSS when in excess. I have found that to be true. So therefore what exactly constitutes the core of the JSS calculation, if all these factors can be ruled out?

 

(successful contract outcomes โ€“ negative contract outcomes) / total outcomes.

 

What these outcomes are, how many per contract, if they are handled separately; I could not say.

 

In concluding, I know less now than I did a month ago. Advice to myself is to forget about figuring out what is important, much less the formula itself.

 

For the record, I took this on months ago because it is something that is not going away and affects every aspect of your success here - unless you have lifelong clients. I was also forcing more information out of management by doing things this way, unfortunately only a little was revealed. I know more about this topic than anyone here because I have had dialogue with the powers that be on this subject - just to indicate that I am not spouting garbage.

 

With all that I am putting it to rest because it is pointless.

---- easy like Sunday morning ----
christian1220
Member

Thanks Setu for all your great work! To me, you had a very important part in better understanding how JS is calculated and what's in there.

 

Personally I am not that interested in JS stats / JS formula but I read all you wrote 3 times ๐Ÿ™‚

 

I say that making JS easier to understand is the real win for this community.

monstersong
Member

Setu, could your results be due to a change in the way the formula is weighted. It seems to me one of the reasons the exact formula and values might be kept secret is Upwork then has the ability to make tweaks and changes with each update.

I appreciate the positive feedback.

 

@ Cristian
Your point is well received. I will no longer consider it a failure since we actually know alot more about the metric in general.

 

@ Patrick
I know long-term contracts weigh heavily on the formula in a positive way.

 

Because of the many uncertainties, I started with something basic hoping to improve on it. What had been working for some time (and with a 'simple' profile) was:

 

completed jobs / n

where jobs:
successfully completed = 1
neutral = 0.5
not successful = 0
n = total jobs

Based on your suggestion it could be possible that there is now a gradient between 5 & 7.

However that would be impossible to prove, since any combination of numbers could be ballparked to give the right score in a test.

 

I believe that it has become more complicated and perhaps a job is no longer one variable in the formula; but multiple individual components.

---- easy like Sunday morning ----
suznee
Member

Setu,

 

I am not great with stats. Math was not a strong point in college for me. Although stats was. Problem with the JS is we don't have the unknown factors to contribute so the numbers will only show a portion of what really affects the JS.

 

I have bee watching mine the past couple months. I have had 3 jobs close over the last 3 two week spans. My public feedback with 5 star and good comments on all 3 jobs. I don't know numbers on the first contract closed but knew it would drop my score. I am sure you remember my post on the job that changed the description and parameters once I got the job. It did drop my score. Closing of first one of the three dropped my score from 100% to 94%,

 

Taking one new client for some reason raised it to 95%, Second to last client I discussed in detail what he left me privately. He left a 7 on Recommending me and not sure on the rest but it dropped my recommended score from 80% to 63%, and my JS to 93%. Last client I know the feedback was good public and private and it had been a long term (although Krisztina said something in another thread about long term having no impact when it was closed.

 

My JS today did not change at all. Recommended stayed at 63% and my JS stayed at 93%. There was absolutely no change with the 3rd closed contract with good feedback public and private.


Their system makes me wonder if factors calculated in do change to fit their needs.

 

It makes absolutely no sense that a JS would not change when it was closed since the other 2 did change and almost immediately after the contracts closed. When the last one closed nothing changed.

Suzanne,

I must admit that it is far more complex for freelancers with many long term clients and few jobs like yourself. Also the point about unknown variables is a fact.

What I know is that long term contracts are special because they are "counted more than once." That's as much as I can explain. This is why the Mods always say it counts as positive.

For instance, consider if you had 4 long term contracts in a two year span, like you do. I can easily consider those counting as 8 contracts in the formula.

People with low contract count suffer greatly due to the law of averages, and this has been evident throughout the platform. If your calculation was conventional, you had 4 great contracts, then the suspect one. That would leave you reeling at 90% (neutral feedback) and 80% (not successful feedback). However I believe your long term contracts cushioned that effect by "making up numbers."

But these are all the reasons / possibilities why I said it is a futile task, to try and figure out the formula.
---- easy like Sunday morning ----

Setu,


Thanks for the reply. I still do not understand why my JS would have stayed the same after a contract (long term) closed postitively. My metrics did not change my score did not change, and I understand my long term clients most likely cushioned the affects of the other scores, but it still makes no sense that stats nor JS did not change even by 1 on the last closure.

It's kinda a waste of time to try to unravel algorithms where you don't know the factors, they likely change during updates, and all you have to go off of is 1 data point -- your own profile.

 

Is the score relative to others? That alone would make it a complete wot since you can't track how others are doing.

kugrin
Member

yup


@Krisztina U wrote:

yup


 

lol! This is getting reposted. Excellent. 

Perhaps Upwork is taking a stab at using fuzzy logic.

Kristina I think the more appropriate defination of Algorithm now would be:

 

 A feature coded by Upwork Engineers which does not work the way it is suppose to work and is portrayed by Mods as a true implementation of what they were trying to accomplish. 

I have read through the threads here and smiled.

 

I am a qualififed assessor and moderator. What this means is that I am qualified to do assessments on job performance (and this is done according to a set list of criteria which is available to the person being assessed), and I am also qualified to moderate...meaning I am qualified to, in the case of a dispute, asses the assessment as done by the assessor and, in conjunction with both, reach a conclusion, i.e. whether the original assessment is a true reflection of the person's work performance, and whether the person doing the assessment was in any way biased. 

 

The key, here, in ALL instances, is transparancy. The reason for this is that it affords the person who underwent the initial assessment the opportunity of 'fixing' their mistakes, if applicable. 

 

The JS is, in effect, an assessment. And it is not transparant.

 

.........................................that's all folks ๐Ÿ™‚ My two cents worth.

Irene,

Since you are a professional assessor, perhaps we can use you in the ongoing fight to have more transparency here.
๐Ÿ™‚
---- easy like Sunday morning ----