🐈 Community
spiderwisp
Member

Inconsistent JS Metrics

This may have already been discussed here, but have a look at the following metrics. One is my Upwork profile and the other Elance. They do not appear to be consistent whatsoever. 

On Elance I was recently in the top 50, and currently ranked 213 (*Out of 329,198 individuals). On Upwork I'm not even top-rated.

This is a 12-month shanpshot on Elance:

 

elance-profile.jpg

And of course Upwork's JS metric

 

upwork-profile.jpg

23 REPLIES 23
spiderwisp
Member

I do quite a bit of work with algorithms, data collection and analysis, and these metrics are just not accurate.

1+1 doesn't equal 3. No matter how you do the math.

I'm sure this JS idea was a good one... keep the data private so freelancers/clients wouldn't be able to exploit it. I'm sure the idea was to better measure freelancer's performance in the hopes of improving the platform.

But, it's not working!

 

I'm sure there's a room full of engineers declaring their calculations are correct, probably some suits in another office brainstorming ways to improve the JS metric.

But, it's a horrible idea.

 

Get rid of it.

 

Whatever perceived benefit to the JS metric said suits and engineers are rooted in, trust me when I tell you the loss of loyalty and the impact it's having on the platform's users far outweighs the possible benefits.

I am agree with you .Man Indifferent

"One thing I know, that I know nothing. This is the source of my wisdom"
suznee
Member

Tim,

 

I am not familar with Elance's metrics, but it looks like it was based on the star system, which the Job Success Score is not based on the star system. Previously it was based on just public feedback and what the client rated you publically here. Now it is based heavily on private feedback, jobs complete, refunds given, jobs not completed. If you had a lot of jobs that were refunded or not completed it counts against you. If you had feedback less then 9-10 on the questions ask privately it affects your score.

 

Looking at your profile you have a lot of jobs with no feedback that counts against you as well.

 

Mind you I am not for the JS just explaining to you on JS vs what it looks like Elance based it on the star rating systems.

 

Sadly many top rated have dropped to below top rated status because of private feedback which weighs heavily.

Good points, Suzanne. However, here's why Elancers are seeing strange stats on their Upwork profiles:

  • Rating/Feedback is not required after finishing the job on Elance.
  • Rating/Feedback for repeating clients after two jobs do not have any impact on profiles on Elance. This is why most repeating clients stop providing feedback because it has no effect.
  • Clients & Contractors won't be able to rate each other after 2 months from finishing the job. So asking your old clients to rate you on old jobs again on Elance is not practical or possible for all jobs (besides of course not being professional).
  • On Elance, cancelled jobs are not necessary a bad sign and can be a result of the client needing to simply cancel out a job, or close out one job to open a second one (happened to me couple of times).

I hope this helps Elancers. I'm ranked too high on Elance (top 500 in the entire community & on all categories) but on Upwork I've got an 88% JS. We can't do anything about it. Either keep complaining about it (nothing will change btw) or work hard to finish more jobs on Upwork and improve your score.

 

I know it's not fair, but I decided to work hard and improve my score. Also, I wouldn't let a number to stand as a barrier from my success. I think a good proposal is the key to winning any job.

 

Best regards,

Wassim

"Previously it was based on just public feedback and what the client rated you publically here."

The recommend metric on Elance is based on private feedback.

christian1220
Member

Tim, have you completed any project on Upwork that might have lowered your score?

 

If not...

 

I understand the frustration, I would be unhappy too if I was told to merge my account and then see a drop like this.

 

However, don't make the assumption that Elance way to measure performance is the same with Upwork JSS. I mean, I never heard someone said "Hey, we have this algorithm from Elance and we are implementing it exactly the same on Upwork". Some things may have changed and yeah, it's annoying when one platform shows a performance score and other shows a different score for the same work!!!

 

From my own experience I can say that increasing JSS is not such a big deal ( not saying it's super easy though ), if that's what you really want. 

 

 

 

 

 

"Tim, have you completed any project on Upwork that might have lowered your score?"

I have not. I have one ongoing job (a referral) which I just recently started.

kugrin
Member

The JSS also only takes into account the last 12-24 months. So if you did good work for 5 years and received good ratings, yet took on a contract that spanned over two years and received a less than stellar rating, that's all Upwork will remember you for. A+ for loyalty.

Tim,

The recommend rate is discrepant, possibly due to some nuanced time-basis difference.

The star ratings are in accord. Elance rounds to one decimal place, Upwork two.

None of the other metrics correspond. There cannot be discrepancy where there is no basis of comparison,

"None of the other metrics correspond. There cannot be discrepancy where there is no basis of comparison,"

 

They may not correspond, but they are relative to Elance data... specifically long-term clients. In that regard I think repeat clients is a fair and useful measurement, perhaps more so than the method "long term" clients are calculated by.

Ultimately my point is this:

How does a freelancer ranked 213 out of over 300,000 freelancers on one platform not achieve "top rated" status on the new platform?

Which platform is more accurate? Am I to believe Elance's metrics have been wrong for as long as I can remember?

Or, are the new metrics skewed?

Both are accurate. It just becomes unfair when you bring a profile from Elance to Upwork when the contractor has few cancelled jobs & some other unrated job.

 

I have covered this before but the most important ones are:

 

  • Rating is optional on Elance but required on Upwork. 
  • Cancelled jobs do not affect your profile on Elance but on Upwork they do. 

 

Best regards, 

Wassim

 

I thought cancelled jobs do affect your rate on Elance? 

 

My last big gig fell off my 6 month rolling period and I dropped like a stone on Elance in the writing section. I do barely anything in IT and sit at 8. Still holding strong here. We'll see how long it lasts.

 

I'm definitely gonna try to leverage this blue badge and see if it helps.

Jennifer, 

 

It does when you receive a partial payment. This is when your client can leave public and private feedback. However, this have no impact on your level if you cancel the job without receiving money for that job. This has been confirmed to me by a Customer Support representative on Elance. 

 

Wassim 


@Wassim T wrote:

Jennifer, 

 

It does when you receive a partial payment. This is when your client can leave public and private feedback. However, this have no impact on your level if you cancel the job without receiving money for that job. This has been confirmed to me by a Customer Support representative on Elance. 

 

Wassim 


oh, interesting. Thanks for that. I always thought cancelled affected you. I never had a cancellation until this year! Even my dispute really didn't affect me much. What affected me was when my monthly big gigs started dropping off after I didn't work much on Elance and came back. 

 

But, I never thought levels mattered much on Elance. I'm the lowest I've ever been and still get nice invites.


@Tim H wrote:

....How does a freelancer ranked 213 out of over 300,000 freelancers on one platform not achieve "top rated" status on the new platform?


Elance rankings are relative against all contractors or companies. Upwork's Job Success score, and its dependent Top Rated status, is not. 



...Am I to believe Elance's metrics have been wrong for as long as I can remember?


Yes. Elance's metrics have always been skewed toward revenue volume. This was somewhat reformed when they started giving greater weight to successful outcomes—the theoretical basis of the Job Success score—and consequently lowered the ranking of a bunch of farmers, including some of their top earners. If you look at the remaining metrics used to calculate "points," you'll see that many if not most of them are still revenue/"productivity"-driven.


@Tim H wrote:


How does a freelancer ranked 213 out of over 300,000 freelancers on one platform not achieve "top rated" status on the new platform?


The same reason a lot of developers on Elance and even on Upwork don't cut it on Toptal.

Ron aka LanWanMan
lanwanman
Member

No offense Tim, but based on your Elance stats alone, if I were looking for a top notch programmer I doubt I would hire you. Only 13 jobs and only 9 reviews, where's the other 4. Only 981 hours. Your competing with programmers here on Upwork that have way over 5,000 hours; some over 10,000 hours. As far as Elance stats vs. Upwork stats, it's apples and oranges--two different worlds.

Ron aka LanWanMan

Ronald,

 

You'll get Tim angry, and have him reel off his staggering portfolio once again.

---- easy like Sunday morning ----

I would "comment," but then they would start deleting our posts again..lol....

Ron aka LanWanMan

Ronald, 

 

I think Tim is doing great. He's at level 11 so do not underestimate this. 

 

I've been doing really well recently and always swing between level 9 and 10.

 

I think I have over 60 jobs but Tim is doing better because of the quality of work he handles. I usually work on rush projects and this is why I have more jobs completed. However, Tim works on less projects with higher values which take more time to achieve and are generally harder to finish. 

 

Best regards, 

Wassim

Wassim, I appreciate your comments and explanation regarding the differences that can exist with regard to programmers / developers' success between individual freelancers on Elance. Actually, you illustrated my point as further discussed in this post.

 

Those differences between individual freelancers exist with groups of freelancers working in every individual job category; those differences become magnified when freelancers within each job category work across multiple freelancer platforms. Add to the mix that freelancers today work in a global economy, and outcomes become even more inconsistent or 'different.'

 

Why the differences? Why the conspicuously inconsistent outcomes? Simple, because there are no baselines or standards (or even governmental regulations) that those multiple freelancer platforms choose or must adhere to.

 

For instance, in contrast to the "freelance world," participants in the "corporate world" might adhere to international standards for "quality," which in most cases impose requirements to "establish and document" policies, procedures, and work instructions (how to's) for all functional areas within an organization. Certainly, procedures and work instructions defining how each individual organization meets or exceeds policy requirements might differ greatly.

 

Nonetheless, the outcomes must meet policy requirements--the requirements of the applicable standards. Hence, a baseline or baselines from which all outcomes can be ethically, effectively, and economically monitored and measured in a consistent manner.

 

Upwork has tried to establish workable, fair baselines for performance measurement (aka algorithms or "mad robots"). They have tried to measure outcomes for each individual freelancer across a wide array of job categories, hours worked, feedback and ratings, earnings, etc. Where Upwork has failed is that instead of simplifying the process they appear to have made everything unnecessarily complex; thus "fragmented" (The "PhD influence," perhaps?).

 

Differences...languages, cultural differences, political views, economies of scale, education, skills and abilities...opinions, biases, and humanity.... Whether we like it or not, we are all one, and we had better start respecting that every day, everywhere--freelancers worldwide working together.

 

Tim, are you a team player or a fierce competitor? We all make choices.... At the moment I can only think of one amenable answer to the question.

Ron aka LanWanMan

That's some awesome stuff, Ronald. If it was possible I would follow you.