Reply
cancel
Showing results for 
Search instead for 
Did you mean: 
Reply

Re: "Interesting Job" Feature and Improving Recommendations

Employee
Tam L Employee Member Since: Aug 31, 2015
1 of 40

Hi every one!

 

We’ve received feedback that our job recommendation algorithm could use some improvement. 

 

We hear you, and definitely don't disagree. That said, we’re hoping to make these improvements as soon as we can, and in order to do so we’ll be conducting a small test on the Job Search results page: using your clicks/not-clicks as a feedback mechanism that will allow for us to analyze & fine-tune the algorithm.

 

“What are the details of this test?”

 

  • It's pretty straight foward. In the upcoming weeks, some of you may see a job with the label “Interesting Job” as the first result of your search. This job is one that you would have seen naturally in your regular search results, and we are highlighting said job because well… we think it might be applicable to you (which begs the question, why don’t we call it applicable job? I don’t know). Labeling aside, the job isn't any more special than any other job in your search – the label is there to help my team track your interaction with that job.

 

Screen Shot 2016-02-18 at 1.56.19 PM.png

 

 

  • As mentioned earlier, interacting with that particular job tells us a lot. If you apply to said job, we know it’s a good recommendation. If you don't interact with the job at all, then we know it's not a good recommendation.
  • Very important note: if you apply to the 'interesting' job, you do NOT receive any preferential treatment amongst the other applicants. Conversely, if you do NOT apply to the 'interesting' job shown, you are not penalized in any way.
  • I want to be really clear about this: applying/not applying to the job does not affect your profile/JSS/candidacy in any shape, way, or form. 

 

“Can I opt in/out of the test?”

 

Not this time around. In designing this feature, we’ve completely randomized who is or isn’t a part of it. With that said, the test is also designed to not interfere with your normal usage. Again, the 'interesting job' is one that already appears in your search results and at most, you may see one ‘interesting job’ per search. That’s it.

 

And again, whether or not you click/apply to the ‘interesting job’ has no effect on your profile or your application.

 

“Tam! I have X concern!”

 

No worries, if you have any questions/comments/criticisms please feel let me know. I’ll respond to what I can (unless you’re just trying to troll me, which I don’t mind too much Smiley Wink ).

 

Appreciatively,

tam



P.S. - I hope every one is having a good Monday =)

Community Guru
Douglas Michael M Member Since: May 22, 2015
2 of 40

@Tam L wrote:

 

  • ....If you don't interact with the job at all, then we know it's not a good recommendation. 

Sorry, Tam, that's an unjustifiable inference. No interaction might mean:

  • I'm on vacation. (Are you controlling for availability?)
  • I'm coasting on invitations right now, and not even looking at the page, even if I occasionally accidentally land on it.
  • It's a "good" invitation, as opposed to "bad," but not worth a click for reasons that might be obvious in the preview, or might be utterly extraneous to the job.

It sounds as if you're chasing after the possibly unattainable "best," when what we need is "better."

 

Best,

Michael

 

p.s. If you've run this test by your research professionals, my estimation of them just dropped. If you didn't—why not?

 

 

Community Guru
Olga Q Member Since: Sep 5, 2012
3 of 40

Douglas Michael M wrote:

[…] 

Sorry, Tam, that's an unjustifiable inference. No interaction might mean:

  • I'm on vacation. (Are you controlling for availability?)
  • I'm coasting on invitations right now, and not even looking at the page, even if I occasionally accidentally land on it.

[…]


 I agree with Michael.

By completely randomizing who is or isn’t a part of it how can you be absolutely sure that lack of interaction isn't related with availability? It's been around two months since I last searched for jobs - imagine I'm part of this test: you'd conclude that I didn't interact with 'Interesting jobs' at all when the real reason was that I wasn't searching in the first place.

Employee
Tam L Employee Member Since: Aug 31, 2015
4 of 40

Hi Michael - 

 

Does it still count as vacation if you're working? Some spouses/partners would argue no =P

 

More seriously, good questions/points as always.

 

To address your last statement first - chasing best vs getting better - we're definitely in the getting better camp. I/we understand that creating a recommender takes time, and even the 'best' recommenders still need work. I mean, Netflix keeps recommending me movies I've already watched, and for whatever reason Amazon Video has a horror movie under the children's suggestions.

 

I may have oversimplified, so allow me to um... unsimplify(?). 

 

Re availability: not directly. There are two signals that an algorigthm is 'better' (to me at least): if the jobs shown are clicked through more; if the jobs that are clicked are actually applied to. I/we view them as two separate measurements. Thus, if in your scenario you're on vacation and skip over the posting, that tells me one thing: for whatever reason you are not clicking the job, and I need to figure out what that reason is. If you click the job and don't apply, again, I need to figure out why. And so forth. Which metric is a better indicator of interest? I/we don't know yet (we all have our own theories though!).

 

Sidenote: I'm a big believer in understanding WHY things happen. I'm not as smart as you all, or as some of my colleagues, but I do take time to really investigate and dig into things. 

 

Some other details that I believe help mitigate the impact of your scenarios:

 

 

  • Large sample size - I'd agree if we had a small sample, your vacation/coasting scenarios would have a significant impact
  • Control group - because the 'interesting job' is always a job from your search results, this allows us to see if that job is 'interesting' regardless of it being called out; we can compare across groups
  • We are currently looking for aggregate level indicators - a simple example: do more people click projects with large budgets? If so, can we agree that budget plus some other job posting value is important to you all? 

To sum up, I may have presented this test/feature as binary, but it's really not. 

 

I hope I provided you with the detail you're looking for. If not, I'll try again. 

 

Thanks!

 

P.S. I am working with one other person on this, who, in my mind, is quite brilliant. That said, we both love to learn and accept your criticism. However, we do have a counter question (for learning sake), given the nature of the site, and a desire to remain transparent, how could we have approached this?

 

Double P.S. Olga - I saw your question. In your scenario, if you never run a search, then you never see an 'interesting job'. In which case, nothing is recorded at all. The job is only shown, and metrics only accounted for, if you run a search.

Community Guru
Douglas Michael M Member Since: May 22, 2015
5 of 40

Thanks, Tam, for your thoughtful reply.

You've reassured me on two points:

  • You are not looking at lack of response in a binary way.
  • You are not sacrificing the better on the altar of the best.

The kind of research I do is unrelated to population-based data gathering, about which I know only odds and ends I've picked up along the way; so no counterproposal, just the general caution—which you and your colleague seem aware of—to be careful about inferences.

Best,
Michael

Community Guru
Gyan D Member Since: Sep 4, 2014
6 of 40

Wouldn't it be more efficient if the participants had a channel to convey their assessment?  That is, instead of trying to infer mental states from an aggregate data set, how about a drop-down or pop-up next to each 'Interesting Job'  badge where the user gets to check one or more feedback options - like the flagging facility - and maybe even a comment box. 

Employee
Tam L Employee Member Since: Aug 31, 2015
7 of 40

Hey Gyan - 

 

Agreed, it would be WAY more efficient. 

 

One major hurdle though...

 

That thumbs down thing we have (similar to what you're suggesting): no one uses it. People are afraid that the data would be used against them negatively. E.g. I tell you I don't like this job because it has low pay, so now you make broad assumptions that I don't like any low paying jobs. 

 

Additionally, we didn't want people to have to go out of the way and take extra actions. As MIchael pointed out, best is sometimes the enemy of better, and for 'better' sake we're acknowledging that particular flaw in what we're currently doing. 

 

Hmmm... you've got me thinking about future things now =)

 

Community Guru
Douglas Michael M Member Since: May 22, 2015
8 of 40

@Tam L wrote:

 

....That thumbs down thing we have (similar to what you're suggesting): no one uses it. People are afraid that the data would be used against them negatively. E.g. I tell you I don't like this job because it has low pay, so now you make broad assumptions that I don't like any low paying jobs.... 

 

Hmmm... you've got me thinking about future things now =)

 


Tam,

 

Some of us rely on saved job searches, where the thumbs down option is not visible. I wasn't aware of its existence till it was mentioned in the forum. Again, you're taking a fact (low use) and constructing a reason out of whole cloth.

 

Your example actually illustrates exactly how I would expect such data to be used, positively, for me. The sticking point would be how an algorithm determines what is a low-paying job, which is context-dependent: A small fixed-price job (low absolute dollars, possibly well-paying) is not the same as a job offering apparently high absolute dollars for a disproportionately high amount or kind of work (= low-paying).

 

Keep thinking! B^)

 

And seriously, research professionals can explain a lot about how the answers you get are determined by the questions you ask. Upwork has access to these—whether internally or externally I don't know. So far this dialog suggests you could use some further help along those lines.

 

Best,

Michael

Highlighted
Community Guru
Jess C Member Since: Feb 18, 2015
9 of 40

The thumbs down thing is only visible in certain versions of search. I would love to be able to use it to dismiss most of the jobs that come up in the "default" feed when I click on "Look for Work" because 99.9% of them are irrelevant to my skills - but the feature isn't available on that page! It's also not available on an individual job post, which is where most of the options would be more pertinent - I have to go back to the search list to rate a certain job, unless I want to flag it as "inappropriate," which isn't all that helpful when it's just an issue of unrealistic expectations by the client or a completely vague description.

 

The place where recommendations really need improvement is invitations. Search would be better improved by honing the categories better, and allowing us to flag jobs that don't belong in certain categories.

Community Guru
Olga Q Member Since: Sep 5, 2012
10 of 40

I only see the 'Thumbs down' feature when searching in categories. Frankly, I never used it because I'm interested in finding jobs that fit my skills, not in evaluating job posts... and, most importantly, I think some things are just silly to ask considering that we get to read 3 or 4 lines and that most of the info is only available if we click to see the job posting!!

 

Screen Shot 2016-02-24 at 20.18.24.png

 

 

 

TOP SOLUTION AUTHORS
TOP KUDOED MEMBERS