🐈
» Forums » Freelancers » Re: Placebo Boosts?
Page options
25005175
Community Member

Placebo Boosts?

I found this article at the top of the Upwork Support "Get Started" section. Like all of the articles, there is no Post date (if there is a date in the metadata of the page source, my cursory check missed it).

 

About The Study

What are your thoughts? If this is a recent article, then Upwork is making small, unobtrusive study of the merits of Boosting.  Although, the placebo seems to only be from the FL side, based on the statistics that they published (next section) and this statement:

This placebo feature allows us to monitor the average benefits users can expect from boosting on an ongoing basis, helping us build and improve its usefulness.

An important clarification - the users in question are specifically those freelancers who outbid everyone else for the advertising slots.

 

PUBLISHED STATS!!! 😲

Our beta testing of boosted proposals revealed that the feature can be helpful, with the highest bidders who occupied the three boosted slots collectively had a 37% higher chance of being hired, with the highest bidder enjoying a 43% boost in their chance of being hired. (Italics mine.)

 

So, taking this statement at face value (and again, assuming that this is a recent article), the advertisement process dramatically increases the proposal success rate for freelancers. But on further inspection, the gains are probably nowhere near as high as the stat implies, because the even 143% of a small chance is still small.

 

Improved Chances - My Analysis

A 43% or 37% improvement sounds like a lot. And, well, it is. But it is easy to overstate/overimagine the practical effect that frequent boosting will have. I remind you of three facts:

 

  1. 143% (or 137%) of a small amount is still a small amount.
  2. The effect is distributed over many proposals.
  3. The stat is an average from many freelancers with varying quality of proposals, profiles, and work histories.

Let us assume that each job posting receives 50 equally qualified proposals, each job results in exactly 1 hire, and that all clients review 100% of the proposals submitted prior to hiring. In this scenario, each proposal has a default 2% chance of converting into a paid gig. Enter Boosting. The owner of the top slot gets a 43% improvement in their conversion chance, resulting in a final chance of (0.02 * 1.43) * 100% = 2.86%. On average, getting a boosted slot improves your chance by 37%, resulting in (0.02 * 1.37) * 100% = 2.74%. For the remaining 47 proposals, their respective chances diminish to (1 - 0.0274 * 3) / 47 * 100% = 1.95%. In this hypothetical, that is an individual chance reduction of 2.5% for the organic and outbid proposals.

 

Statistical effects are best understood as distributed effects. The impact is really only noticeable across many iterations. So, if I were to boost under the hypothetical situation that I described above, over the course of 1000 boosted proposals, I can expect to see 7.4 more conversions than if I elected to not boost. Of course, the resulting revenue can be all over the place, influenced by many factors. Whether or not the increased revenue will offset the cost of the Connects is a calculated risk assessment.

 

And of course, as point 3 should remind you, your Boosting improvement stat is not guaranteed to be average - it could very well be less.

 

What the Study Should Really Inspect

So, the study is focused on the benefit of Boosting for the freelancers who boost (and indirectly, on the opportunity cost of electing to not boost). Which is all well and good for the freelancer side of the platform. But the study - crucially - ignores the effects of Boosting on the client side. There are three questions that are important - and one of them is vital for Upwork's bottom line! I've listed them with my respective null hypotheses.

  1. What are the quality differences (measured by interviews/hires) between boosted and organic proposals? (Null hypothesis: the choice to boost a proposal has no relationship to the quality of the proposal, such that viewed boosted proposals yield the same rate of interviews and hires as viewed organic proposals.)
  2. What effect does the presence of boosted proposals have on the client's diligence (measured by percentage of proposals viewed)? (Null hypothesis: clients view the same percentage of submitted proposals when Boosting is in effect as when all proposals are presented organically.)
  3. What effect does the presence of boosted proposals have on the client's hiring decisions? (Null hypothesis: clients hire at the same rate and at the same prices when Boosting is in effect as when all proposals are presented organically.)

So the null hypotheses are just that - expectations that Boosting has no correlation. The actual correlations could be found to be positive or negative.

 

Summary

I am eager for more information on this placebo study. I think that the study parameters and goals, as announced, have tunnel vision. The study sample ought to be much larger, because the existing data analysis shows that the effect is pretty paltry, so there need to be many data points to clearly identify correlations. And since the practical effect is so small, affected freelancers won't notice the difference. Affected clients, on the other hand, will likely rejoice at the removal of the boosts' obstruction of their organizing efforts. 

 

(Edited for formatting)

ACCEPTED SOLUTION
25005175
Community Member

Updating this thread with information on the age of the published statistics. I don't know why it took me so long to identify this, since the information trail was plainly evident, thanks to the history of the Product Release Notes that Christine linked earlier.

 

The 43% and 37% improvement statistics, published 2022-03-22, are from the Boosting study that ran from 2021-09-08 until 2021-10-13.

 

Also, in a separate thread, Valeria K confirmed my analysis of the application of the 43% and 37% chance improvement.

View solution in original post

16 REPLIES 16
JoanneP
Moderator
Moderator

Hi Jonathan,

 

We continue testing our features to make sure they are successful tools. Our beta testing of boosted proposals revealed that the feature could be helpful, with the highest bidder enjoying a 43% boost in their chance of being hired. This placebo feature allows us to monitor the average benefits users can expect from boosting on an ongoing basis, helping us build and improve its usefulness.

~ Joanne
Upwork

If you wouldn't mind, what is the percentage of freelancers that boost to each spot (one, two, and three)?

I am mostly interested in learning the percentage of freelancers that boost for the number one slot.  Would you mind sharing that stat?  Thank you!

feed_my_eyes
Community Member


Jonathan L wrote:

I found this article at the top of the Upwork Support "Get Started" section. Like all of the articles, there is no Post date (if there is a date in the metadata of the page source, my cursory check missed it).

About The Study

What are your thoughts? If this is a recent article, then Upwork is making small, unobtrusive study of the merits of Boosting. 


It's not recent, because I first saw those statistics when Upwork made the announcement that the boost feature was coming back, on 22 March of this year. https://community.upwork.com/t5/Product-Release-Notes/Boost-Your-Proposal-to-the-Top-of-List/ba-p/10...

 

But - and this is important, I think - during the beta test of the boost feature, freelancers who boosted their proposals were being labelled "highly interested" in the project, whereas when Upwork re-introduced the feature, they also announced this change: "Second, we are changing the label from “Highly Interested” to “Boosted”." This was a good call, IMO, because telling clients that freelancers who were in the top three spots were more "interested" in the project than everyone else (and hadn't just paid to be there) was false advertising. But now I'd like to see more recent data, because I doubt that clients are as likely to be swayed by the "boosted" label as they were by the "highly interested" label. 

 

It's difficult to extract any meaningful conclusion from the stats, however, because I think it's pretty safe to say that freelancers who are struggling can't afford to boost their bids, so it's likely that the top three spots are freelancers who are fairly well-established and confident that they're not throwing their money away. (Anecdotal evidence suggests that this is the case.)

 

At the end of the day, don't we all need to make our own conclusions? I've been doing my own "study" by boosting about half of my bids, and I do find that clients are slightly more likely to view them, but it doesn't make a big enough difference for me to feel like I need to use it all the time. Who's to say that clients wouldn't have viewed my bid on those projects if I hadn't boosted? The only way to know for sure is if I were able to place one boosted bid and one non-boosted bid on every project and compare the results, but that's not possible. So, it doesn't make any difference if Upwork tells me that people are 43% more likely to be hired if they boost; if I don't get those kind of results myself, then I'm not going to bother.

 

What I do want to know is how clients feel about boosted proposals, and I'm sure that Upwork is monitoring this as well, but we'll probably only find out when and if the feature changes or disappears.

Considering that I started in May, I completely accept that I may be late in finding this information. I just don't recall seeing this article when I was first learning.

w62n
Community Member

survey aside, the whole "boost proposal" thing is COMPLETELY UNETHICAL. Upwork already double (if not triple) dips into it's freelancers pockets. 

25005175
Community Member

In exchange for providing this marketplace where freelancers can find gigs. Same as every other freelance marketplace - except that Upwork's fees are less if you have repeat clients, which incentivizes freelancers to do great work to get clients to keep coming back and spending money.

 

Seems a fair trade to me.

adhanet_
Community Member

It looks like gambling which is forbidden in many religions. I personally couldn't boost connects in a conscious way, I tried it a couple of times but it's a strange secretive thing that makes me feel confused.

I've never seen/heard auctions described as gambling before today. A boosted spot is strictly an advertisement - boosted slots are shown at the top of the client's applicant list.

An auction is when you see the amount that people are bidding. With this you have to guess an amount to boost, to me that seems like gambling. 

to me that seems like gambling. 

 

Exactly.

2 problems with that assessment.

  1. When gambling, your money is lost if you lose the gamble. With an auction, you don't pay if you lose.
  2. Previously, boosting was a blind auction (which still isn't gambling). For a few months now, it has been a visible auction. You always get to see what quantity of Connects currently hold the top 3 spots.

In a sense, placing any bid on Upwork is gambling, because there's no guarantee that you'll get a job. Why would boosting be more wrong or more of a gamble?

In a sense, placing any bid on Upwork is gambling, because there's no guarantee that you'll get a job.

Excellent point! Any proposal requires a time down-payment, non-refundable. From that perspective, the boosting auction is less of a gamble than bidding on a job in the first place!

6bfcdaf8
Community Member

Why don't you purchase a few services as a client, create a few job posts and decide for yourself if you find boosting feature useful? I personally did, and i couldn't care less about who boosts. 

If it's good for you than that's great. I'm not against it, I was just making the point that a lot of people are getting their services cheaper by hiring in developing countries that are more religious. 

I personally can't boost cause it doesn't make any sense to me.  I'm even more focused on different sources like an agency as a middle man or to be a middle man, then I can take time to understand boosting, bidding, etc better.

 

I have tried to purchase a product. It didn't feel secure.

25005175
Community Member

Updating this thread with information on the age of the published statistics. I don't know why it took me so long to identify this, since the information trail was plainly evident, thanks to the history of the Product Release Notes that Christine linked earlier.

 

The 43% and 37% improvement statistics, published 2022-03-22, are from the Boosting study that ran from 2021-09-08 until 2021-10-13.

 

Also, in a separate thread, Valeria K confirmed my analysis of the application of the 43% and 37% chance improvement.

Latest Articles
Top Upvoted Members