I found this article at the top of the Upwork Support "Get Started" section. Like all of the articles, there is no Post date (if there is a date in the metadata of the page source, my cursory check missed it).
About The Study
What are your thoughts? If this is a recent article, then Upwork is making small, unobtrusive study of the merits of Boosting. Although, the placebo seems to only be from the FL side, based on the statistics that they published (next section) and this statement:
This placebo feature allows us to monitor the average benefits users can expect from boosting on an ongoing basis, helping us build and improve its usefulness.
An important clarification - the users in question are specifically those freelancers who outbid everyone else for the advertising slots.
PUBLISHED STATS!!! 😲
Our beta testing of boosted proposals revealed that the feature can be helpful, with the highest bidders who occupied the three boosted slots collectively had a 37% higher chance of being hired, with the highest bidder enjoying a 43% boost in their chance of being hired. (Italics mine.)
So, taking this statement at face value (and again, assuming that this is a recent article), the advertisement process dramatically increases the proposal success rate for freelancers. But on further inspection, the gains are probably nowhere near as high as the stat implies, because the even 143% of a small chance is still small.
Improved Chances - My Analysis
A 43% or 37% improvement sounds like a lot. And, well, it is. But it is easy to overstate/overimagine the practical effect that frequent boosting will have. I remind you of three facts:
- 143% (or 137%) of a small amount is still a small amount.
- The effect is distributed over many proposals.
- The stat is an average from many freelancers with varying quality of proposals, profiles, and work histories.
Let us assume that each job posting receives 50 equally qualified proposals, each job results in exactly 1 hire, and that all clients review 100% of the proposals submitted prior to hiring. In this scenario, each proposal has a default 2% chance of converting into a paid gig. Enter Boosting. The owner of the top slot gets a 43% improvement in their conversion chance, resulting in a final chance of (0.02 * 1.43) * 100% = 2.86%. On average, getting a boosted slot improves your chance by 37%, resulting in (0.02 * 1.37) * 100% = 2.74%. For the remaining 47 proposals, their respective chances diminish to (1 - 0.0274 * 3) / 47 * 100% = 1.95%. In this hypothetical, that is an individual chance reduction of 2.5% for the organic and outbid proposals.
Statistical effects are best understood as distributed effects. The impact is really only noticeable across many iterations. So, if I were to boost under the hypothetical situation that I described above, over the course of 1000 boosted proposals, I can expect to see 7.4 more conversions than if I elected to not boost. Of course, the resulting revenue can be all over the place, influenced by many factors. Whether or not the increased revenue will offset the cost of the Connects is a calculated risk assessment.
And of course, as point 3 should remind you, your Boosting improvement stat is not guaranteed to be average - it could very well be less.
What the Study Should Really Inspect
So, the study is focused on the benefit of Boosting for the freelancers who boost (and indirectly, on the opportunity cost of electing to not boost). Which is all well and good for the freelancer side of the platform. But the study - crucially - ignores the effects of Boosting on the client side. There are three questions that are important - and one of them is vital for Upwork's bottom line! I've listed them with my respective null hypotheses.
- What are the quality differences (measured by interviews/hires) between boosted and organic proposals? (Null hypothesis: the choice to boost a proposal has no relationship to the quality of the proposal, such that viewed boosted proposals yield the same rate of interviews and hires as viewed organic proposals.)
- What effect does the presence of boosted proposals have on the client's diligence (measured by percentage of proposals viewed)? (Null hypothesis: clients view the same percentage of submitted proposals when Boosting is in effect as when all proposals are presented organically.)
- What effect does the presence of boosted proposals have on the client's hiring decisions? (Null hypothesis: clients hire at the same rate and at the same prices when Boosting is in effect as when all proposals are presented organically.)
So the null hypotheses are just that - expectations that Boosting has no correlation. The actual correlations could be found to be positive or negative.
Summary
I am eager for more information on this placebo study. I think that the study parameters and goals, as announced, have tunnel vision. The study sample ought to be much larger, because the existing data analysis shows that the effect is pretty paltry, so there need to be many data points to clearly identify correlations. And since the practical effect is so small, affected freelancers won't notice the difference. Affected clients, on the other hand, will likely rejoice at the removal of the boosts' obstruction of their organizing efforts.
(Edited for formatting)
