I found someone posting information about the maintanance here on another site. Here is what he/she had to say:
"In short, they are migrating to a new, modern, and micro-service oriented platform. So, some rough time is expected. Probably months of it.
Their new platform is decent though, and after the transition is complete it should be much more stable. Their current state of affairs sometimes closely resembles a textbook example of a cluster**bleep**, but this is expected because of the transition. Oh well, on the other hand it was probably not expected by upper management. As usual. Still, financial information is being processed by a separate subsystem, which is not likely to get seriously affected by all this happening.
Engineering team is also expanding and reorganizing quickly, and not in a bad direction, key people are quite good."
When asked about the situation yesterday, when people where logged in into different profiles he replied:
"What happened was that either engineering team **bleep**ed up, or someone from outside found a way cripple Upwork via their API, and engineering were unable to respond fast enough. Even while Upwork is using CloudFlare as their DDoS (denial of service attacks) protection, someone could for example pull information about jobs from Upwork website (crawl all freelancers open profiles or job postings to use them in some application or database) too fast for Upwork to work properly, and this kind of problem could only be fixed by developers making appropriate changes to the code (throttle requests after threshold, more aggressive caching, optimizing code, etc), because otherwise it will interfere with normal usage of the site. Those requests can't be just blocked by firewall at network level.
Such changes are made on the fly and deployed as incident goes, that means errors happen a lot.
So as an attempt to mitigate this issue, caching settings probably were tweaked to use caching more aggressively, in an attempt to lower load on application and database servers. This could break cache invalidation mechanism, it's when website makes sure that cache is not expired, i.e. real data has not changed. In the end this caused cached data from some users to appear on other users pages.
About the scope of the impact, lots of info could be cached like this. Messages with clients should not have be in that cache because it's a separate subsystem (their chat system is mostly separated from main website). Though in the attempted to mitigate impact I can't completely exclude that someone has not found a way to screw this up and cache messages too. Read: this is improbable but not completely impossible.
As I know banking information is not cached at all at the level where the problem occurred, and thus couldn't have been leaked."
I found it more useful than the official explanations. Maybe it will help someone else here as well.
@Karolis K wrote:
Their new platform is decent though, and after the transition is complete it should be much more stable.
Thanks for posting, and yes you can link, worst comes to worst a mod deletes the link. That said, we've been hearing the bigger, better, more stable song for a whooping two years now. It's the slowest merger ever. By the time this platform is "stable" it won't have any customers left.
In that case here is the link:
Doesn't seem like discussion there is active, but hopefuly we can ask more questions.
@Karolis K wrote:
Their current state of affairs sometimes closely resembles a textbook example of a cluster**bleep**, but this is expected because of the transition.
Only of the transition is poorly planned. I've managed such things before, and there's really no reason it has to become a SNAFU.
Engineering team is also expanding and reorganizing quickly, and not in a bad direction, key people are quite good.
All evidence is to the contrary.
Those requests can't be just blocked by firewall at network level.
Yes, they can. While interesting, whoever is feeding you this information doesn't seems to be particularly experienced.
Thanks for this information. Some of it sounds plausible amd other bits a little far fetched but that's Reddit for you. But you are right it is more helpful than the official information.
Learn more about our awesome Community member Lisa!Learn More
Virtual Talent Bench enables you to easily discover and connect with talent. Learn more about building custom lists of talent, adding tags, notes, and more to move your business forward.Learn More
Loom addition in messages provides more ways to easily communicate and share information on Upwork!Learn More
Beginning Your Freelancing Journey on UpworkLearn More