Reply
cancel
Showing results for 
Search instead for 
Did you mean: 
Reply

Some info regarding the state of upwork.

kkpmw
Ace Contributor
Karolis K Member Since: Sep 5, 2015
1 of 9

I found someone posting information about the maintanance here on another site. Here is what he/she had to say:

 

 

 

"In short, they are migrating to a new, modern, and micro-service oriented platform. So, some rough time is expected. Probably months of it.

Their new platform is decent though, and after the transition is complete it should be much more stable. Their current state of affairs sometimes closely resembles a textbook example of a cluster**bleep**, but this is expected because of the transition. Oh well, on the other hand it was probably not expected by upper management. As usual. Still, financial information is being processed by a separate subsystem, which is not likely to get seriously affected by all this happening.

Engineering team is also expanding and reorganizing quickly, and not in a bad direction, key people are quite good."

When asked about the situation yesterday, when people where logged in into different profiles he replied:

 

"What happened was that either engineering team **bleep**ed up, or someone from outside found a way cripple Upwork via their API, and engineering were unable to respond fast enough. Even while Upwork is using CloudFlare as their DDoS (denial of service attacks) protection, someone could for example pull information about jobs from Upwork website (crawl all freelancers open profiles or job postings to use them in some application or database) too fast for Upwork to work properly, and this kind of problem could only be fixed by developers making appropriate changes to the code (throttle requests after threshold, more aggressive caching, optimizing code, etc), because otherwise it will interfere with normal usage of the site. Those requests can't be just blocked by firewall at network level.

Such changes are made on the fly and deployed as incident goes, that means errors happen a lot.

So as an attempt to mitigate this issue, caching settings probably were tweaked to use caching more aggressively, in an attempt to lower load on application and database servers. This could break cache invalidation mechanism, it's when website makes sure that cache is not expired, i.e. real data has not changed. In the end this caused cached data from some users to appear on other users pages.

About the scope of the impact, lots of info could be cached like this. Messages with clients should not have be in that cache because it's a separate subsystem (their chat system is mostly separated from main website). Though in the attempted to mitigate impact I can't completely exclude that someone has not found a way to screw this up and cache messages too. Read: this is improbable but not completely impossible.

As I know banking information is not cached at all at the level where the problem occurred, and thus couldn't have been leaked."

I found it more useful than the official explanations. Maybe it will help someone else here as well.

firescue17
Community Guru
Steven E. L Member Since: May 5, 2015
2 of 9

What is the source?

kkpmw
Ace Contributor
Karolis K Member Since: Sep 5, 2015
3 of 9

I found it on reddit. I am not really sure about the TOS. Can I post a link to another website here?

kugrin
Community Guru
Krisztina U Member Since: Aug 7, 2009
4 of 9

@Karolis K wrote:

 

Their new platform is decent though, and after the transition is complete it should be much more stable. 


Thanks for posting, and yes you can link, worst comes to worst a mod deletes the link. That said, we've been hearing the bigger, better, more stable song for a whooping two years now. It's the slowest merger ever. By the time this platform is "stable" it won't have any customers left.

kkpmw
Ace Contributor
Karolis K Member Since: Sep 5, 2015
5 of 9

In that case here is the link:

https://www.reddit.com/r/freelance/comments/3k7ai2/upwork_has_been_down_a_lot_lately_i_know_some/

Doesn't seem like discussion there is active, but hopefuly we can ask more questions.

lanwanman
Community Guru
Ronald T Member Since: Sep 14, 2009
6 of 9

Thank you for the information Karolis!

Ron aka LanWanMan
droleary
Community Guru
Darrin O Member Since: Jan 20, 2015
7 of 9

@Karolis K wrote:

Their current state of affairs sometimes closely resembles a textbook example of a cluster**bleep**, but this is expected because of the transition.


Only of the transition is poorly planned.  I've managed such things before, and there's really no reason it has to become a SNAFU.


Engineering team is also expanding and reorganizing quickly, and not in a bad direction, key people are quite good.


All evidence is to the contrary.


Those requests can't be just blocked by firewall at network level.


Yes, they can.  While interesting, whoever is feeding you this information doesn't seems to be particularly experienced.

 

digitalant
Active Member
Manish C Member Since: Sep 9, 2015
8 of 9

Thanks for this information. Some of it sounds plausible amd other bits a little far fetched but that's Reddit for you. But you are right it is more helpful than the official information. 

jsutherland
Community Guru
Jean S Member Since: Oct 22, 2007
9 of 9

Reddit is always the place to go for more information.

 

It's just stunning that the CEO just ignores cliens and freelancers **edited for Community Guidelines**

TOP SOLUTION AUTHORS
TOP KUDOED MEMBERS