Reply
cancel
Showing results for 
Search instead for 
Did you mean: 
Reply

Update on Recent Site Incident on Feb 11th 2019

Employee
Han Y Employee Member Since: Feb 5, 2016
1 of 38

Hi,

 

My name is Han Yuan, and I lead the engineers here at Upwork.  I want to let you know that an engineering error recently impacted a small subset of our customers.  Specifically, we discovered that some data from your Upwork account might have been visible to another Upwork customer during a two-hour period. We want to make sure you have the facts about what happened, what information was involved, and the steps we are taking to help protect you.

 

What happened?

 

On February 11, 2019, we discovered that around 0800 UTC, some customers were able to see data from other customers who happened to be logged in at the same time during a two-hour period.  As customers browsed around the site, they may have suddenly noticed that their sessions appeared as if they were logged in as someone else.

 

Immediately upon notification, our Site Reliability Engineers shut down our production platform.  The site was unavailable for four hours while we undertook the necessary measures to ensure that no further misconfiguration was possible.  In practical terms, this means that we had to remediate the issue, understand the impact and provide a solution.

 

In instances like this, we want to do everything possible to protect our customers. Therefore, we were prepared to leave the site down for as long as it took so that we had confidence that the problem did not appear again. We know our customers rely on our site and apologize for the unexpected downtime.

 

During this downtime, we learned that the root cause was due to a bug in our code which was triggered when we had an unexpected network failure.

 

Who does this impact and what information got shared?

 

In all, we believe that potentially up to 20,000 customers were impacted by this issue during a two-hour period.  However, active session data suggests that the number of customers affected could be fewer than 4,000 customers with many members of our community logging out immediately upon noticing the issue.  As I write this, we are continuing to parse through the large amount of data from the time period involved in order to further understand the impact.

 

The nature of the issue allowed freelancers, in theory, to view and edit each other's public profiles.  But we are still reviewing the logs to determine if any of this actually occurred and will reach out directly to anyone impacted.

 

For clients, they would have been able to view and edit job posts, edit and end contracts, run reports, and change notification settings. Again, we are determining if any of these actions actually took place and will reach out directly to any individuals who were impacted.

 

We also do not believe that customers were allowed to enter certain areas of the site, including access to read other customers’ messages.

 

We wanted to share this information so you are aware of what happened and can reach out if  you believe unauthorized activity has occurred on your account. If that is the case, please reach out to us at support.upwork.com.

 

What are we doing to improve and how are we moving forward?

 

We are taking the following actions:

 

  • Our architects and platform engineers are evaluating the necessary steps to modernize our authentication and session management infrastructure. We hope to be able to accomplish this soon.  
  • We have instrumented real-time alerting to notify us when an issue of this nature occurs so that we can reduce or mitigate the impact much more quickly in the unlikely event it happens again.
  • We continue to analyze, document and reach out to impacted customers to make this right.

 

Your information and privacy are important to us, and I want to apologize for any inconvenience this may have caused you. As always, if you have any questions or need any additional information, please do not hesitate to contact us at support.upwork.com.

Han Yuan
Community Guru
Tiffany S Member Since: Jan 15, 2016
2 of 38

Can you verify that you have sent the legally-required disclosures directly to affected users and have identified and acted upon any requirement that you send disclosures to government agencies such as the NY Attorney General?

Community Manager
Lena E Community Manager Member Since: Apr 7, 2015
3 of 38
Thanks for inquiring. I can confirm that we have sent all legally required disclosures thus far and as we continue to investigate will send out any further required.
Untitled
Community Guru
Petra R Member Since: Aug 3, 2011
4 of 38

Han Y wrote:

 

 

What happened?

 

On February 11, 2019,


I think you'll find it was Tuesday the 12th Smiley Wink

 

Thank you for the Update!

Community Guru
Isabelle Anne A Member Since: May 19, 2014
5 of 38

This post really should be linked to on Upwork's social media pages, the main site, and in an email to users.

Community Guru
Wendy C Member Since: Aug 24, 2015
6 of 38

And emailed to all active buyers and freelance users ....

Community Guru
Mary W Member Since: Nov 10, 2014
7 of 38

Can someone specifically address whether payment information could be accessed through the "wrong profiles"?

Community Manager
Lena E Community Manager Member Since: Apr 7, 2015
8 of 38
Hi Mary,

Any part of the site that requires the profile owner to answer security questions (the payments section included) could not be accessed. Full payment information such as an entire credit card number is also never shown to any customer once it is added. That said our investigation found that, in very rare cases where a customer attempted to add their own payment method to their account during the two hour time period, another customer may have viewed limited information on that payment method... In this case, only the last four digits of the credit card would have been visible since full numbers are never shown. We believe this happened in only one case and are in communication with the customers involved. It is also possible that in some cases past transaction information such as invoices were visible to the other party. We continue to investigate and are communicating directly to any customers who we find may have been impacted.

Untitled
Community Guru
Wendy C Member Since: Aug 24, 2015
9 of 38

As posted in the freelance thread about the same subject:

As some of you know I encountered problems trying to change my password throughout the week. 

 

After Han Yuen, the lead engineers @ U., posted an explanation for what had happened and steps taken, I PMed him details of what I was encountering plus applicable screenshots.

 

A gazillion kudos to Han and Cheryl, the member of the Executive Escalations team, he asked to assist me.   

 

I figure we all b**ched a lot - justifiably; so it is only fair that I express my appreciation and praise to two of U's staff that did a great job.

Active Member
Heba S Member Since: Feb 20, 2019
10 of 38

I've not any problem in my profile

 

i hope that