My name is Han Yuan, and I lead the engineers here at Upwork. I want to let you know that an engineering error recently impacted a small subset of our customers. Specifically, we discovered that some data from your Upwork account might have been visible to another Upwork customer during a two-hour period. We want to make sure you have the facts about what happened, what information was involved, and the steps we are taking to help protect you.
On February 11, 2019, we discovered that around 0800 UTC, some customers were able to see data from other customers who happened to be logged in at the same time during a two-hour period. As customers browsed around the site, they may have suddenly noticed that their sessions appeared as if they were logged in as someone else.
Immediately upon notification, our Site Reliability Engineers shut down our production platform. The site was unavailable for four hours while we undertook the necessary measures to ensure that no further misconfiguration was possible. In practical terms, this means that we had to remediate the issue, understand the impact and provide a solution.
In instances like this, we want to do everything possible to protect our customers. Therefore, we were prepared to leave the site down for as long as it took so that we had confidence that the problem did not appear again. We know our customers rely on our site and apologize for the unexpected downtime.
During this downtime, we learned that the root cause was due to a bug in our code which was triggered when we had an unexpected network failure.
Who does this impact and what information got shared?
In all, we believe that potentially up to 20,000 customers were impacted by this issue during a two-hour period. However, active session data suggests that the number of customers affected could be fewer than 4,000 customers with many members of our community logging out immediately upon noticing the issue. As I write this, we are continuing to parse through the large amount of data from the time period involved in order to further understand the impact.
The nature of the issue allowed freelancers, in theory, to view and edit each other's public profiles. But we are still reviewing the logs to determine if any of this actually occurred and will reach out directly to anyone impacted.
For clients, they would have been able to view and edit job posts, edit and end contracts, run reports, and change notification settings. Again, we are determining if any of these actions actually took place and will reach out directly to any individuals who were impacted.
We also do not believe that customers were allowed to enter certain areas of the site, including access to read other customers’ messages.
We wanted to share this information so you are aware of what happened and can reach out if you believe unauthorized activity has occurred on your account. If that is the case, please reach out to us at support.upwork.com.
What are we doing to improve and how are we moving forward?
We are taking the following actions:
Your information and privacy are important to us, and I want to apologize for any inconvenience this may have caused you. As always, if you have any questions or need any additional information, please do not hesitate to contact us at support.upwork.com.
Can you verify that you have sent the legally-required disclosures directly to affected users and have identified and acted upon any requirement that you send disclosures to government agencies such as the NY Attorney General?
As posted in the freelance thread about the same subject:
As some of you know I encountered problems trying to change my password throughout the week.
After Han Yuen, the lead engineers @ U., posted an explanation for what had happened and steps taken, I PMed him details of what I was encountering plus applicable screenshots.
A gazillion kudos to Han and Cheryl, the member of the Executive Escalations team, he asked to assist me.
I figure we all b**ched a lot - justifiably; so it is only fair that I express my appreciation and praise to two of U's staff that did a great job.