Update on site performance and issues
09-09-2015 04:23 PM
Many of you may have noticed that our site has been slow or not loading at all in the last few days. This is unacceptable, especially given the role that Upwork plays in many of your businesses and livelihoods. I feel this pain acutely and I sincerely apologize for the deep frustration this has caused.
Resolving these issues is our highest priority. Multiple teams are working hard to restore the site and provide the high level of service you deserve.
We appreciate your patience, and will share updates via this thread as they become available.
VP of Product Management
*Please note: We are keeping this thread closed to further comments so that we can give members of the Community one place for updates. Should you wish to comment on this or other topics, please visit other discussion threads in the Community.
09-10-2015 05:00 PM
I wanted to share an update on the slow load times on Upwork. Resolving this issue continues to be our highest priority.
We have made significant progress on restoring service although I know some of you are still experiencing intermittent slowness. We’ve found that these problems are being caused by a series of abnormal spikes in traffic to our site. We are actively investigating and working urgently to prevent further disruption.
I know this has been very hard for many of you and I want to reiterate my heartfelt apology. Thank you for your continued patience.
09-14-2015 03:37 PM Edited 09-14-2015 07:14 PM
I wanted to check in with a quick update. The site was stable through the weekend and today (an unrelated bug did arise Saturday but was quickly addressed).
We’re still closely monitoring the site speed situation and will keep you updated. Every member of our engineering team is focused on resolution, including ongoing upgrades to our infrastructure to prevent future disruptions. We'll hold off on planned feature releases for a bit longer while we continue to monitor site stability. I’ll share more details soon but wanted to provide everyone with an update after the weekend.
09-16-2015 11:58 AM
The site has now been stable for the last five days. I wanted to take the opportunity to to update you on where things stand and provide some further details on the situation.
Site slowness began seriously impacting our community last Monday the 7th. We had a volatile few days before achieving speed improvements on Thursday. Since then the measures we’ve put in place have returned the site to prior performance levels. (Some unrelated bugs have occurred, but the site slowness has abated.)
A key question from you and for us internally has been about the underlying causes of the degradation of our service. I believe we suffered from two unrelated events:
- As I shared earlier, we experienced some abnormal spikes in traffic which overwhelmed a small number of specific services. We’ve pinpointed the traffic’s origins and are now investigating. We’ve also taken measures to make our services more resilient.
- We’ve been undertaking a major modernization and re-architecture effort. We’re rebuilding legacy services and systems on a new technology stack to increase site performance, reduce downtime, reduce bug rates, resolve bugs more quickly, and make feature improvements more rapidly. Unfortunately these efforts have resulted in some growing pains. New demands on our server and hosting requirements combined with last week’s unexpected traffic surges have added strain to the system and caused the slowness many of you experienced.
We are taking several measures to bolster our infrastructure and prevent disruptions in the future. This includes:
- Increasing server capacity to handle unpredictable traffic volume
- Re-configuring servers to handle greater traffic volatility
- Forming a dedicated engineering group to optimize site performance, incident detection and response time
I’ve been at oDesk, Elance-oDesk, and now Upwork for almost four years. Last week was one of the most challenging times I’ve experienced, as I saw the service we have worked so hard to build fall far short of our promise and of our standards. Every day my team comes into work motivated to build a platform that we believe is improving your lives by creating new opportunities for collaboration across the globe, for flexible careers, and high velocity hiring. I’m here to empower pioneers like you who embrace this revolutionary model. I acutely feel your disappointment in our recent performance.
While I can’t promise that we won’t have other challenges ahead, I can say that we have learned a lot and are doing things differently as a result. We have new systems and protocols in place to guard against future setbacks and to help us identify issues and respond more quickly. We will demonstrate new vigilance in communicating more quickly and clearly about these types of setbacks going forward. Our team is 100% dedicated to you, our extraordinary community. I look forward to helping you succeed.