Reply
cancel
Showing results for 
Search instead for 
Did you mean: 
Reply

Our Commitment to Site Stability

Employee
Han Y Member Since: Feb 5, 2016
1 of 17

Hi all,

 

As head of Engineering here at Upwork, I want to personally apologize to members of our community for the recent troubles we’ve had with the site’s stability. Our teams have continued to battle infrastructure issues that were causing the site to go down intermittently over several days at the end of January. While doing so, we’ve been gathering your questions and concerns so we could provide an update for you here.

 

I’d like to give you more insight into what we’ve done to improve site stability, what we’re doing to resolve these infrastructure issues and to respond to common questions we’ve heard from our community in light of these issues.

 

What have we done to improve your experience and stabilize the site:

 

We temporarily paused all new feature and platform updates.Though our recent issues were not a result of any one feature or activity on the site, we thought it best to delay any new code changes to the Upwork site. This allows our engineering team to be ultra-focused on the most important task: stabilizing the site and improving our infrastructure so you don’t experience these types of outages in the future.

 

We’re investing heavily in our infrastructure. Over the past few months we’ve been steadily upgrading our infrastructure so that we can improve the speed, stability and overall performance of the Upwork platform and effectively scale with the rapid growth of our marketplace. We are moving with increased urgency in light of these recent issues.

 

We’re incessantly monitoring. Our engineering and customer-facing teams are on “high alert” so that we can quickly report and address any site issues that occur. This includes staffing up our teams and improving our monitoring tools so we can more quickly react when something does go wrong. We are following up on any anomalies we identify, even if customers aren’t impacted.

 

 

Responding to your questions:

Here are responses to some of the specific questions you’ve asked.

 

1. Is my information secure? With these site issues, is any of my personal information in jeopardy?

Your information continues to be treated with the highest level of security and there has been no breach of privacy. We quickly confirmed these recent issues were not related to any malicious activity on our site or attempts to hack our systems.

 

2. When there are issues with stability, why do you say the site is functioning normally when I’m experiencing otherwise?

We’re trying to get better at this. As many of you know, last October, we launched our first iteration of a site-status blog. We’ve since updated this page as we look to provide you even more transparency and notice when known issues arise. There are a number of systems, and people, that go into identifying an issue, notifying others and making the necessary updates. First, we’re looking to avoid any site issues in the first place, but we know we can get better at more quickly notifying you. This is a work in progress, but it it’s important to us and we’re committed to getting better.

 

3.Why does it take so long to get these issues fixed? Can’t you just hire some good freelancers to work on this stuff?

It takes a while to make sure that while we’re fixing the issue at hand we’re not breaking something else. We want to rush to get the site back up, but we also want to be deliberate and calculated in our efforts so we provide a meaningful fix that will fully resolve the issue. We are extremely sensitive to any lost time this means for freelancers and clients. Providing a consistent and stable platform is my team’s number one objective. I can’t rightfully defend the recent performance of our site, but I will defend the team members who are part of our engineering team. We maintain the highest standards when hiring for our team, many of whom are freelancers, and demand a lot from those who do work with us. These recent issues are not a result of incompetence or carelessness but rather of our evolving technology and infrastructure.

4. Is this over? Are you going to maintain a stable site moving forward?

We’re going to do whatever we can to ensure site stability. These most recent issues were unexpected and are something we will do everything to avoid moving forward. If we anticipate some disruptions due to changes we’re making, we’ll communicate so you can be prepared and avoid any disruptions to your business.

 

We are entirely focused on giving you the stability and performance you count on for building your businesses online. I hope you understand we take this matter very seriously, we are taking steps to prevent site instability and we’re listening.   

 

Han Yuan

Senior VP, Engineering

Han Yuan
Community Guru
John K Member Since: Feb 17, 2015
2 of 17

Han Y wrote:

 

 

What have we done to improve your experience and stabilize the site:

 

We temporarily paused all new feature and platform updates.Though our recent issues were not a result of any one feature or activity on the site, we thought it best to delay any new code changes to the Upwork site. This allows our engineering team to be ultra-focused on the most important task: stabilizing the site and improving our infrastructure so you don’t experience these types of outages in the future.

 


 Thank you for the open and detailed communication. This is so much better than the standard response along the lines of "We apologize... developers are working hard to resolve issues". Will there be a follow up announcement when you've decided to resume new features and updates?

Community Manager
Garnor M Moderator Member Since: Oct 29, 2014
3 of 17

Hi John,

As we resume rolling out new features and updates, we will notify the community as we've done historically via posts in this Announcement area.

Employee
Han Y Member Since: Feb 5, 2016
4 of 17

I want to add some additional color to how we're thinking about releases moving forward.

 

As engineers, we're passionate about fixing bugs and building a better product. We read your messages on the forum, and we do want to fix things, but you need to ship software to do so.

 

On the other hand, there is some non-zero probability that every time we change the code, we might introduce new bugs or in the worse case take the site down.

 

Starting in Q4 of last year, we made an internal decision to treat all customer impacting bugs as the most severe type described internally as "p0." Before this, we used to try to assess the severity. Internally, we now view anything impacting customers as a crisis even if it means the site does not go down. To wit, this is why even when the site does not go down, we will try our best to update our status page and send tweets. The latest incident this week resulted in system degradation of about 10%-15%. Under the new policy, we inform you of such incidents despite the fact that most customers did not notice this.

 

As a result of the latest site down issues we've had this month, we instituted a new policy. Every time we have a customer impacting incident (even if the site is not down), we will call a 3-business day moratorium for us to analyze why this happened and create stability for our community.

 

Andy Grove's "paired metric" is the inspiration for this. In a sense, what we want to do is encourage our team to ship code but balance this with the real need to keep the site up with minimum customer impact. My hope is that this policy creates the right incentives.

 

The availability of the site is an existential concern for many of you and while we can't guarantee things are going to be smooth always. I can assure the community that we are treating the matter seriously by actively changing the way we do work to serve the community, and this process of continuous improvement will not stop.

Han Yuan
Active Member
abhinand v Member Since: Sep 6, 2015
5 of 17

Hi Han,

 

Thanks for the detailed Messsage. 

Any way you will be having a QA site for parallel development. If you can do the QA well before pushing the changes to live . this can avoid many of stability issues, but i know creating a replica with same load as the live site is a huge process but for a critical business like this such an activity will provide a stable environment. The users like me are solely depending upon the platform and any ups and down are directly affecting us.  Also if possible a Report Issue button can be placed in site if possible so users can report if QA missed something.  We are ready to provide whatever we can to make the platform stable.

 

Please advise.

Thanks,
Abhinand 

Ace Contributor
Armela C Member Since: Jun 6, 2015
6 of 17

Thank you  for your kind announcement.  I  have  reported  such stability problems with your  customer  Support,  stabilities  I never  experienced except  when  Windows  upgraded  96  and to  Win xp, which was  very rare and  short, national due to  our service  provider  technicalities,.   It was a   remote access  assistance  and  a  Windows update  that corrected  my non-Upwork  and browse-Upwork applications,    

 

As  regards, my notifications, and doubted  lost of communication,  it  was  clear  that an  open candidacy  did not register  in  my   candidacy  page, and did  not enter a  notification. But  the message  came in.   But let  me reiterate that a  notifcation  beep  maybe necessary  to  remind or  advise  me, a  feature that  is  not  in  place  aside  from  physically  unnotified  candidacy but  deductively may  have worked   in the  room  creation.   So vise  versa,  some how my website drop  downs   werent working  and  I  had  open  proposals.   I have  been  informed  that   a  candidacy is not  a  contract  but  some are  unresponsive   in  an open candidacy,  which is unlikey,  according to my  expectations.

 

I hope  you can  monitor the settings your  back -end  have  on  my  website.   I am  an  Intellectual  Property  Holder and  a TradeMark  and I put  care  on  carrying  the Philiipine flag and  International Property and business laws that it is covered and   protects  it.

 

I thank you for the wonderful stability  I have of my Upwork  website  today  and hopefully,  it is henceon.   For behind the scenes team of engineers and developers, your solutions and  efforts  are  appreciated.

 

 

ARMELA  Y CAMARA (Tm)

 

Community Guru
Isabelle Anne A Member Since: May 19, 2014
7 of 17

Agree with John. This is the best response I've seen so far regarding this issue. At least you addressed the major concerns instead of pretending they don't exist.

 

Not everyone visits the forums, though, so make sure to post this message on the Upwork blog, FB page, etc. 

 

I'm still doubtful about the future of site but let's hope that it only gets better from here.

Community Guru
Irene B Member Since: Feb 17, 2015
8 of 17

This is probably one of, if not the most, detailed and explained apology ever provided on this site. 

 

Thank you!

Community Guru
Vesna M Member Since: May 15, 2012
9 of 17

Thank you for this. Finally a feedback from responsible person.

 

Although, I wasn't concerned very much about stability because I knew you would fix it eventually, nature of this problem is so important that you don't much have other option than work hard to repair.

 

What I am concerned about are those little site functionality problems that last for longer period of time, not being solved, and I don't see when it will be put on schedule. Like messages in wrong, client account, or search filters with awkard results, eg. economists, writers showed in Design, Presentations category.

 

There were other things in past, but it seems it's working properly today.

It must be extremely hard job, and I believe you'll make everything work nicely. Just it needs some time, I guess.

____________
Don't correct my grammar!
Frequent Visitor
Daniel T Member Since: Jul 28, 2015
10 of 17

"We temporarily paused all new feature and platform updates " Are you guys going to implement any features the old Elance had ? Some of them were really usefull, also i must say that on the old Odesk as here on Upwork too, i never get invites to jobs, as on old Elance i was invited daily at least once to a job. This makes me think that somehow this actions is not as user friendly on these two sites as it was on Elance. And it`s not about me getting invited it`s per general, when i go to a job`s post i only see "initiated by freelancer" and rarely "initiated by client" in the "Proposals" list. This is only one thing. 

 

Any way as other users said, thank you for coming to us with this message, and keep up the good work.

Best regards,

Daniel.

___

(and sorry for any mistakes in my grammar or logic, i`m not a native english speaker.)

TOP KUDOED MEMBERS