Is it important to you to use your company name as your profile name? Asking because I also operate as an LLC, and my tax information and banking information with Upwork is for my LLC, but I use my own name on my profile (which works as long as your name is on the LLC bank account as an authorized signer).
Can I work on projects under this company name? No. You must use your real, verifiable name (as it appears on a drivers license or other government issued ID) according to Upwork's Terms of Service.
Do I need to become an agency? If your question is do you need to become an agency if you want to use your LLC name...the answer is YES, if it's not the same as your real, verifiable name. If your question is do you need to become an agency if you want to operate under your LLC, the answer is NO. You can use your real name, and still operate your business as an LLC if you wish. Tiffany already explained how you can do this.
If so, how do I become one? Go here for instructions on how to create an agency: https://support.upwork.com/hc/en-us/articles/211067598-Create-an-Agency. There are also a plethora of resources just for agencies in the help section to help you along the way.