At the start of my internship at AWS, I shadowed another researcher for a few weeks to get an idea of how my team does research, and how they manage the resources available to them. Once I had done that I was assigned my first solo project, to test a new visual interface for an AWS service. This was a first for AWS and involved coordinating with a 3rd party contractor who was developing the visual tool, as well as working with the project developers for the specific AWS service.

Objective

• The objective of this study was to discover how users would utilize the visual tool for creating templates in the CloudFormation service.
• A secondary objective was to discover bugs in the software and coordinate with the development teams to fix them

Process

• I utilized an internal database of study participants to recruit users who were familiar with the CloudFormation product and were sufficiently advanced users.
• I then worked together with a designer and developer to put together a research plan that would test various elements of the tool which the developers were keen to get feedback on.
• 7 lab studies were then organized and administered by myself, for the first several studies the developer was in attendance to help hot-fix significant bugs that were inhibiting the flow of the user study
• Following the completion of the user study, I analysed my results and wrote up a detailed report.

Challenges

• In the very first lab session I ran none of the equipment worked. This resulted in my having to create an ad-hoc work around using a computer in an empty participant room to rig a system that would enable the study to continue. Luckily this worked well and the session went smoothly after that.
• The software was extremely buggy early on, this required close working with the developers between sessions to get fixes in place or adjust the study protocol as required.

Outcomes

• The primary outcome of the study was a detailed research report detailing several important results from the study
• Most important of these results was the discovery that the audience the tool was aimed at was incorrect, and this resulted in a pivot from targeting highly experienced users, to more novice users.