As part of a larger review of current offerings ordered by senior Amazon management, I was tasked to investigate the services provided by a leading competitor to a specific AWS product.

Objective

• The objective of this study was to identify specific pain-points, ease of use issues, and opportunities for improvement for the AWS service over the competitor.
• The study was also tasked with discovering how the AWS service out performs the competitor, and how such differences can be further leveraged to improve the AWS product.

Process

• The first stage of the project was recruitment. I worked with a 3rd party company to recruit a dozen participants for the project. Participants were required to have a knowledge base that would enable them to use the service being studied, but not have used the service itself before.
• The dozen participants were split into two groups, half using the AWS product, and half the competitor.
• The study began with 1.5 hour lab studies, where participants signed up for the specific service and performed set of tasks (these tasks were the same across both services).
• Following the completion of the lab studies, participants were asked to continue using the product over a multi-week period, and to record their observations and usage patterns in a diary.
• The diary was designed as a short Word document, designed by myself, to record their interactions with the products.
• The requirements for the diary study were specific tasks that had to be completed each week, and a specific number of usage hours spread across the week. Participants submitted diary updates to me once weekly, and I started a rolling analysis of their input, updating their insights each week.
• Once the multi-week diary study was complete I analysed the results and built up a preliminary findings document in order to prepare for follow-up interviews with participants
• In the last phase users were interviewed again to gain further insights into their usage patterns, and the specific inputs that they had provided from their diaries.

Challenges

• Recruitment proved to be highly challenging, due to the very specific requirements, and so an external vendor was utilized.
• The scale of this study proved to be highly challenging. Conducting a dozen lab studies over the course of a week while writing up results and administering the start of the diary study portion was significantly difficult to manage.
• Keeping track of the participants, issuing and recovering diaries, organizing times for interviews and lab studies was a lot of work. Despite this only 1 participant dropped out of the study mid-way.

Outcomes

• Through the large amount of very rich data that was collected from the labs studies, diary studies, and interviews I was able to produce a detailed report documenting the specific areas where the competitor had an advantage over the AWS product, and areas where AWS was leading.
• My report was then incorporated into the larger report and sent to senior management, where it was very well received for it's thoroughness and detail.

This research study was somewhat unusual for AWS, as it ended up running as an almost multi-month endeavour. Because of this, and the resources that were deployed into the study, there was a high-level of expectation from both my team and AWS management in producing significant results. I managed to meet these expectations with a great deal of work, and produce a rich research report with actionable feedback for the relevant AWS development teams, as well as management, on how the AWS product can better compete in its specific space.