Whatsapp diary study

Yup, you read that right I used WhatsApp messenger to conduct a diary study. Yup, it felt like a full-time job chatting with 20 participants for 7 days (including the weekend😲), but I would 100% do it again.

“A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences over time. In a diary study, data is self-reported by participants longitudinally — that is, over an extended period of time that can range from a few days to even a month or longer. During the defined reporting period, study participants are asked to keep a diary and log specific information about activities being studied.”
NNgroup
whatsapp-promo.png

WhatsApp & our users

WhatsApp was chosen as through a marketing audit of the Boiler Room community, the majority of users were familiar and confident with the instant messaging app. Weighing this up against onboarding them with more specific tools would have made the study longer and potentially cause participants to opt out prior to the start.

WhatsApp & budget

Money is a sensitive topic, but with WhatsApp being a free app, It meant the research budget which was partially going to be used to pay for a specific diary study tool, was used to reward participants for their time and effort over the 7 days.

WhatsApp & Drop off rates

Although every diary study goes through the onboarding phase of recruiting committed participants, things happen and sometimes you don't get a reply. The joy of using WhatsApp meant I was able to quickly see who had read messages and prompt them in a friendly manner. Messages sent were always in a relatable Boiler Room-esque tone of voice so that users never felt as if they participating a repetitive study. What was interesting and refreshing during the study was some users became proactive and treated the messages as a personal diary without any prompts, which went beyond what I expected and gave the results more context.

WhatsApp diary study in action

Learnings

  • Only use WhatsApp if it's an app your user base are familiar with.

  • Download the WhatsApp app on a laptop, desktop or tablet, the larger screen made managing participants messages and off script personal replies easier as well as giving a fuller view of the participants as a whole.

  • Plan your messages thoroughly and maintain a tone of voice, I scripted every message I could send so that depending on participant replies/ lack of replies I didn't have to overthink what i had to say.

  • Due to the speed of replies, data collection can be a time consuming task if left until the end. I’d recommend making an excel sheet split by participants, and inputting their replies as they come in or end of day.
     

I’ve attached my Listening Habits Test Plan for anyone who wants to give this a try.

What I learnt from measuring everything

During the implementation of Mixpanel in the new Boiler Room app, I was asked what seemed like a simple question at the time “What should we measure in the app?” which I replied “everything!” in my head, it was simple, measure everything and you can find out anything.

Weeks of product analysis passed however only simple product insights could be gathered. My whole idea of measuring everything made finding anything a complicated task, the data had no focus.

Hangover calculations.gif

Looking for inspiration and potential techniques to make our metrics useful during analysis, I ploughed through a range of medium post and articles from Product people who had the same issue. After reading approximately 20 articles, 2 stood out :

Both made sense for Boiler Room and fusing them together I created a more focused set of metrics from which insights could be gained.

Actionable metrics document

To communicate the metrics clearer, quick visualisations were made of how I imagined the metrics to be presented. Through iterating and refining the metrics with the Head of Engineering we settled on a mix of metrics to implement into our apps.

Visualisation of metrics