Whatsapp diary study

Yup, you read that right I used WhatsApp messenger to conduct a diary study. Yup, it felt like a full-time job chatting with 20 participants for 7 days (including the weekend😲), but I would 100% do it again.

“A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences over time. In a diary study, data is self-reported by participants longitudinally — that is, over an extended period of time that can range from a few days to even a month or longer. During the defined reporting period, study participants are asked to keep a diary and log specific information about activities being studied.”
NNgroup
whatsapp-promo.png

WhatsApp & our users

WhatsApp was chosen as through a marketing audit of the Boiler Room community, the majority of users were familiar and confident with the instant messaging app. Weighing this up against onboarding them with more specific tools would have made the study longer and potentially cause participants to opt out prior to the start.

WhatsApp & budget

Money is a sensitive topic, but with WhatsApp being a free app, It meant the research budget which was partially going to be used to pay for a specific diary study tool, was used to reward participants for their time and effort over the 7 days.

WhatsApp & Drop off rates

Although every diary study goes through the onboarding phase of recruiting committed participants, things happen and sometimes you don't get a reply. The joy of using WhatsApp meant I was able to quickly see who had read messages and prompt them in a friendly manner. Messages sent were always in a relatable Boiler Room-esque tone of voice so that users never felt as if they participating a repetitive study. What was interesting and refreshing during the study was some users became proactive and treated the messages as a personal diary without any prompts, which went beyond what I expected and gave the results more context.

WhatsApp diary study in action

Learnings

  • Only use WhatsApp if it's an app your user base are familiar with.

  • Download the WhatsApp app on a laptop, desktop or tablet, the larger screen made managing participants messages and off script personal replies easier as well as giving a fuller view of the participants as a whole.

  • Plan your messages thoroughly and maintain a tone of voice, I scripted every message I could send so that depending on participant replies/ lack of replies I didn't have to overthink what i had to say.

  • Due to the speed of replies, data collection can be a time consuming task if left until the end. I’d recommend making an excel sheet split by participants, and inputting their replies as they come in or end of day.
     

I’ve attached my Listening Habits Test Plan for anyone who wants to give this a try.

What I learnt from measuring everything

During the implementation of Mixpanel in the new Boiler Room app, I was asked what seemed like a simple question at the time “What should we measure in the app?” which I replied “everything!” in my head, it was simple, measure everything and you can find out anything.

Weeks of product analysis passed however only simple product insights could be gathered. My whole idea of measuring everything made finding anything a complicated task, the data had no focus.

Hangover calculations.gif

Looking for inspiration and potential techniques to make our metrics useful during analysis, I ploughed through a range of medium post and articles from Product people who had the same issue. After reading approximately 20 articles, 2 stood out :

Both made sense for Boiler Room and fusing them together I created a more focused set of metrics from which insights could be gained.

Actionable metrics document

To communicate the metrics clearer, quick visualisations were made of how I imagined the metrics to be presented. Through iterating and refining the metrics with the Head of Engineering we settled on a mix of metrics to implement into our apps.

Visualisation of metrics

DIY usability testing 101

The thing with usability testing is that there are so many tools and services available that it's easy to dive into their pool of free trials, get infatuated with one, only for the price to bring you back to reality. I often told myself...

“if you used the tool 100 times then technically it's only 3 quid a go… per year”😖.

The Delboy in me wouldn't let me do it, I had to find a way to improvise.

Breaking down the usability products I loved and recalling visits to custom built test centres. I simplified what I wanted the DIY usability testing rig to do:

  • Record the device screen

  • Record users interactions

*Nice to have: Record facial expressions

Using the expertise at my disposal, I had a heart to heart Larry (Boiler Room’s Head of Live production) and explained what I was trying to achieve, within 15 minutes he slacked me “OBS” and nothing was the same.

OBS Project is a free and open source software for video recording and live streaming and is typically used by gamers and up and coming youtubers

Screen Shot 2017-11-29 at 23.51.21.png

How to setup your own rig

You will need recording devices.
I used a Webcam and the Mac screen recording functionality

  1. Download  and install OBS studio https://obsproject.com/download
  2. Selected the type of the input source
  3. Add a “Video capture device”
  4. Add another depending on where you want to record from, this could be a “Video capture device” or a “Window capture”
  5. Layer them Correctly on the screen feedback and start recording

Using this setup I was able to setup a Apple TV usability test to understand how our users interacted with the the device during the research phase of our Boiler Room TV platform.

Usability test script

User research when everyone loves your brand!

User research at Boiler Room is none like anywhere else. Through my experience at agencies and client side, we always had to include a hefty monetary incentive to entice users and even then I had to cross my fingers that they would show up.

This was not the case at Boiler Room, with 1.9 million Facebook followers and a private community of 9000 fans at my disposal, my biggest issue was our incentives were too good and all our fans wanted exclusive show entry and merchandise.

The three biggest issues I faced were:

  • Recruiting the right users

  • Overwhelmed and nervous feedback

  • When to present the incentive

Inspiration from Google

Through leading the Boiler Room community and holding regular user research sessions I became familiar with the fans. Some fans simply just wanted to visit our HQ and get the incentives without participating in any user research session.  Although we had a screener in place I had to be diligent in recruiting the right user as time was always precious.

Taking a leaf out of Google’s Captcha.

I introduced another screener typically in the form of a video call, voice call or another survey as a way to separate useful users from our fans to ensure the right users came in.

Every little thing is gonna be alright

At Boiler Room, things move at 100mph, from teams travelling across the globe to your favourite artist or Dj popping in. I always felt user's apprehensiveness prior to entering our HQ.

To limit this and make them feel comfortable:

  • I always met users at the door and greeted them by name

  • Dedicated 10 minutes to showing users around our HQ and to key rooms they’d be aware of from our Live shows

  • Introduced users to members of the Boiler Room team and anybody they knew (Users always knew someone).

The Incentive

Through past experiences, users were given their incentives at the end of user research sessions. However, with our offering of exclusive show entry and merchandise, waiting until the end didn’t work as users were eager to say the right thing to guarantee they’d receive the incentive. I thought it was simple, hand over the incentive early and everything is all good… I was wrong! Offering the incentive early sparked up a conversation that compromised the duration of the session. Through mixing and matching a tactic rose to the top:

Let the incentive be seen- During the 10 minute show around, I informed users that their incentive was waiting for them. Their incentives were placed in the meeting room where the session was being held with their names clearly visible. These simple tactic helped them feel at ease and allowed the session to flow.