MASTERCLASS: A UX RESEARCH ANALYSIS

Image courtesy: Masterclass

Diving deeper into the concepts introduced in User Research and Design Process for Learning, this independent UX research study examines and evaluates the learning potential of the online learning platform Masterclass. Through the lenses of research-backed learning theories and thorough UX research and analysis, I will propose a redesign of this learning experience according to best practices and stakeholder input.


PROBLEM STATEMENT

Adult (lifelong) learners who engage with informal learning content online are presumed to be motivated by a desire to enrich themselves and improve their quality of life rather than to gain skills or expertise for professional application. Because time is a valuable and limited resource, it’s assumed that this population of learners may expect any investment made in self-betterment to be as entertaining and engaging as it is educational, and for their user experience to be excellent and worthwhile.

The online learning platform Masterclass gained popularity in recent years due to its cadre of renowned and talented instructors, and videos with exceptionally high production value. However, the learning design of Masterclass appears to disregard most of what learning science research promotes as best practice. Ultimately, this may undermine Masterclasses’ long-term viability as a high-quality online learning platform. It’s assumed that Masterclass might disappoint its users who might expect the quality of why and what content is delivered and what they derive from it to match the how of its slick aesthetic.


THE CHALLENGE

RESEARCH GOALS

USER RESEARCH PLAN

I conducted 2 interviews and 1 direct observation with 5 participants via Zoom. The study started with a pre-study stakeholder interview that lasted 15-30 minutes. Then, participants engaged in usability testing by sharing Zoom screens and conducting a “think-aloud” as they navigated the course intro and 1st content module, which took an additional 30 minutes.

Over the following week, participants logged on independently and completed another 2-3 modules of their course. Once finished, we regrouped to conduct a post-study interview and debriefing that lasted 15-30 minutes. I payed particular attention to the participants motivations and expectations, experiences, user flow, and reflections on the learning content in order to develop empathy maps for each participant and 2 journey maps—one ideal and the other reflecting the actual learning experience. This report of findings will include recommendations of ways for this learning experience to better align with it’s users needs and wants.


RESEARCH QUESTIONS

Several questions, categorized into three overarching themes, investigate how users experience the Masterclass platform.

    1. Why do learners use Masterclass?

    2. What do learners expect from Masterclass?

    3. What are learners' perceptions of Masterclass prior to engaging with the content and how do they compare to their actual learning experience?

    4. What are learners’ (implied or explicit) learning goals and objectives?

    5. Are learner communities important to users of Masterclass?

    1. Does Masterclass leverage specific, distinguishing features (as a website technology platform) that are important in comparison to other educational media (ex. tv series or podcasts?

    2. How do learners perceive the personas of the Masters and respond to the storytelling methods that are integrated into the platform?

    3. What factors motivate lifelong learners’ use of asynchronous online learning platforms like Masterclass?

    4. Does observing a Master in a didactic learning environment offer sufficient motivation to sustain learner engagement?

    5. What content formats do learners find the most engaging?

    6. What content formats do the learners use?

    1. What are the affordances of Masterclass?

    2. What activities are available to learners?

    3. What are the user flows?

    4. What affordances does Masterclass leverage (in comparison) to other multimedia learning platforms?

    1. What interactions are possible on this platform?

    2. What content did learners focus on?

    3. What activities are available?

    4. What types of social interactions or other means of support could facilitate learning through feedback, troubleshooting, or discernment?

    5. What do learners like/dislike about this learning experience?

    1. How is learning occurring?

    2. What is Masterclass really teaching?

    3. How is the effectiveness of the learning experience as determined by the learner?

    4. What does the Masterclass learning community look like?

    5. How are members of the learning community interacting?

    6. Is this feature well-integrated into the design?

    7. What is the learner's perception of Masterclass’ community of learners?

FIVE USER’S EXPERIENCES

 

TWO DIFFERENT JOURNEYS


Customer Journey Maps exploring what learner’s actually experience vs. what we want them to.


RESEARCH ARTIFACTS

  • Introduction Interview Script

    [Interviewer] I’m a graduate student in the Digital Media Design for Learning program at NYU, and this summer, I’m working on an independent study of informal online learning experiences.

    Our work will focus on understanding your experiences as an adult learner engaging with a writing (or communication) course offered by Masterclass. Today, we’ll start with a brief interview so I can understand a little more about you and your background, then you’ll share your screen for a  “think aloud”, which is where you'll engage with the course for the first time and talk me through your experience watching the introduction and doing the first module of the course. (Maybe check screen sharing is enabled on your zoom now?) After that, you’ll go off and some time over the next week you’ll log in on your own time and complete another 2 modules on your own. They can be sequential but they don't necessarily have to be. 

    I have a few other participants sharing this login so we should set aside some time in advance so you don’t get booted from the platform from too many people being logged in at the same time. Once you’re done with that, we’ll meet again here on Zoom in about a week or week and a half from now, for a short exercise called a user flow where you’ll walk a bit more through your experiences with the interface and I’ll focus more on specifics interactions, and then we’ll wrap up with a few final interview questions. Sound good? Great.

    At this point, I’d like to start recording the interview to document all the data, which will help support me during the analysis process. I will only use this information for educational purposes, and you will remain anonymous. Would that be okay with you? [If yes, begin recording. If no, use notes].

    I’ll start by asking you a few questions. [Conduct Pre-Study Interview]

    Introduction:

    Tell me a little bit about yourself and what you do for work or what you’re studying?

    What drew you to participate in this study?

    Have you ever taken a Masterclass before?

    (If yes) what have your prior experiences been like? Ask for examples

    Participants [Expectations and Motivations]:

    What are your thoughts about this experience as you prepare to start? Why? Ask for examples

    How motivated are you to take this course? Why? Ask for examples

    What are your expectations for XX course? Why? Ask for examples

    What are you hoping to get out of this experience? Why? Ask for examples

    Are there any goals or specific objectives you’re hoping to achieve? Ask for examples

    You’re welcome to complete the entirety of the course beyond the three modules for this study, do you plan to do that?

    Participants [Learning experiences]:

    In the past, what kinds of resources have you used when looking to learn about or improve your writing or communication skills? (What do you do to learn things?) Give examples

    How do you usually go about that? Give examples

    Beyond writing, what kinds of things do you most often seek out to learn? Give examples

    What motivates you to learn new things? Give examples

    Can you tell me the last time you did something similar to this Masterclass to learn a new skill? Give examples

    You chose XX’s writing (or communication) course, what about that one jumped out at you? Give examples

    Were there any specific reasons why you didn’t select one of the other courses? Give examples

    Outside of Masterclass are there other examples of learning platforms or learning content online that you’ve used? Give examples

    Which ones did you love/hate and why, give examples. 

    What types of digital content do you find most engaging (either outside of the scope of Masterclass or within it?) Give examples 

    Does an online learning community matter to you?

    What are your expectations for online learning communities? Give examples

    What have your past experiences with online learning communities been like? Give examples

    [Second Phase]

    Now, we'll start the second phase, and so I’m going to ask for you to start screen sharing. Once you're doing that, you’ll use the login information that I emailed to access the Masterclass website. After you’re logged in, I’ll ask that you navigate through the website naturally to find the course you (previously) selected if it doesn’t show up automatically, and begin by starting the course introduction (sometimes it’s called the class trailer, and it may not appear at the top of the page). Have you watched this Masterclass intro before? What did you think of it? (Give examples)

    As you’re going through this process, I’ll ask for you to “think aloud” and tell me about your experience as you go about exploring and engaging with each step of the introduction. You can either pause if you want to make an observation or take notes on your own whatever helps you think aloud in this process. Again my role now is to just observe you, so I’m going to try not to answer any questions if possible, I really want to see what it’s like for you to work through this on your own. Try to go through this portion of the study as though I'm not here and you’re talking out loud to yourself.

    Do you have any questions?…

    [Participant shares screen, watches introduction and 1st course module]

    [Participant does Think-Aloud]

    [Wrap-Up]

    Ok so thank you so much for doing this As we wrap up, is there anything you think I should know that’s relevant to this course that I forgot to ask you about?

    And what did you think of what you saw so far? Has your perspective changed at all from before we started?

    Is there anything that you’re apprehensive about in terms of what you’ve seen here so far here or based on your prior learning experiences? 

    Ok so the next step, is that you’ll set up a time frame in the next week or so where you’ll log back in independently and complete 3 modules and then we’ll also schedule a time to meet for a follow up. Which will take an hour. That’s where we’ll look more closely at some of your experiences and wrap up the study. 

    Thank you!

 

Participant #1’s “think-aloud” of the Chris Voss Masterclass

Participant #4 provided a brief summary of their experiences in lieu of a follow-up interview

User flow study of Participant #2’s David Sedaris Masterclass

Participant #5 using the embedded note taking feature during of Amy Tan’s Masterclass during the screen sharing observation