Remesh – 2019
Waiting Room
Increase participant engagement and provide moderators with the demographic balance for their study to be successful
Role
Senior Product Designer ― research, design, and testing
Team
James Ong - Senior Product Manager
Dan Reich - Lead Engineer
Problem
Low participation rates, opaque process, and unbalanced conversations
After tracking the participation data on Remesh, we realized that there is a high number of drop-offs after joining the conversation - participants who join but never engage with the conversation. This hurts the moderator's experience and our relationship with the clients. Also, moderators are unaware of how participants access the conversation and when which leads to more questions to our customer success team and visits to our help documents.
Publishing conversations
Publish conversation becomes the main action users need to make in order to give access to participants and to start the conversation, as opposed to auto-starting the conversation based on time.
Publish also serves as the save action that the date, time, title of the conversation are set, and to change they need to Unpublish.
Participants waiting room
Before the start time of the conversation, all participants go through the waiting room first. Adding this screen and the Join Conversation made sure that only active participants at the start time are the ones joining so we do not save a seat for passive/idle participants.
Clear state handling for participants
Give the participants a clear idea of the different states of the conversation: Unpublished, Published, Started, Full Capacity.
Quota ratios on demographic filters
Give the participants a clear idea of the different states of the conversation: Unpublished, Published, Started, Full Capacity.
Research
Analyze the drop off in participation
Our data showed that our first come first serve flow was causing the low participation numbers because we were saving seats for participants who become idle or passive after answering a couple of onboarding polls.
In fact, looking at the data we found that the participants who are arriving closer to the start time of the conversation are more likely to participate in the entire conversation and answer all questions asked by the moderator. Our previous workflow was benefiting and preferring the early participant but they were the most likely to leave while waiting for the conversation to start.
Understand the old UX pitfalls
First, I evaluated the Remesh legacy platform, focused specifically on the transition between Build to Live experience. It was clear that users were not sure what to do once they have created and set up their conversations. The issues they encountered are:
What do participants see once they click on the participant link
Does the link work?
What do I do now? What’s next?
Until when can I keep changing the information of the conversation ie. date, time, onboarding polls?
What is Quota Management? And how does it work?
Quota management and balancing the demographics of respondents is a feature that is only attempted and solved in quantitative asynchronous tools, it aims at giving researchers a way to target specific groups for their studies and to make sure that their work is statistically significant.
When it comes to asynchronous research, it’s easy to only collect the data from the participants that fit the specific group you need as a user or decide based on screeners if a participant will get the full experience or not. The big question was how to control quotas in the real-time Remesh conversation while participants keep joining.
Solution
Introduce explicit actions by the users and build a quota management system
We introduced two actions to be taken by both moderators and participants: Publish Conversation and Join Conversation. This made expectations and flow very transparent between moderators and participants.
We also built a way for moderators to associate ratios to demographics and an algorithm that admits and reject participants based on the demographic ratios.
Outcome
In 6 months from the rollout, we increased the participation rate and engagement by 73% – and quota management helped our client get the perfect balance for their sample.
Next Steps