Message testing is a crucial process for pharmaceutical and biotech companies when evaluating the effectiveness of communications geared toward physicians, patients, and other stakeholders. Traditional message testing research involves manual surveys, focus groups, and interviews where participants rank messages based on their motivational impact. While traditional approaches remain effective, there are advantages and disadvantages when contrasted with newer generative AI-powered message testing solutions.
We conducted a comparison between a traditional message test and an AI message test using the same message set. Here, we explore the differences between both methods, highlighting their respective advantages and potential applications.
Traditional Message Testing
Direct human interaction grounds traditional message testing and allows for nuanced feedback that captures the subtleties of motivations and preferences.
This type of study typically involves a two-step process:
- Qualitative Exploration: in-depth interviews (IDIs) conducted with a sample of patients and healthcare professionals (HCPs); in some cases, focus groups.
- Quantitative Validation: a survey with a broader sample to validate the qualitative insights.
While comprehensive, this approach can face constraints due to timing, resources, and sample sizes (especially with harder-to-reach populations). Researchers may spend weeks designing, recruiting, conducting, and analyzing data to execute a comprehensive research plan that will provide actionable insights. For those looking to move quickly to implement new marketing strategies, this can feel like a challenge. Additional constraints can come in the form of biases (i.e., social dynamics of a focus group setting) and difficulty with comprehensively testing large message sets (often the case when testing visual aids).
Generative AI-powered Message Testing
Generative AI-powered platforms can streamline the message testing process by analyzing and ranking messages based on motivational factors and other criteria, without direct input from patients or healthcare providers. Paired with human expertise, this method supports stakeholder-centric decision-making for those who need insights quickly. This approach is also valuable when there is a need to cull down a message list for testing to ensure better, more comprehensive insights coming out of primary research.
Example Prompt: “Please rank each message based on how motivating they are for physicians to prescribe each medication.”
The generative AI tool used in our comparison considers factors specific to the message set when assessing message impact. In this instance, it identified and evaluated the following elements:
- Trust and Acceptance: The level of trust within the medical community.
- Age Range Approval: Approval for different age groups, such as pediatric approval.
- Prescribing Volume: Historical data on prescribing volumes.
- Mode of Administration: Differences in medication administration.
- Quality of Life: The focus on improving patients’ overall well-being.
However, generative AI-powered message testing can also have drawbacks to consider. The quality of generated insights heavily depends on the model used by the system. Choosing a tool that uses an up-to-date LLM^ and a model well suited to your population is critical. Similarly, the prompt used heavily influences the quality of the results. In other words, researcher skill in prompt engineering influences the research outcome. This approach also lacks the qualitative depth of insight from human interactions, potentially missing out on the emotional nuances that drive message optimization.
^Large Language Model
Comparative Analysis of Generative AI vs. Traditional Methods
While the theoretical application of generative AI-powered message testing is sound, it’s critical to compare to the results from a traditional message test. Below, we compare the results from a real quantitative data set versus the results from a generative AI-powered tool. We found the generative AI method performed well, arriving at the same top two messages for both patients and HCPs. The differences beyond that are minimal, indicating that AI can provide comparable insights.
Traditional vs. Generative AI Message Ranking:
What Method Should You Choose?
When to Use a Generative AI-powered Method:
- Quick Insights: useful when there is limited time to gather evidence to inform decision-making.
- Budget Constraints: it is a cost-effective alternative when budget constraints exist.
- Preliminary Testing: can help narrow down the message set, or iterate through high-level message ideas, in preparation for research using more nuanced traditional methods.
- Refreshing a Campaign: it can inform decisions that iterate on an existing campaign already built on a wealth of real data.
When to Use a Traditional Method:
- Precise Validation: ideal when final validation is critical, such is often the case when launching a new brand.
- Planned Message Iteration: provides deeper insights into underlying emotions and nuances that impact message preference (critical for evolving messages throughout the research process).
- Complex Analysis: can better handle the complexity of emotional and contextual factors that impact message preference (especially important when messages serve a diverse audience). It also provides a better understanding of the semantic interpretation of the messages, which can be highly nuanced.
Your specific business need and research objectives, the product’s lifecycle, and available resources should dictate use of traditional and generative AI-powered message testing methods. Traditional methods are ideal for in-depth, qualitative analysis where understanding the “why” and emotion behind preference is crucial. Early in the product lifecycle traditional methods are particularly useful as hypotheses are still being formed. Conversely, generative AI-powered methods are excellent for large-scale, quantitative analysis where speed and efficiency are paramount. During later stages of the product lifecycle a generative AI-powered method is especially beneficial since there is already a base of audience understanding. It is also beneficial for quickly generating evidence to fine-tune messages.
Ultimately, there is a place and purpose for traditional and generative AI-powered methods. When possible, integrating both into your research may offer the most comprehensive insights, combining the depth of human understanding with the efficiency of AI.
November 2024
The Evolving Market Research Landscape Generative AI & Ensuring a Human Touch
Generative AI technology is evolving rapidly. In our discussion, we explore the current state of LLMs, their applications and limitations, and anticipated advancements with GPT o1. However, as LLMs and their application to market research evolves it’s important to keep research human-centered, which requires a clear framework for accountability, ethics and integration.
- Michaela Gascon
- Dan Wasserman
KJT’s Cheat Sheet for Using Generative AI for Patient-Centered Market Research
Download our guide to help you navigate using generative AI tools, with a human touch, for market research.
Author
Troy Allen
Business Operations Manager
Troy is the Business Operations Manager at KJT. He oversees proposal and budget generation, software administration, and strategic projects. With 9+ years of experience in research and operations, he excels in organization, communication, and problem-solving, ensuring optimal business efficiency and top-tier internal customer service.
