There’s a joke in the marketing world that A/B testing actually stands for “Always Be Testing.” It’s a good reminder that you can’t get stellar results unless you can compare one strategy to another, and A/B testing examples can help you visualize the possibilities. I’ve run thousands of A/B tests over the years, each designed to help me hone in on the best copy, design, and other elements to make a marketing campaign truly effective. I hope you’re doing the same thing. If you’re not, it’s time to start. A/B tests can reveal weaknesses in your marketing strategy, but they…
How do you know your content marketing is effective? It’s not a rhetorical question, though it may seem like it. While it’s difficult to measure the success and return on investment of content purely quantitatively, you can absolutely use digital analytics to get some directional insights as well as insights that help you improve your approach. The good thing: there are no shortage of tools and guides to help you do that nowadays. The bad thing: it can be a bit overwhelming when you think about how to get started with digital analytics, especially if you’ve got a content calendar…
It’s 2018 already, and countless front-end developers are still leading a battle against complexity and immobility. Month after month, they’ve searched for the holy grail: a bug-free application architecture that will help them deliver quickly and with high quality. I am one of those developers, and I’ve found something interesting that might help.
We have taken a good step forward with tools such as React and Redux. However, they’re not enough on their own in large-scale applications.
Webinars are one of the most popular tools used by marketers for lead generation. Not only are they great for generating demand but they’re also a less pushy way of nurturing cold leads. The reason is that you are offering to provide information that your audience will value in your webinars. You can also demonstrate your expertise and showcase your knowledge of the industry and domain using webinars. However, webinars can be truly beneficial for your company if they are planned and implemented well. Here, we’ll take a look at some of the things you need to do to ensure…
Google Analytics (GA) is capable of generating incredibly detailed and comprehensive data. It provides the insights needed to fine-tune your site, reduce UX friction and ultimately maximize conversions. But there’s a catch. It’s only effective if you actually know how to interpret the data. Unfortunately, not all users fully understand the core metrics, and there’s uncertainty as to how to decipher them. Here, we’ll take a look at six of the most misunderstood metrics in GA to find out what the data means and how to apply it in order to optimize your site. 1. Direct Traffic At first…
(This is a sponsored post). Websites with long or infinite scrolling are becoming more and more common lately, and it’s no mere trend or coincidence. The technique of long scrolling allows users to traverse chunks of content without any interruption or additional interaction — information simply appear as the user scrolls down the page.
Infinite scrolling is a variety of long scrolling that allows users to scroll through a massive chunk of content with no finish line in sight (it’s the endless scrolling you see on Facebook, Twitter and Tumblr feeds).
Today, we are talking about user research, a critical component of any design toolkit. Quality user research allows you to generate deep, meaningful user insights. It’s a key component of WiderFunnel’s Explore phase, where it provides a powerful source of ideas that can be used to generate great experiment hypothesis.
Unfortunately, user research isn’t always as easy as it sounds.
Do any of the following sound familiar:
During your research sessions, your participants don’t understand what they have been asked to do?
The phrasing of your questions has given away the answer or has caused bias in your results?
During your tests, it’s impossible for your participants to complete the assigned tasks in the time provided?
After conducting participants sessions, you spend more time analyzing the research design rather than the actual results.
If you’ve experienced any of these, don’t worry. You’re not alone.
Even the most seasoned researchers experience “oh-shoot” moments, where they realize there are flaws in their research approach.
Fortunately, there is a way to significantly reduce these moments. It’s called pilot testing.
Pilot testing is a rehearsal of your research study. It allows you to test your research approach with a small number of test participants before the main study. Although this may seem like an additional step, it may, in fact, be the time best spent on any research project.
Just like proper experiment design is a necessity, investing time to critique, test, and iteratively improve your research design, before the research execution phase, can ensure that your user research runs smoothly, and dramatically improves the outputs from your study.
And the best part? Pilot testing can be applied to all types of research approaches, from basic surveys to more complex diary studies.
Start with the process
At WiderFunnel, our research approach is unique for every project, but always follows a defined process:
Developing a defined research approach (Methodology, Tools, Participant Target Profile)
Pilot testing of research design
Recruiting qualified research participants
Execution of research
Analyzing the outputs
Reporting on research findings
Each part of this process can be discussed at length, but, as I said, this post will focus on pilot testing.
Your research should always start with asking the high-level question: “What are we aiming to learn through this research?”. You can use this question to guide the development of research methodology, select research tools, and determine the participant target profile. Pilot testing allows you to quickly test and improve this approach.
WiderFunnel’s pilot testing process consists of two phases: 1) an internal research design review and 2) participant pilot testing.
During the design review, members from our research and strategy teams sit down as a group and spend time critically thinking about the research approach. This involves reviewing:
Our high-level goals for what we are aiming to learn
The tools we are going to use
The tasks participants will be asked to perform
The research participant sample size, and
The participant target profile
Our team often spends a lot of time discussing the questions we plan to ask participants. It can be tempting to ask participants numerous questions over a broad range of topics. This inclination is often due to a fear of missing the discovery of an insight. Or, in some cases, is the result of working with a large group of stakeholders across different departments, each trying to push their own unique agenda.
However, applying a broad, unfocused approach to participant questions can be dangerous. It can cause a research team to lose sight of its original goals and produce research data that is difficult to interpret; thus limiting the number of actionable insights generated.
To overcome this, WiderFunnel uses the following approach when creating research questions:
Phase 1: To start, the research team creates a list of potential questions. These questions are then reviewed during the design review. The goal is to create a concise set of questions that are clearly written, do not bias the participant, and compliment each other. Often this involves removing a large number of the questions from our initial list and reworking those that remain.
Phase 2: The second phase of WiderFunnel’s research pilot testing consists of participant pilot testing.
This follows a rapid and iterative approach, where we pilot our defined research approach on an initial 1 to 2 participants. Based on how these participants respond, the research approach is evaluated, improved, and then tested on 1 to 2 new participants.
Researchers repeat this process until all of the research design “bugs” have been ironed out, much like QA-ing a new experiment. There are different criteria you can use to test the research experience, but we focus on testing three main areas: clarity of instructions, participant tasks and questions, and the research timing.
Clarity of instructions: This involves making sure that the instructions are not misleading or confusing to the participants
Testing of the tasks and questions: This involves testing the actual research workflow
Research timing: We evaluate the timing of each task and the overall experiment
Let’s look at an example.
Recently, a client approached us to do research on a new area of their website that they were developing for a new service offering. Specifically, the client wanted to conduct an eye tracking study on a new landing page and supporting content page.
With the client, we co-created a design brief that outlined the key learning goals, target participants, the client’s project budget, and a research timeline. The main learning goals for the study included developing an understanding of customer engagement (eye tracking) on both the landing and content page and exploring customer understanding of the new service.
Using the defined learning goals and research budget, we developed a research approach for the project. Due to the client’s budget and request for eye tracking we decided to use Sticky, a remote eye tracking tool to conduct the research.
We chose Sticky because it allows you to conduct unmoderated remote eye tracking experiments, and follow them up with a survey if needed.
In addition, we were also able to use Sticky’s existing participant pool, Sticky Crowd, to define our target participants. In this case, the criteria for the target participants were determined based on past research that had been conducted by the client.
Leveraging the capabilities of Sticky, we were able to define our research methodology and develop an initial workflow for our research participants. We then created an initial list of potential survey questions to supplement the eye tracking test.
At this point, our research and strategy team conducted an internal research design review. We examined both the research task and flow, the associated timing, and finalized the survey questions.
In this case, we used open-ended questions in order to not bias the participants, and limited the total number of questions to five. Questions were reworked from the proposed lists to improve the wording, ensure that questions complimented each other, and were focused on achieving the learning goals: exploring customer understanding of the new service.
To help with question clarity, we used Grammarly to test the structure of each question.
Following the internal design review, we began participant pilot testing.
Unfortunately, piloting an eye tracking test on 1 to 2 users is not an affordable option when using the Sticky platform. To overcome this we got creative and used some free tools to test the research design.
We chose to use Keynote presentation (timed transitions) and its Keynote Live feature to remotely test the research workflow, and Google Forms to test the survey questions. GoToMeeting was used to observe participants via video chat during the participant pilot testing. Using these tools we were able to conduct a quick and affordable pilot test.
The initial pilot test was conducted with two individual participants, both of which fit the criteria for the target participants. The pilot test immediately pointed out flaws in the research design, which included confusion regarding the test instructions and issues with the timing for each task.
In this case, our initial instructions did not provide our participants with enough information on the context of what they were looking for, resulting in confusion of what they were actually supposed to do. Additionally, we made an initial assumption that 5 seconds would be enough time for each participant to view and comprehend each page. However, the supporting content page was very context rich and 5 seconds did not provide participants enough time to view all the content on the page.
With these insights, we adjusted our research design to remove the flaws, and then conducted an additional pilot with two new individual participants. All of the adjustments seemed to resolve the previous “bugs”.
In this case, pilot testing not only gave us the confidence to move forward with the main study, it actually provide its own “A-ha” moment. Through our initial pilot tests, we realized that participants expected a set function for each page. For the landing page, participants expected a page that grabbed their attention and attracted them to the service, whereas, they expect the supporting content page to provide more details on the service and educate them on how it worked. Insights from these pilot tests reshaped our strategic approach to both pages.
The seemingly ‘failed’ result of the pilot test actually gave us a huge Aha moment on how users perceived these two pages, which not only changed the answers we wanted to get from the user research test, but also drastically shifted our strategic approach to the A/B variations themselves.
In some instances, pilot testing can actually provide its own unique insights. It is a nice bonus when this happens, but it is important to remember to always validate these insights through additional research and testing.
Still not convinced about the value of pilot testing? Here’s one final thought.
By conducting pilot testing you not only improve the insights generated from a single project, but also the process your team uses to conduct research. The reflective and iterative nature of pilot testing will actually accelerate the development of your skills as a researcher.
Pilot testing your research, just like proper experiment design, is essential. Yes, this will require an investment of both time and effort. But trust us, that small investment will deliver significant returns on your next research project and beyond.
Do you agree that pilot testing is an essential part of all research projects?
Have you had an “oh-shoot” research moment that could have been prevented by pilot testing? Let us know in the comments!
AdWords can be an incredibly powerful platform for growing your business, but the truth is that most businesses are wasting thousands of dollars every month due to poor account management. If you feel like you could be generating more revenue from AdWords, chances are that you’re making some of these ten common mistakes that are inhibiting your performance. In this post I will be showing you ten common AdWords mistakes that I’ve seen literally hundreds of advertisers make. And, more importantly, I’m going to show you exactly how to fix them, so you can take next steps towards creating an…
The virtual realm is uncharted territory for many designers. In the last few years, we’ve witnessed an explosion in virtual reality (VR) hardware and applications. VR experiences range from the mundane to the wondrous, their complexity and utility varying greatly.
Taking your first steps into VR as a UX or UI designer can be daunting. We know because we’ve been there. But fear not! In this article, we’ll share a process for designing VR apps that we hope you’ll use to start designing for VR yourself.
There is a lot to learn this week. It starts with non-technical things like going for a walk to refresh your mind and finishes with how to prevent reverse XSS attacks in forms. But it doesn’t matter whether you learn how to build self-contained web components using the new specification or to maximize the efficiency of your Angular 2 app or just how you can write less code. What matters is that you keep asking questions and that you try to get better and smarter at your craft.