Tag Archives: participant

Tips For Conducting Usability Studies With Participants With Disabilities

Over the last few years, I ran several usability studies with participants with various disabilities. I thought it would help others if I shared some of my experiences.

In this article, I provide lessons learned or tips to consider in planning and executing usability testing with participants with disabilities. The lessons learned are divided into general that can apply to all types of disabilities; and lessons learned for three specific disability categories: visual, motor, and cognitive. These tips will help you regardless where you work: If you work with an established user research team where usability testing is part of your design process or if you work on your own with limited resources but want to improve the quality of the user research by expanding the diversity of participants.


Windows 10 high contrast mode of Google.com


Windows 10 high contrast mode of Google.com. (Large preview)

Background

Several of our clients from a state government agency to several fortune 500 companies came to us at the User Experience Center (UXC) for help with their websites. They wanted to make sure users with disabilities could access their site and accomplish their goals.

There are many different kinds of disabilities, however, there is a general agreement to categorize people with disability into four general categories: visual, auditory, motor (also referred to as “physical”), and cognitive. There are different conditions and much variability within each category, e.g., within visual disabilities, color blindness, low vision, and blindness. There is also a distinction as to when a disability is contracted, e.g., a person who was born blind as opposed to one who lost vision later on in life.

Furthermore, as we age or encounter unique situations (such as multi-tasking), we may have a similar experience to people we think of as disabled. Therefore, disabilities should be thought of as a spectrum of abilities that should be accounted for during the design of all user interfaces and experiences.

Typically, in order to ensure that disabled people can use their digital products and services, companies aim for compliance with accessibility guidelines such as the Web Content Accessibility Guidelines (WCAG 2.0). While this is critical, it is also important to have users with disabilities try to accomplish real tasks on the site in usability testing. There may be gaps in the overall user experience…

Think about the typical doors found in buildings. How many times have you tried to open a door one way and realized they actually open the other, for example, push instead of pull. Technically the door is accessible, but it is usable?


Example of doors that are technically accessible, but not usable


Example of doors that are technically accessible, but not usable (handles give impression you should push but must pull to open). (Large preview)

Even if a site follows the accessibility guidelines and is technically accessible, users may not be able to accomplish their goals on the site.

Lesson Learned

In most ways, usability testing with this segment of the population is no different than testing with anyone else. However, there are several areas you need to pay just a bit more attention to so your sessions run smoothly. The lessons or tips are broken down into general ones that can apply to all participants and specific tips for various disability types such as visual, motor, and cognitive.

General Lessons Learned

1. Ensure a baseline level of accessibility before usability testing

Ensure a baseline level of accessibility before usability testing: Planning usability testing, especially recruiting participants can take time both for the project team and the recruited participants.

Two good examples of basic accessibility issues that should be addressed prior to usability testing are:

  • Missing alternative (alt) text.
    Usability testing can be used to see if the alt text used is appropriate and makes sense to participants, but if all the participants are doing is confirming that the alt text is missing then this is not a good use of their time.
  • Appropriate color contrast.
    All page designs should be reviewed beforehand to make sure all foreground and background colors meet WCAG 2.0 AA color contrast ratios.
2. Focus the recruiting strategy

If you work with an external recruiter ask them if they have experience recruiting people with disabilities; some do. If you are recruiting internally (without an external recruiter), you may need to reach out to organizations that have access to people with disabilities. For example, if you need to recruit participants with visually disabilities in the United States, you should contact a local chapter of the National Federation of the Blind (https://nfb.org/state-and-local-organizations) or a local training center such as the Carroll Center for the Blind in Massachusetts (http://carroll.org/). If you use social media to advertise your study, a good approach is to use the hashtag #a11y (stands for accessibility — there are 11 letters between the “a” and “y”) in your post.

3. Bring their own equipment/assistive technology

Allow and encourage participants to bring their own equipment such as their own laptop, especially if they use assistive technology. This way, you can truly see how people customize and use assistive technology.

4. Have a backup plan for assistive technology

As stated above in #3. It is best if participants can bring their own equipment. However, it is always wise to plan for the worst, for example, if a participant does not bring their equipment or if there is a technical problem such as you can’t connect their equipment to your Wi-Fi network. In the case of visually impaired participants, install assistive technology (AT) such as screen reader software they will be bringing in on a backup PC. For many of the AT software packages, you can get a free trail that should cover you for the usability testing period. This has saved us several times. Even though the configuration was different than what the participants had, we were able to run the session. Participants were quickly able to go into the settings and make some adjustments (e.g., increase the speech rate) and get started with the session.

5. Allow additional time

Provide additional time in-between sessions. Typically we like to reserve 30 minutes between participants. However, when participants plan to bring in their own equipment additional time may be required for setting up and resolving any issues that may arise. When testing with individuals with disabilities, we schedule an hour between sessions, so we have extra time for setting up assistive technology and testing it.

6. Confirm participant needs

Either with the recruiting screener or via email or telephone, confirm what equipment participants will bring in and need to be supplied beforehand. In our lab, we can connect external laptops (that in this case, were outfitted with special accessibility software and settings) to our 1Beyond system via an HDMI cable. In a recent study, all of our participants’ laptops had HDMI ports. However, we forgot to check this beforehand. This is an example of a small but important thing to check to prevent show-stopping issues at the time of the test.

7. Consider additional cost

Depending on the disability type transportation to the usability testing location may add additional burden or cost. Consider the cost of transportation in the incentive amount. If feasible, consider providing an extra $25-$40 in your incentive amount so participants can take a taxi/Uber/Lyft, etc. to and from your location. Depending on access to public transportation and taxi/ride-sharing rates in your area the amount may vary. Our participants came to the UXC in different ways — some more reliable and timely than others.

8. Revise directions

Check the directions you provide for accessibility. Make sure they include an accessible path into your building. Test them out beforehand. Do you need to provide additional signage? If so, ensure all signs are clear, concise, and use plain-language directions.

9. Review the emergency evacuation plan

Review the plan in the event of a fire or other emergency. Map out the emergency evacuation plan in advance.

10. Consider logistics

Consider remote usability testing as an option. One of the benefits of bringing individuals with disabilities into the lab for usability testing is observing first-hand participants’ use of the product or website in question. However, the logistics of getting to your location may be just too much for participants. If it’s possible to test remotely (we typically do this through Zoom or GoToMeeting), it should be considered. This poses the additional challenge of making sure your process for capturing the remote session is compatible with all of the participant’s assistive technology, as well as accessible itself. Troubleshooting remotely is never fun and could be more difficult with this segment of the population.

11. Hearing impaired participants

Some participants may have a hearing impairment where the position of the moderator and participant is critical for adequate communication. In the case of hearing-impaired participants, it is important to get their attention before talking to them and also to take turns when engaging in conversation.

To get the most of this research, it is best if the participants are not discovering basic accessibility issues that should have been discovered during an accessibility review and/or testing.

Lessons Learned For Participants With Visual Disabilities

Participants with visual disabilities range from people who are blind and use screen readers such as JAWS, to people that need to the text or the screen to be enlarged using software such as ZoomText or relying on the native screen enlargement in the browser. People that are color-blind also fall into this category.

  • For any documents needed prior to the study such as the consent form, send via email beforehand and ask them review and send back in lieu of a physical signature. If you don’t, be prepared to read aloud the consent form and assist in signing the documents for some participants.
  • Make sure directions provide step-wise directions; do not rely only on graphical maps as these may not be accessible.
  • For all documents, make sure color is not used as the sole cue to convey information. Print out all documents on a black and white printer to make sure color is not required to make sense of the information.
  • Get participants mobile phone numbers in advance and meet them at their drop-off point. Be prepared to guide them to the testing location. Review best practice for guiding blind individuals:
  • While Braille documents can be helpful for participants that read Braille, the time and cost involved may not be feasible. Furthermore, all blind people do not read Braille, especially people that have lost sight later in life. It is best to make sure all documents can be read via a screen reader. Unless you are sure if there are no accessibility issues avoid PDF documents and send out simple Word documents or text-based emails.
  • If participants bring guide dogs do not treat them as pets, they are working. Provide space for the dog and do not pet it unless the participant gives you permission.
  • Make sure to explain beforehand any sounds or noise that are or may be present in the room such as unique audio from recording software. This may avoid the participant from becoming startled or confused during the session.
  • Initially when I started to work with blind participants I was worried my choice in language might offend. However, over the years I have learned that most blind participants are fairly relaxed when it comes to speech. Therefore, during moderation do not be afraid to use phrases such as “see” or “look” and similar words when talking to blind participants; for example, “please take a look at the bottom of the page” or “what do you see in the navigation menu?” In my experience, blind participants will not be offended and will understand the figurative meaning rather than the literal meaning.
  • Test out all recording equipment/processes beforehand. Ensure all audio including both human speech in the room and audio/speech from AT such as screen readers will be recorded correctly. During testing of the equipment adjust the locations of the microphones for optimal recording.

Lessons Learned For Participants With Motor Disabilities

Motor disabilities refer to disabilities that affect the use of arms or legs and mobility. These individuals may need to use a wheelchair. Some people may not have full use of their hands or arms and cannot use a standard mouse and keyboard. These people may need to voice recognition software which allows to use voice input or use a special pointing device, for example, one that is controlled by their mouth.

  • In the directions, make sure the route is accessible and routes them via elevators rather than stairs. Also, if participants are driving note the location of accessible parking.
  • Note if doors have accessible door controls. If not you may need to meet the participant and guide them to the testing location.
  • Make a note of the nearest accessible restrooms to the testing location.
  • As with all participants with disabilities, it is best if they can bring in their own laptop with their assistive technology software installed and any other required assistive technology. However, in the case of participants (such as Adriana in Figure 3) that use voice recognition software such as Dragon Naturally Speaking this is critical because they have trained the software to recognize their voice.
  • Make sure the desk or table where the participant will be working can accommodate a wheelchair and the height is adjustable. According to the American with Disabilities Act (ADA), conference tables must be 27 inches high in order to accommodate knee clearance for individuals in wheelchairs..

Adriana Mallozzi conducting a usability test at the User Experience Center. Picture-in-Picture view of the computer page taking up most of the screen and small video feed of Adriana n the lower right-hand corner


Adriana Mallozzi conducting a usability test at the User Experience Center. Adriana has a motor disability (cerebral palsy) which affects her entire body. She uses a wheelchair, a sip-and-puff joystick that acts as a mouse, along with Dragon Naturally Speaking. (Large preview)

Lessons Learned For Participants With Cognitive Disabilities

Individuals with these disabilities cover a wide range of relatively mild learning disabilities such as Dyslexia to individuals with a more profound cognitive disability such as Down syndrome. In general, people with cognitive disabilities have challenges with one or more mental tasks. Rather than looking at specific clinical definitions it best to consider functional limitations in key areas such as memory, problem-solving, attention, reading or verbal compensation. Consider how best to accommodate participants during usability testing. Many of the tips below should also apply to all participants, however for this group you need to be extra aware.

  • Sometimes participants will be accompanied by a caretaker or an aide. This person may assist with transportation or may need to be present with the participant during the usability test. If the caretaker is present during the usability test, make sure they understand the structure of the usability test and what will be required of the participant. If you know the participant will be accompanied before the study, you review the goals and protocol prior to arrival via email or phone. That is as much as possible the participant should be one conducting the usability testing, and the caretaker should not be involved unless it is completely necessary.
  • In some cases, the caretaker or aide may act like an interpreter. You may need to communicate with this interpreter in order to communicate with the participant. If this is the case, make sure you record the audio coming from both the participant and the interpreter.
  • Provide instructions in multiple modalities, for example, both written and verbal. Be patient and be prepared to repeat the task or ask the same question multiple times.
  • Be prepared to break tasks into smaller sub-tasks to support memory/attention challenges or fatigue that may set in.
  • Ideally, it is best to be consistent with tasks for all participants however for some participants with cognitive disabilities you should be prepared to go off-script or modify tasks on the fly if the current approach is not working.
  • Have the participant’s comfort and well-being the number one priority at all times. Don’t be afraid to take multiple breaks or end the session early if things are just not working out or the participant is not comfortable.



Amazon.com home page zoomed in at 250%. (Large preview)

Additional Insights

The tips above should serve as guidelines. Each participant is unique and may require various accommodations depending on their situation. Furthermore, while some of the tips are categorized for specific disability types, specific individuals may have multiple disabilities and/or benefit from a tip from a different category than their primary disability.

If you or your company have conducted user or customer research, you know the value of gathering feedback about the issues and benefits of products and systems. Testing with individuals with disabilities is no different, as you learn many insights that you would not gain otherwise. However, an additional takeaway for us was the realization that people use assistive technologies in different ways. The following example is specific to people with visual disabilities, but there are similar examples across all groups.

An assumption might be someone that is blind only uses a screen reader such as JAWS and is an expert at it. We found that people with visual impairments actually differ greatly in the level of support needed from assistive technology.

  • Some users need a screen reader for accessing all content.
  • Some users (with more sight/with low vision) only need to enlarge content or invert page colors to increase contrast.
  • Others may need a combination of approaches. One visually impaired participant used both a screen reader along with the zoom function embedded in the web browser. She only used a screen reader for large paragraphs of text, but otherwise simply zoomed in with the web browser and got very close to the screen when navigating around the website.

Furthermore, just like anyone, all users are not experts on the software they use. While some users would consider themselves experts, some only learn enough about the software to accomplish what they need and no more.

Moving Forward

Hopefully you have learned some useful information that will help you include more diversity into your usability testing. However, since there is variability with different disabilities, this may seem overwhelming. I recommend starting small; for example by including one or two participants with disabilities as part of a larger group of 5 to 10 participants. In addition, initially bring in someone that has both experience with usability testing and a lot of experience with their assistive technology so you can focus on getting their feedback rather than how the usability testing process works or their use of their assistive technology.

Acknowledgements

I would like to thank Jocelyn Bellas, UX Researcher at Bank of America and Rachel Graham, UX Researcher at Amazon. When Rachel and Jocelyn worked at the User Experience Center as Research Associates in 2016, they worked with me on some of the projects referenced in this article and also contributed to a related blog post on this topic.

References

Smashing Editorial
(cc, ra, yk, il)

View this article: 

Tips For Conducting Usability Studies With Participants With Disabilities

How pilot testing can dramatically improve your user research

Reading Time: 6 minutes

Today, we are talking about user research, a critical component of any design toolkit. Quality user research allows you to generate deep, meaningful user insights. It’s a key component of WiderFunnel’s Explore phase, where it provides a powerful source of ideas that can be used to generate great experiment hypothesis.

Unfortunately, user research isn’t always as easy as it sounds.

Do any of the following sound familiar:

  • During your research sessions, your participants don’t understand what they have been asked to do?
  • The phrasing of your questions has given away the answer or has caused bias in your results?
  • During your tests, it’s impossible for your participants to complete the assigned tasks in the time provided?
  • After conducting participants sessions, you spend more time analyzing the research design rather than the actual results.

If you’ve experienced any of these, don’t worry. You’re not alone.

Even the most seasoned researchers experience “oh-shoot” moments, where they realize there are flaws in their research approach.

Fortunately, there is a way to significantly reduce these moments. It’s called pilot testing.

Pilot testing is a rehearsal of your research study. It allows you to test your research approach with a small number of test participants before the main study. Although this may seem like an additional step, it may, in fact, be the time best spent on any research project.
Just like proper experiment design is a necessity, investing time to critique, test, and iteratively improve your research design, before the research execution phase, can ensure that your user research runs smoothly, and dramatically improves the outputs from your study.

And the best part? Pilot testing can be applied to all types of research approaches, from basic surveys to more complex diary studies.

Start with the process

At WiderFunnel, our research approach is unique for every project, but always follows a defined process:

  1. Developing a defined research approach (Methodology, Tools, Participant Target Profile)
  2. Pilot testing of research design
  3. Recruiting qualified research participants
  4. Execution of research
  5. Analyzing the outputs
  6. Reporting on research findings
website user research in conversion optimization
User Research Process at WiderFunnel

Each part of this process can be discussed at length, but, as I said, this post will focus on pilot testing.

Your research should always start with asking the high-level question: “What are we aiming to learn through this research?”. You can use this question to guide the development of research methodology, select research tools, and determine the participant target profile. Pilot testing allows you to quickly test and improve this approach.

WiderFunnel’s pilot testing process consists of two phases: 1) an internal research design review and 2) participant pilot testing.

During the design review, members from our research and strategy teams sit down as a group and spend time critically thinking about the research approach. This involves reviewing:

  • Our high-level goals for what we are aiming to learn
  • The tools we are going to use
  • The tasks participants will be asked to perform
  • Participant questions
  • The research participant sample size, and
  • The participant target profile

Our team often spends a lot of time discussing the questions we plan to ask participants. It can be tempting to ask participants numerous questions over a broad range of topics. This inclination is often due to a fear of missing the discovery of an insight. Or, in some cases, is the result of working with a large group of stakeholders across different departments, each trying to push their own unique agenda.

However, applying a broad, unfocused approach to participant questions can be dangerous. It can cause a research team to lose sight of its original goals and produce research data that is difficult to interpret; thus limiting the number of actionable insights generated.

To overcome this, WiderFunnel uses the following approach when creating research questions:

Phase 1: To start, the research team creates a list of potential questions. These questions are then reviewed during the design review. The goal is to create a concise set of questions that are clearly written, do not bias the participant, and compliment each other. Often this involves removing a large number of the questions from our initial list and reworking those that remain.

Phase 2: The second phase of WiderFunnel’s research pilot testing consists of participant pilot testing.

This follows a rapid and iterative approach, where we pilot our defined research approach on an initial 1 to 2 participants. Based on how these participants respond, the research approach is evaluated, improved, and then tested on 1 to 2 new participants.

Researchers repeat this process until all of the research design “bugs” have been ironed out, much like QA-ing a new experiment. There are different criteria you can use to test the research experience, but we focus on testing three main areas: clarity of instructions, participant tasks and questions, and the research timing.

  • Clarity of instructions: This involves making sure that the instructions are not misleading or confusing to the participants
  • Testing of the tasks and questions: This involves testing the actual research workflow
  • Research timing: We evaluate the timing of each task and the overall experiment

Let’s look at an example.

Recently, a client approached us to do research on a new area of their website that they were developing for a new service offering. Specifically, the client wanted to conduct an eye tracking study on a new landing page and supporting content page.

With the client, we co-created a design brief that outlined the key learning goals, target participants, the client’s project budget, and a research timeline. The main learning goals for the study included developing an understanding of customer engagement (eye tracking) on both the landing and content page and exploring customer understanding of the new service.

Using the defined learning goals and research budget, we developed a research approach for the project. Due to the client’s budget and request for eye tracking we decided to use Sticky, a remote eye tracking tool to conduct the research.

We chose Sticky because it allows you to conduct unmoderated remote eye tracking experiments, and follow them up with a survey if needed.

In addition, we were also able to use Sticky’s existing participant pool, Sticky Crowd, to define our target participants. In this case, the criteria for the target participants were determined based on past research that had been conducted by the client.

Leveraging the capabilities of Sticky, we were able to define our research methodology and develop an initial workflow for our research participants. We then created an initial list of potential survey questions to supplement the eye tracking test.

At this point, our research and strategy team conducted an internal research design review. We examined both the research task and flow, the associated timing, and finalized the survey questions.

In this case, we used open-ended questions in order to not bias the participants, and limited the total number of questions to five. Questions were reworked from the proposed lists to improve the wording, ensure that questions complimented each other, and were focused on achieving the learning goals: exploring customer understanding of the new service.

To help with question clarity, we used Grammarly to test the structure of each question.

Following the internal design review, we began participant pilot testing.

Unfortunately, piloting an eye tracking test on 1 to 2 users is not an affordable option when using the Sticky platform. To overcome this we got creative and used some free tools to test the research design.

We chose to use Keynote presentation (timed transitions) and its Keynote Live feature to remotely test the research workflow, and Google Forms to test the survey questions. GoToMeeting was used to observe participants via video chat during the participant pilot testing. Using these tools we were able to conduct a quick and affordable pilot test.

The initial pilot test was conducted with two individual participants, both of which fit the criteria for the target participants. The pilot test immediately pointed out flaws in the research design, which included confusion regarding the test instructions and issues with the timing for each task.

In this case, our initial instructions did not provide our participants with enough information on the context of what they were looking for, resulting in confusion of what they were actually supposed to do. Additionally, we made an initial assumption that 5 seconds would be enough time for each participant to view and comprehend each page. However, the supporting content page was very context rich and 5 seconds did not provide participants enough time to view all the content on the page.

With these insights, we adjusted our research design to remove the flaws, and then conducted an additional pilot with two new individual participants. All of the adjustments seemed to resolve the previous “bugs”.

In this case, pilot testing not only gave us the confidence to move forward with the main study, it actually provide its own “A-ha” moment. Through our initial pilot tests, we realized that participants expected a set function for each page. For the landing page, participants expected a page that grabbed their attention and attracted them to the service, whereas, they expect the supporting content page to provide more details on the service and educate them on how it worked. Insights from these pilot tests reshaped our strategic approach to both pages.

Nick So

The seemingly ‘failed’ result of the pilot test actually gave us a huge Aha moment on how users perceived these two pages, which not only changed the answers we wanted to get from the user research test, but also drastically shifted our strategic approach to the A/B variations themselves.

Nick So, Director of Strategy, WiderFunnel

In some instances, pilot testing can actually provide its own unique insights. It is a nice bonus when this happens, but it is important to remember to always validate these insights through additional research and testing.

Final Thoughts

Still not convinced about the value of pilot testing? Here’s one final thought.

By conducting pilot testing you not only improve the insights generated from a single project, but also the process your team uses to conduct research. The reflective and iterative nature of pilot testing will actually accelerate the development of your skills as a researcher.

Pilot testing your research, just like proper experiment design, is essential. Yes, this will require an investment of both time and effort. But trust us, that small investment will deliver significant returns on your next research project and beyond.

Do you agree that pilot testing is an essential part of all research projects?

Have you had an “oh-shoot” research moment that could have been prevented by pilot testing? Let us know in the comments!

The post How pilot testing can dramatically improve your user research appeared first on WiderFunnel Conversion Optimization.

Source – 

How pilot testing can dramatically improve your user research

Using Brainwriting For Rapid Idea Generation

When a group wants to generate ideas for a new product or to solve a problem, you will usually hear the clarion call, “Let’s brainstorm!” You assemble a group, spell out the basic ground rules for brainstorming (no criticism, wild ideas are welcome, focus on quantity, combine ideas to make better ideas) and then have people yell out ideas one at a time.
Brainstorming is often the method of choice for ideation, but it is fraught with problems that range from participants’ fear of evaluation to the serial nature of the process — only one idea at a time.

See original:

Using Brainwriting For Rapid Idea Generation

The Rainbow Spreadsheet: A Collaborative Lean UX Research Tool

Liverpool FC fans sing “You’ll Never Walk Alone” to their players during matches. UX research is best done when a team is involved. When you run UX research on your own without active observers, you are missing its point. This article describes and gives you a tool I created called the Rainbow Spreadsheet.
With it, you will be able to collaboratively observe UX research sessions with team members (or clients).

Original post: 

The Rainbow Spreadsheet: A Collaborative Lean UX Research Tool