Reducing Medical Device Risk with Usability Testing: The Why, the How, and the Who

Industry research shows that each year, approximately 400,000 hospitalized patients experience some kind of preventable harm.

Medical device and equipment flaws caused by poorly researched design, mishandling, user error, and malfunction are common causes of medical errors. Many of these errors can be attributed to lack of standardization, poor design, poor maintenance, differences between devices from different manufacturers, and more.

What are the product’s intended uses?

• What’s the intended use? How and in what circumstance should the product be used? What is the intended medical indication the product is meant to treat?

• Are patients using the product to self-treat? Could there be any physical limitations that prevent a certain patient from interacting with your product as intended?

• Is training required? What level of training is required, if necessary? (We highly recommend training be kept front of mind throughout the development process.)

• What is the user interface of the product? This includes all points of interaction — physical and digital — between the user and the device.

In what use environment is the product being deployed?

• Is the product intended to be used exclusively in the home by a patient? In an emergency room or ambulance?

• Does the product use differ, depending on the environment of use? Whether the user is in an adult ICU, a NICU, or in an ER trauma bay — each of these contributes to significant differences in the way a product is used.

From a regulatory standpoint, the FDA will be looking to see that all elements of the users, uses, and use environments have been researched and incorporated into the plan for conducting proper usability testing in the relevant simulated-use environment with the intended users and uses.

The ideal process might include multiple rounds of formative usability testing, which can be used to inform a Use Failure Mode and Effects Analysis (uFMEA). As you go through the cyclical design process — adding design inputs, testing them, refining, then adding more design inputs — you’ll eventually reach a point where you’re comfortable with the product you’ve created.

A final production-equivalent medical device should be put through validation, or summative, testing, to assess whether or not the product can be deemed safe and effective to use.

2. Assess! Create a Use Failure Mode and Effects Analysis (uFMEA)

A uFMEA assessment is meant to identify components of the user interface and the impact of task failures. What are the possible failures that could occur? Why would these failures occur? What are the consequences of the failure and what’s the associated harm?

Start by creating a step-by-step list of all tasks, required to accomplish the device’s end goals. (Tasks are defined as the action or set of actions performed by a user to achieve a specific goal.)

The uFMEA also notes what could go wrong if any of these tasks are not completed correctly. To help determine this, go through a PCA (perception, cognition, action) analysis process. 

As your team goes about defining tasks, determining failure modes, and assigning levels of severity of harm, you may next want begin to categorize your tasks as critical versus non-critical. 

Critical tasks are those which, if performed incorrectly or not performed at all, could cause serious harm to the patient or user where harm is defined to include compromised medical care. 

User tasks are often tied to product requirements, and usability testing (formative and validation) will be needed to prove these user needs have been met.

Finally, when developing your product’s uFMEA, make sure your team has thought through the level of mitigation that will, ultimately, be required by the IFU and labeling within the context of intended use. During this process, you may also come to a point where you recognize whether or not your product will require some level of training.

Be diligent about identifying all potential use errors and outcomes before getting into validation or even pre-validation testing. Be as clear as possible about what could go wrong and what needs to go right in order to merit a task has been completed successfully. These success criteria will feed into a task analysis table that will define what needs to be tested from a usability standpoint.

And finally, keep in mind that a uFMEA should be treated as a living, breathing document that evolves as the product progresses through the design process. Expect it to change as the device goes through rounds of formative usability testing, is put in the hands of users, and as the design team gains a better understanding of how the product can and should be used.

3. Research! Use These Three Approaches To Usability Testing

There are three usability testing approaches to consider:

• Rapid Insight Testing

• Formative Testing

• Validation Testing

Rapid Insight Testing may be appropriate if all that’s needed is a quick touch-base to keep the design grounded in user needs. Roughly five to six participants are recommended, but it’s possible to test fewer in certain situations. This type of testing typically occurs prior to entering design control.

We recommend conducting Rapid Insight Testing on all elements of the product, including human factors. Sample questions are: How did it feel to use or handle the product? Is the grip intuitive or did someone need to show me how to grip it? Are we testing all of the possible users of the product or are there others that could be seen as a potential user?

This is your opportunity to evaluate all tasks, not just the critical ones. Additionally, this is your chance to get as much subjective feedback on the product as possible! What are their preferences? What are they used to seeing and what would they like to see changed?

Formative Testing begins to inform the design process. Anywhere from five to ten representative users per user group are recommended and aim to simulate the use of the product in the simulated use environment with a high-fidelity prototype. We suggest conducting a minimum of two to three rounds of formative testing during a product development cycle. This is what’s going to lead to product design change mitigations, which is ideal — better now than later! Design changes are hugely preferable to changing labeling or instructions-for-use (IFU) late in the game.

Validation testing is required by regulatory bodies in the U.S. and Europe, and elsewhere around the world. At this stage, testing with a production-equivalent product is expected, and when dealing with FDA, 15 representative users per user group is required, as is representative simulated use in the simulated environment.

It’s important to note that once you’ve reached validation testing, only critical user tasks should be evaluated, so it is important to evaluate all non-essential user tasks earlier in the process via rapid insight and formative testing.

4. Comply! Resources and Tips for Regulatory Consistency

A long discussion could be had when it comes to regulatory compliance, but here are a few resources, tips, and misconceptions to keep in mind. 

Resources for Regulatory Compliance

The Association for the Advancement of Medical Instrumentation’s (AAMI) standard HE75:2009 provides general considerations and principles to help manage the overall risk of use error with best practice design elements and integrated solutions. This document acts as a comprehensive reference for how to incorporate human factors engineering into the medical device design process.

As far as IEC 62366 goes, it guides the entire usability engineering process, including the elements of accompanying documentation and training. The main purpose of this document is to help define the human factors engineering and usability process as it pertains to medical device design, including consideration of risk management.

The FDA guidance — the 2015 “Applying Human Factors and Usability Engineering to Medical Devices” — lays out the needs and expectations for your Human Factors/Usability Engineering Report. FDA developed this guidance to “assist industry in following appropriate human factors and engineering processes to maximize the likelihood that new medical devices will be safe and effective for the intended users, uses, and use environments.”

The Center for Devices and Radiological Health (CDRH) considers human factors testing a valuable component of product development for medical devices, so it is very important to be mindful of usability throughout the design process.

You Say FDA, I Say IEC

It’s good to know some of the key differences in expectations between FDA (Food and Drug Administration — U.S.) and IEC (International Electrotechnical Commission):

• FDA expects a single report — the HFE/UE Report — to hold all relevant information while the IEC 62366 requests similar information and data but does not require a single report.

• FDA expects to see conclusions at the beginning of the document. The IEC does not have these expectations.

• IEC 62366 uses the term “summative study,” but FDA uses the term “validation”; the terms are interchangeable, which confuses those who are not familiar with the industry. With that in mind, note that “usability validation” is a very specific element you need to test for that is within an overall umbrella of design validation.

• FDA only accepts validation usability studies conducted in the United States; IEC does not state such requirements for location. Note however, that any differences in the intended uses, users, and use environments between the U.S. and Europe and any other international body needs to be appropriately tested. 
  
  

Don’t Go It Alone!

We know this is a lot to take in. If you need a development partner, Delve is always up for a good challenge. With our design strategists, mechanical and electrical engineers, and industrial and interaction designers, we’re well-equipped to assist in making a better product.

In addition to providing design and process recommendations and collaborating on system enhancements, we can plan, conduct, and report on all elements of your usability testing needs — from rapid insight testing all the way through validation. Let us know how we can help. Learn more about Delve Product Development Services for Medical Devices.