![podcast_standard_2]()
Are you ready for audits? ISO? FDA? Is your quality management system working the way you want it to? Is it still just a checkbox activity? Or, are you ready to elevate yourself to be a true quality professional?
A quality management system should help us run our business better and produce safer medical devices. It should be built well and run well - effective and efficient.
On today’s episode, we have Larry Mager, a senior advisor for UL EHS Sustainability. He shares a tool he developed called the Quality System Performance Assessment that determines the operational quality of a quality system.
Listen Now:
Like this episode? Subscribe today on iTunes or Spotify.
Some highlights of this episode include:
- The Quality System Performance Assessment is built upon three foundational aspects: 1) Stakeholders require compliance with regulations and standards; 2) Processes must be effective in attaining expected results; 3) Business stakeholders expect those processes are efficient in operational execution.
- The three parts of a performance assessment are an :1) Audit for the compliance; 2) Appraisal of the management system; and 3) Analysis of the data.
- A compliance audit focuses on the identification and verification of the policies, processes, and procedures that a company developed to satisfy regulations and standards - what should be done.
- Companies do not need to wait for audits from ISO or FDA. They can initiate internal audits to determine the effectiveness of their quality system.
- An appraisal is focused on the management of a quality system, and the goal is to understand the results of the management processes - how they do it.
- It takes five integrated components, 17 key principles around those components, and three primary objectives to appraise what works and what doesn’t with a quality management system.
- Data has become very important. Gather data to look at the quality of the output and the adherence of quality processes. UL EHS has a method that helps companies how to measure each process within a quality system for effectiveness of required results. Also, measure the efficiency of operational execution.
- A company’s products are evaluated with similar items on the market to determine if its products are behaving differently and if they are being identified for malfunction, injury, or death.
- The FDA is coming! The FDA is coming! Companies can have a competitor that has issues with its products. If your company has similar products, then the FDA may just walk through your door one day to review your products, as well. What’s the benefit risk?
- Take information from the compliance audit, appraisal of the management, and the analysis of the data to process it as a performance report to identify concerns and plan for improving quality.
Links:
Larry Mager
UL EHS Sustainability
Quality Management System Performance Assessment for Medical Device Industry
Greenlight Guru
U.S. Food and Drug Administration (FDA)
Manufacturer and User Facility Device Experience (MAUDE)
International Organization for Standardization (ISO)
Memorable Quotes by Larry Mager:
“The last 12-15 years of my career have been primarily focused on an understanding of how best to transform a quality system.”
“(Companies) They don’t look at their quality system as the thing that helps them out operationally...or something that needs to be assessed...”
“A lot of times, these systems are created and they’re not known or followed. ”
Transcription
Announcer: Welcome to The Global Medical Device Podcast, where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the worlds leading medical device experts and companies.
Jon Speer: Is your quality management system performing the way that you want it to? Are you still just focused on it being a check box activity or are you ready to elevate to being a true quality professional? I mean folks we spend so much time putting together our quality management system, it should help us better run our business. That's why I'm thrilled about this episode of The Global Medical Device Podcast.
With me on this episode is Larry Mager. Larry is with UL EHS Sustainability. He shares with us, a little bit about this tool that he has perfected over decades of experience, called the quality system performance assessment. So, enjoy this episode of The Global Medical Device Podcast.
Hello, and welcome to The Global Medical Device Podcast. This is your host, the founder and VP of quality and regulatory at Greenlight Guru, John Speer. Folks we've talked a lot about different challenges and things that we're faced with in the medical device industry. We're always trying to improve our practices and make sure that we're on the right path with our quality system, with our regulatory strategies and that sort of thing, sometimes we get stuck.
Some of those areas that we get stuck in is, “Are we ready for audits? Are we ready for the FDA? Are we ready for ISO?” Yeah of course those are important things to consider, but there's probably an intermediate step that we can do internally, to make sure we are ready.
Joining me today on The Global Medical Device Podcast, is Larry Mager. Larry is the Senior Advisor for UL EHS Sustainability. Larry welcome to The Global Medical Device Podcast.
Larry Mager: Thank you Jon.
Jon Speer: Well Larry, I know you and I have talked quite a bit about things that companies can do and should do, and being proactive and that sort of this, but you mentioned something to me the other day that was really interesting and I was intrigued. I thought, “If I'm intrigued, I'm sure our listening audience is as well.” You said audit or assessment and in my mind I'm like, “Okay, well what does that mean? What's the difference?” Maybe you can just take from a 30,000 foot view, just give us a bit of an overview of what you mean by audit versus an assessment and then we can dive into some of the details today, if that's okay.
Performance assessment
Larry Mager: Absolutely, yes. I think I need to start by stepping back just a little bit, and talk about the foundation about why I'm talking about more than an audit, and this thing that we call the performance assessment. It's really more designed to provide an understanding of the operational quality of a quality system. It's built upon three foundational tenants. First is really that stakeholders, whether they're regulatory stakeholders or business stakeholders, they require that the quality system is compliant to the regulation or standards that the company must adhere to. Nothing new there.
The second is really that the processes within the quality system must be actually effective at obtaining the results that are expected of them. While you might say that's a given, I don't think that's always something people think about when they're auditing.
The third, is that the business stakeholders actually have an expectation that those processes are efficient in operational execution, because we're in business. I think that's important to understand first and foremost.
As I step out from that, the three parts of an assessment would then be, an audit for the compliance to regulatory requirements or standards, and appraisal of the management system in order to understand the effectiveness and efficiency, and finally the analysis of the data, because the truth is in the data. That would be a high level view of what I call a performance assessment.
Jon Speer: All right, so folks' performance assessment, let's just recap. There's a few different layers to this. Historically our minds seem to have been wrapped around audit, audit, audit. Audit is certainly a component of that, but there's a couple of different pieces of this, that we're gonna explore a little bit with Larry today.
Larry it's probably good for our audience ... I made the huge assumption that people know who Larry Mager is, partly because I know who Larry Mager is. Take a moment and tell the listeners a little bit about your background. I know you've worked with some big companies some small companies, so you've been around the block a time or two, and not to put the age on you, but you know what you're doing. So tell us all a little bit about your background.
Larry Mager: I have about 30 years in the Life Sciences Industry. Started out as an engineer, working as an engineer in a fairy significant medical device company. Went to the West Coast and worked in the start up environment. I've been the fourth person in companies, the 12th person in companies, developed quality systems for those companies. I have patents for cardiovascular products, as a result of working in that environment. I've worked back in larger companies, where I was involved in executive roles, involved in quality system remediation. I think probably it's fair to say, the last 12 to 15 years of my career, have been primarily focused on an understanding of how best to transform a quality system.
Jon Speer: Well, I think that last point is really important, because my experience is ... You and I have had the pleasure of crossing paths a time or two throughout my career, and it always seems like Larry's the guy that comes in to get it right. You know? That's my take on my experience in working with you. That last piece is important. I think companies, in my experience, they have often a view that a quality management system is there because it has to be, not because it's there to help them. It sounds like you've had some similar types of experiences.
Larry Mager: Yeah. Actually a lot of times when companies are in trouble, that is exactly the reason. They don't look at their quality system as the thing that helps them operationally. They don't actually look at is as something that needs to be assessed from a perspective of the quality of that system itself. They typically throw resources at it when it's in trouble, but not necessarily use it to make the performance of the organization better.
Jon Speer: I've been a part of companies that are just getting started. It seems like they're trying to check the box, on the quality system side of things, not that they're really spending enough time, effort and energy to design an approach that makes sense from their business perspective. They're trying to check that box. Like you have been involved in that other side where, it's not too late, but a lot of damage is already done. Whether it be from an FDA investigation or an ISO audit, now we're in a remediation effort, and that's not fun. Let's talk a little bit about what companies can do to stay on top of things.
appraisals, Audits and inspections
You talked about, there's a few different parts as part of the performance assessment. You talked about the audit piece, the appraisal piece, and the analysis piece, so let's dive into a little bit from an audit piece. Are we talking about an audit in a traditional sense? Like an ISO audit, an FDA inspection, or an internal audit. Maybe expand a little bit about that.
Larry Mager: Sure. The audit is actually more in the traditional sense, where the auditors focus will be on the identification and verification of the policies, process and procedures, that the organization has developed to satisfy the regulatory requirements and standards that the company is supposed to adhere to.
But the regulation and standards, they really only detail what it is that should be done. They don't necessarily detail how to do it, although there is some information there, but they're primarily about what a company is supposed to do. For that reason, compliance audits tend to only be used to confirm to conformance to the required regulations and standards. They don't go beyond compliance for that reason.
Jon Speer: Yeah, but I mean at this point in time, for your model, the performance assessment that you've shared a little bit and we're obviously gonna talk a little bit more about that, but I don't have to wait for my next ISO audit or my next FDA inspection. I can initiate this audit activity internally, maybe in sync with my internal audit schedule or maybe if I really have an interest in improving my quality system, to realize that it's more than just a check box activity, more than just a compliance activity, that this is something that can help me in my business. I can initiate this audit cycle on my own. Maybe give a tip or two what a company might be able to do to start the ball rolling on improving their overall approach to quality management systems.
Larry Mager: Great. It then begins with that audit, for an understanding of that compliance, but then really taking a look at how the management system is structured so that the quality system can run.
Really an appraisal, and I like to stay away from the word auditing when I talk about everything beyond compliance. An appraisal is focused on the management then of the quality system. The goal is to understand the effectiveness of the results have pained, really through the execution of the management of the processes within that system. While the compliance audit is focused on what should be done, the appraisal of the management of the system, focuses on how they do it. That's a big difference.
Jon Speer: Yeah, my-
Larry Mager: ... Assumptions.
Jon Speer: No, go ahead. I was just going to share a story. This goes back quite a few years, but it hopefully will help connect some dots to those listening about the difference. I was being pulled into ... This was right when I was starting my consulting practice a dozen or so years ago. I was being asked to do work on the quality system side with a start up company. There had been a lot of work that had been done on the quality system before my involvement. I was looking through all the procedures, I was comparing it against 1345 against FTA20. Essentially I was doing in audit of sorts. I was essentially saying, “Yes, this is a quality system that seems to be compliant.”
I remember the first day that I was on site at the company. I was looking for the proof. I was doing an appraisal of sorts to see, “Okay, have the resources at this company, the people who are following these processes and procedures, do they have the proof, the documentation, the objective evidence, the mindset, to show that this is working for them?” Low and behold, I found out that most of the people I talked to, had no idea that there was even a quality system. That was like, “Oh, big disconnect.” I'm sure that never happens right?
Larry Mager: Right. Yeah, it's interesting. The assumption is out there, that if we simply create a system and follow the procedures, that it will produce the desired result. The problem is, a lot of time these systems are created, and they're not known, or they're not followed. The answer is, just because you have a quality system, doesn't necessarily mean it's going to work. That's really where we have to start to focus, is making sure that the quality system works. To that, for the output of that system, is dependent on both the design of the system and on the management of the system to produce the results. It has to be built well and it has to run well.
Jon Speer: I've heard a lot of things where people don't want the FDA in their business telling them how to do things. They accept that there needs to be a quality system, but they also have this mindset that, “That's that quality and regulatory thing. This is how I run my business, I'm not going to do it over here.” But designed properly, a quality system should accomplish both, right?
operational quality
Larry Mager: Absolutely. If you think about anything you do in life, whether it's brushing your teeth or anything at work, it's all process. Right? If you think of a system, a quality system as a system of processes, they're supposed to work together to create some output, which would be quality product as an example. In and of themselves, each process might not accomplish that, so what you're really looking for, is to make sure that the system is integrated and aligned in terms of sequence and interaction with defined inputs and outputs between the processes to properly function and perform within that system. Each process perform within that system as intended. That's the system level.
Then when you get to the process level, there's an expected outcome, that each process within that system should be defined and measured in terms of effectiveness and efficiency. That's what we would expect of a process in our personal lives, that's what we should expect of a process we execute in our professional lives as well.
Jon Speer: Yeah. People that have talked to me about quality systems, I'm a big fan of ... I use the term right sizing my quality system. In my position I see some good things and I see some not so good things, as far as implementation of quality systems. I know we could share war stories all day, I'm sure Larry, but I've seen companies at two extremes.
They have very little, as far as defined processes. Maybe they have processes, but it's very disconnected and it's not defined, so it's very, very informal at best.
Then I've also seen the other extreme, where you've got this smaller company, who has implemented processes that were designed for a very large company. In both cases it doesn't work. So what do you do? How do you appraise what works and what doesn't work? I know the audit is a feeder into that, but obviously there's a little bit more magic than just an audit, to help you appraise the success of a quality system.
Larry Mager: Yeah. The appraisal aspect, there is actually a formal process of structure that we have, that I believe is really good. I won't be able to get into the detail in this call, but it takes five integrated components that define what should be integrated, and 17 key principles around those components. It does so across three objectives, three primary objectives that the company should have. Then you apply that concept to the processes at the functional level. Then with that information, we're able to aggregate it to all levels of the organization, so we can get a complete understanding of the quality management system, from a perspective of management control. It's a nice process, because it's repeatable, it's scorable, it's not subjective. It's a really good appraisal process, to look at a quality system holistically.
Jon Speer: Folks again, I'm talking to Larry Mager. Larry is a Senior Advisor with UL EHS Sustainability. We'll be sure to provide his contact information, to accompany this podcast. You can reach out to him at Larry.Mager@UL.com, to learn more about the performance assessment that he's speaking to us about today.
So far, we've talked a little bit about the audit side of things. We just spent a little bit of time talking about the appraisal side of things, but now that I've got these five integrated components that I've compared against the key principals across the few primary objectives, then what happens? Now are we ready for that next step in your performance assessment?
Importance of Data
Larry Mager: Well, actually we are. Data has become so important. That's what everybody is understanding now, is that you have to start taking good care of how you gather and use your data. The FDA in particular, is very focused on using data to come in and better understand organizations. The idea would be that, we want to gather data to again, look at the quality of the output of those quality system processes, and do it in a consistent manner so we actually gather data that is around the compliance two requirements. This isn't just the verification that the components exist. There's two parts to compliance. This primarily looks at the adherence as well. While an audit looks at the requirements being verified that they exist within the organization, the data should be able to show, to a degree, the compliance, the adherence aspect.
The other thing though, if you have a process and you're measuring it, you know what you want out of that process. You should be able to, and we have a method where we can help organizations understand, how to measure each process within a quality system for effectiveness of required results.
Finally, because we're business, we need to be efficient, so we also have to be able to measure the efficiency of operational execution. The data is collected in that way to understand the quality system from the perspective of being compliant, effective, efficient.
The second part of data, is around the quality of the product that the organization produces. Certainly the FDA, this is the data that they use. They walk in with what they have, which is they're armed with the data from the MAUDE database. So, a MAUDE analysis of post market effectiveness and being able to understand that, but more importantly being able to understand that from an industry benchmark perspective against competitive products. That's the other part of the data analysis that we'll perform.
Jon Speer: There's two things. There's the effectiveness of your quality systems, that looks at the process side of things. Then the product piece is also important. You mentioned FDA MAUDE. For those of you that may not be familiar, the MAUDE it's an acronym. I'm going to confess, I can't remember what all the letters stand for, but basically this is the FDA database where adverse advents, MDR type of things, are reported, and FDA had record of that. That's in some respects, a measure of at least things that aren't going so well with products that are in the field.
Talk a little bit about that benchmarking side of things. Are you looking at benchmarking from both a product side and a QMS process side?
Larry Mager: No, the QMS process side is really measured in terms of percent effective and percent efficiency. We're not able right now to be able to benchmark, without collaborating with other organizations. The MAUDE data however, is reported. The FDA is able to align [inaudible 00:18:29] product families or competitive products and they can understand if your product is behaving differently than the rest of the market. While they look at it in terms of malfunction, injury and death, if your product is not in alignment, in any one regard, in other words, it's outperforming or underperforming the field in terms of, say malfunction, the FDA is going to want to know why is that. Why does your product malfunction more than similar products in the market?
There are many examples that can be used, but more importantly in recent times, and I'm talking about in just the year/year and a half, the FDA actually walks in the door with that information, and chooses those products against which to perform and pull the threads of a quality system investigation. So it's very important to understand what the post market data is telling the world about your product, and then if you can understand from that perspective, then you know what in essence is critical to quality, which products you need to focus on the most and which aspects of those products or how they're failing you need to focus on.
Jon Speer: For sure. I think the thing that I just picked up on that is ... In recent week or so, I've talked to a lot of companies who said, "Hey, FDA is coming. FDA is coming." Even if you don't have an FDA inspection on the books, you gotta know when you subjected to audits of various kinds, and I'm sure we can expect this from an ISO perspective, especially with some of the programs that are in place now with MDSAP and things of that nature, that the auditors and FDA investigators ... You know, the internet is a powerful thing. They're going to come with all sorts of data and information about your product, but also your competitor product.
This is the one thing that I think a lot of people don't realize. You may have a competitor, who has a product that has problems in the marketplace. That may in fact be a reason for FDA to show up to your door. It may not have anything to do with you. I think that some people are real naive about that.
Larry Mager: That's right. Put it in another light. So, think of the guidance that the FDA put out at the end of December, the 27th of December of 2017, and that was enforcement guidance. If you read that and just step back and look at it, it's basically called benefit risk. What they say is, if we have to take enforcement action, what we want to know first and foremost, is how did that quality system effect the product in the market? What they want from companies, is for companies to be able to tell them what's the benefit risk. In other words, you have to understand how your product fails post market relative to the risk system and the benefit right? They're going to want to know, what are the controls you have in place to mitigate that from occurring in the market?
Regardless of what the FDA does, isn't that really what we all have a responsibility for? We're in this business to help patients.
Jon Speer: We do. We are. I remind people of that all the time Larry, and you're spot on. We need to be aware of what's happening, certainly in our own world it's important. We shouldn't just be waiting on the bad stuff to happen. We shouldn't just be waiting on the adverse events and the complaints. We should be proactive. I know you're a huge fan of [Kappa 00:21:32], and that might be another topic for a different podcast, things that people can do.
But wrapping things up a bit on the performance assessment side of things, now that we've done the audit, and the appraisal and the analysis, now what? I'm guessing that there's some sort of summary or some sort of action plan, that we can now take to improve the situation.
Larry Mager: Yeah. We basically take the information from the compliance audit from the appraisal of the management system and the analysis of the data, and we actually are able to pull it in and process it through some software that we have, that then is able to push that back out into a really nice form of a performance report. We can actually give an organization a really good picture of those things that are of high concern, from every one of those perspectives. What that does, is it enables an organization to properly plan for improvement activity. To risk, rank, prioritize and then make a nice plan to go forward and target the things that are going to be the most impactful, in terms of improving the quality of the organization.
Jon Speer: Right. You know folks as Larry has talked about this, it has reminded me of some things that I've seen in practice, that frankly we should be doing anyway, as medical device companies. Remember, we are developing products for patients. We want to make sure that the processes that we have in place to design, develop, manufacture those products, are as robust, thorough, complete and efficient as possible. We want to make sure that our product performance is there as well.
This is truly the whole idea behind a quality management system. This is taking it way beyond focusing on the compliance and shifting your mindset and just checking a box, to becoming a true quality professional.
I'm going to thank Larry in talking to us a bit today. Again, this is high level but about his approach of doing a performance assessment. Larry any parting words that you'd like to share with the audience before we wrap up this episode of podcast?
Larry Mager: I think you summed it up quite well. We're in an ultra competitive business environment, and a very unforgiving regulatory environment in the Life Sciences Industry these days. Organizations just don't have the time, money, resources, energy to support inefficient, ineffective or non value added activities. We have to have a good way to understand that quality of our quality system, and how it impacts the operations, and a good way to go after improvement activity.
Jon Speer: Larry that's well said. Folks, this guy knows what he's talking about. He's been there. He's been in your shoes. He works with companies large and small, and his focus with UL EHS Sustainability, is all about making sure that your quality management system is designed in a way that helps you be a better business to develop and deliver safer more effective products. I would encourage you to reach out to Larry Mager, my guest on this episode of The Global Medical Device Podcast.
Before we wrap things up, that's another area where we at Greenlight, we strive and excel, is that we built a QMS software platform to help companies better manage this data: this information, your processes, your procedures, in a way that is in a single source of truth, in a platform that's designed specifically for the medical device industry. If you're interested in learning more about the Greenlight Guru software platform, I would highly encourage you to go to www.greenlight.guru and request more information from our website. Larry thank you once again for being my guest, on The Global Medical Device Podcast.
Larry Mager: Thank you for having me Jon.
ABOUT THE GLOBAL MEDICAL DEVICE PODCAST:
![medical_device_podcast]()
The Global Medical Device Podcast powered by Greenlight Guru is where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the world's leading medical device experts and companies.
Like this episode? Subscribe today on iTunes or Spotify.
