![podcast_Devin-Hubbard-Renee Rogge]()
Have you ever thought about the versatility of an eQMS? As it turns out, the use of one medical device eQMS solution in particular is extending across multiple sectors.
In this episode of the Global Medical Device Podcast, Jon Speer talks to Renee Rogge and Devin Hubbard, two academic professors/faculty members of biomedical engineering programs which are involved with the Greenlight Guru Academic Partner Program.
Jon, Renee, and Devin discuss their work together in the program and Renee and Devin share their experiences using the eQMS from Greenlight Guru in the classroom and how it's helping students engage and better understand medical device risk and quality management concepts and practices.
LISTEN NOW:
Like this episode? Subscribe today on iTunes or Spotify.
Some highlights of this episode include:
- Introducing Greenlight Guru to students has helped them to understand eQMS and how they can take the things they learn in theory and apply them in practice.
- Biomedical engineers need to be taught design principles, technical writing, and how to conduct tests and case studies with a risk matrix.
- Students should have a working knowledge and understanding of quality and regulatory affairs to prepare them for real-world.
- The traditional approach to teaching risk in an engineer setting is to focus on failure modes and effects analysis (FMEA) and design failure modes and effects analysis (DFMEA). However, this singular approach is not efficient and can leave some medical device risks undetected, resulting in harm.
- Case studies should include ways to be reasonable, realistic, and helpful in product design and quality systems to think about how other options could fail.
- It is challenging in capstone design to deal with risk. Capstone courses are overwhelming for students and difficult to teach. Tackle risk in manageable way.
- Students don’t like doing busy work associated with quality management and risk assessments, but there’s a reason for it: ensuring the safety of people using a device you designed.
Links:
Renee Rogge
Rose-Hulman Institute of Technology
Devin Hubbard
University of North Carolina/NC State - Joint Biomedical Engineering Program
GuideWire Podcast
FDA - Medical Devices
FDA - Design Controls
FDA - 510(k) Premarket Notification
ISO 14971: Medical Devices - Application of Risk Management
GuideWire Podcast
Greenlight Guru Academy
The Greenlight Guru True Quality Virtual Summit
MedTech True Quality Stories Podcast
Greenlight Guru YouTube Channel
Greenlight Guru
Memorable quotes from this episode:
“Using Greenlight in particular, introducing that to our students has really opened a lot of eyes and doors for our students at understanding what quality management systems look like and how they can take the things they are learning in theory and then apply them in practice.” Renee Rogge
“There’s things that our students are getting out of an eQMS platform like Greenlight that they wouldn’t get if we were trying to do a paper-based system. There’s just subtleties and nuances that are built in that we don’t really have to teach. They can learn by experience.” Devin Hubbard
“It’s not just a bunch of checkbox things they have to do. It’s critical for the development of the product that they communicate their ideas because someone’s going to pick it up and run with it later.” Renee Rogge
“Risk is bigger than just a thing that you do as part of your design. It is a thing that drives your design.” Devin Hubbard
“I think it’s easy to make risk boring. I think it’s hard to make risk engaging in a way that is educational for the students.” Devin Hubbard
Transcription:
Announcer: Welcome to the Global Medical Device podcast, where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge direct from some of the world's leading medical device experts and companies.
Jon Speer: Wow. On this episode of the Global Medical Device podcast, I get to talk to two academic professors, faculty members of biomedical engineering programs, both are involved with the Greenlight Guru Academic Partner program. I spoke with Renee Rogge. She's a faculty member at Rose- Hulman Institute of Technology, biomedical engineering department. And Devin Hubbard, he is faculty at the University of North Carolina, North Carolina State and joint biomedical engineering program. And we talk a lot about their experiences of incorporating Greenlight Guru, use of Greenlight Guru in the classroom to help teach complicated topics like risk management, as well as quality management systems. And they share a little bit of work that they've been doing on some studies around risk management and EQMS work. I hope you enjoy this episode of the Global Medical Device podcast. Hello and welcome to the Global Medical Device podcast. This is your host and founder at Greenlight Guru, Jon Speer. I'm excited about today. Of course, you probably hear me say that in just about every single podcast episode, which is usually the truth but today I'm especially excited because we have a couple of university professors joining us. First, let me introduce Renee Rogge. Renee is part of the faculty at Rose- Hulman Institute of Technology. Hello, Renee.
Renee Rogge: Hi Jon, great to be here.
Jon Speer: Fighting Engineers, my alma mater. Sorry, Devin. I might have a little bit of bias, but our other guest is Devin Hubbard. Devin is part of the faculty at the joint biomedical engineering program at the University of North Carolina and North Carolina State. Welcome, Devin.
Devin Hubbard: Thanks for having me today. Actually, I think you're the reason that I know Renee. This actually may have just come full circle.
Jon Speer: Yeah. And I've known Renee sort of tangentially for quite a few years and I've been on the industry advisory board for Rose- Hulman's biology and biomedical engineering department for quite some time. We interacted and then, gosh, I guess it was a couple of summers ago Renee's like," Hey, I'm going to be on sabbatical and I'm looking to get some industry experience." And she actually worked at Greenlight for a year. That was an awesome experience for me and Renee. I think you had a good time too, right?
Renee Rogge: Absolutely, learned so much. It was awesome.
Jon Speer: And that's part of why the three of us are talking today. Devin, I know folks may not realize, but UNC, NC State has been a university academic partner with Greenlight Guru now for, I think three years. Don't hold me to that. But something like that. And when Renee joined Greenlight during her sabbatical, we wanted to explore and expand and better define that program. And so Greenlight has been working with UNC, NC State, Rose- Hulman, Virginia Tech, University of Pittsburgh, University of Colorado, Denver, University of Texas, Austin, Mississippi, Santa Clara, I think is another.
Renee Rogge: Santa Clara.
Jon Speer: Valparaiso. A lot of different university programs and really the intent behind that is how can Greenlight augment or help you as professors and teachers of biomedical engineering better articulate, communicate, disseminate what's important about quality and regulatory. I guess that's a place I'd like to start is how has that experience been for each of you? And at the risk of not both of you talking to at the same time, we'll start with Renee.
Renee Rogge: Using Greenlight in particular has been, so introducing that to our students has really opened a lot of eyes and doors for our students that understanding what quality management systems look like and how they can take the things they're learning in theory and then apply them in practice. We're new to the scene using Greenlight Guru. Just this last year, we were part of that kind of inaugural class of your academic program. And I think the students really got a lot out of it. I learned a lot about how to teach students and make it practical and real. I'm really excited about continuing the program and all of the opportunities I have working with these faculties to figure out how to help our biomedical engineers and other engineers enter the medical device industry with some power and some energy to help make some difference and some awesome medical devices out there.
Jon Speer: Sure. Devin, I guess, high level summary, how has the experience been for you and the students at UNC and NC State?
Devin Hubbard: Well, first of all, we were with you guys pretty early on. And I think part of the reason that I keep coming back and asking for more is because we have had such a great experience. We've talked about this before. I think that there's things that our students are getting out of an EQMS platform like Greenlight that they wouldn't get if we were trying to do a paper based system. I think there's just subtleties and nuance that are built in that we don't have to really teach. They can learn by experience. Other than that, it's actually some of the research that we're working on and that's the whole reason, the questions and hypothesis we're developing were based on sort of anecdotal experiences that we've had. High level, awesome, great experience. I think I've had more than one student who's gotten a job because they've had experience specifically with Greenlight and I have one student who actually called me up not long ago who said," Hey, we started using Greenlight because I had a great experience within your classroom." Yeah, it's been good for us.
Jon Speer: Cool. These stories are great. And remember even before we started the academic program and one of the first times that I met Renee and her colleagues at Rose- Hulman, that was sort of the questions that were being asked of industries. What more can we as faculty at these biomedical engineering institutions, what can we do? What should we be focusing on? And I'm like, well, one of the big challenges is, you all do a great job of teaching engineers how to be engineers, design principles, technical writing and all of the things that go into being a good engineer but the role of an engineer, I think has changed quite a bit specifically from when I entered the workforce back in 1998 to today. Back then, you could just be an engineer, you could just learn design and technical writing and how to conduct a good test study and all that sorts of things. But today I think it's really important for undergrads and even graduate level students coming into the workforce to have at least some working knowledge or understanding of quality and regulatory. What are you hearing? I know you shared some of your comments, Devin, but what are some of the other anecdotes that you all are hearing either from students or from employers who are hiring your students?
Renee Rogge: I think a lot of it is just understanding how the different players work together and how important it is that there be open communication. And that's one area that the platform helps is with formation will shift from one group to the other. If they're working on design controls or risk or it's post- market, it shows how it does all fit together and work. They're not thinking I'm just going to be an engineer in a cubicle, they're seeing no, it's more complicated than that and why it's not just a bunch of checkbox things they have to do. It's critical for the development of the product that they communicate their ideas because someone's going to pick it up and run with it later and I think that helps them understand that.
Devin Hubbard: Yeah. I think I'll add to that too, if you don't mind. The other aspect is quality and regulatory affairs is a really moving landscape. And I think the ability for a student to get immersed in a system like this before they graduate gives them a huge jumpstart because not only when they leave and take a job, regardless of where they are in the medical devices or therapeutics realm, for that matter, they're going to have to get up to speed on all that. And I think if they can get over the hurdle of the basics of what is quality management? Why do we do it? How is that related to risk? Those sorts of questions, they can take time to figure that out and understand fundamentally. And I think giving that to the students before they leave gets them that much further along so they can then pick up at the industry specific components like, oh, here are the specific sets of regulations to your device or field or therapeutic.
Jon Speer: Yeah, personally, I can relate to that. I'm a little bit of an odd boy, I happen to have a chemical engineering degree, not a biomedical engineering degree. The fact that I entered into the medical device industry, it is a little bit of an outlier and an unusual career path for someone with that degree. But once I got into my role, for some reason, I think in systems. The engineering controls classes were some of my favorite because they made logical sense to me. And really when you get into a company, especially one that's regulated, a regulated industry like med device, it is a system of systems and you have to understand how all these different parts and pieces work together. And it made sense to me. Because it made sense to me, I was able to really grasp and understand the concepts of quality and regulatory much quicker, which frankly I believe led to more career opportunities and advancement sooner than maybe some others who didn't have that. I think that's really important. I know the two of you have been leading the charge on study. You've been doing a lot of work and I want to learn a little bit about that, but I guess before we dive into that, risk assessment or risk management in general. This is one of those complicated, convoluted, I think really challenging topics, at least on the surface to grasp. I guess this is where the medical device world is going, is that risk management is so important today. How have students grasped or really embraced concepts of risk management? Devin, I'll start with you.
Devin Hubbard: Yeah, sure. I'll a brief history of how I've approached and my teaching has evolved in that subject because it has changed a lot since I started teaching design. I think the traditional approach to teaching risk in an engineering setting is to focus on failure modes. You hear a lot of folks focusing on FMEA, DFMEA, that sort of stuff. Great. That's a helpful tool, however, one thing that I think is really important and I know that you've talked about on your podcast before, Jon, is the fact that that's not the only kind of risk associated with a medical device. Even when something's used normally and everything goes perfect, it can still cause harm. In fact, I use the example of a hypodermic needle all the time. When used correctly, a hypodermic needle will leave a hole in you, and so obviously that's a low risk kind of item compared to some of the other devices that we're working on. When I started, I focused on here are the mechanics of doing risk using FMEA. And I think the thing that changed for me when I started using, trying to get into the quality systems realm and then eventually switched to Greenlight was the evolution of thought that risk is bigger than just a thing that you do as part of your design. It is the thing that drives your design. And the FDA's entire design process is risk based design. And I think the nice thing about Greenlight is that the risk assessment built into that of course is obviously built on 14971. There's a bigger picture of risk. And the evolution of my teaching is focused less on the mechanics of here's the chart risk and more on how does this relate to your design? And what aspects of your risk can you do as soon as you start designing and not so much when it starts failing or as an afterthought? There's kind of the evolution of my thought process. And I think that gets our students at least anecdotally closer to what they should be doing if they go into the medical devices world.
Jon Speer: Sure. Renee, you and Devin have done a lot of work on sort of assessing and evaluating risk assessment and quality management system in the classroom. And so much so that it prompted some research. Can you talk a little bit about what prompted the research? And I want to get into, learn a little bit more about some of the study that you guys have been working on.
Renee Rogge: Yeah. I'll talk a little bit about why it's challenging in capstone design to deal with risk. There's a lot that goes into a capstone design course. You're just shoving a lot into maybe a two semester or two quarters or three quarters course. And so it gets a little overwhelming sometimes for faculty to think about how to teach it. And it kind of gets a bad rap like, oh, it's too hard to teach quality or it's too hard to teach risk. And so it's easy to kind of back up and just teach the basics of FMEA and say," They'll learn the rest on the job." And so I think that's where it Devin and Casey Howard and I kind of got together and thought about a way to help our colleagues and ourselves tackle risk in a manageable way that can be inserted into a capstone design course. What we started thinking about was kind of a case study approach to teaching risk and providing resources and kind of an instructor's guide that faculty could use as a part of their capstone course because not every capstone project will lend itself to a great example for risk assessment or risk analysis. Is that we're trying to help ourselves and our colleagues figure out ways to implement risk.
Devin Hubbard: I'll add to it, if you don't mind, Renee. We were on a call together with a bunch of universities from around the world on the subject of design, in fact, and in the conversation at the end, one of the questions that came up, so there was an institution that's working on developing some new teaching materials for the whole design process. Pretty well known institution and at the end of it, they were like, what are the things that frustrate you about teaching it? And without exception, every single other program on the call was like, I hate teaching risk. I hate teaching quality systems. It's so complicated and the students don't get anything out of it. And it's so burdensome, the amount of paperwork and documentation is just frustrating and the students just don't see the value in it. And so that was what prompted us to say," Hold on a minute, maybe we can study that and maybe we can actually make a difference and improve the way it's done."
Jon Speer: Well, what you just described is indicative I think a lot of engineers' experience in industry. I'll relate it at least one of my first experiences. I was working on one of the first product development projects of my career and going through our design control and product development process and following the different checklists. Because some of these concepts, at least initially are kind of foreign, at least they were to me. And I'm really encouraged that you guys are teaching these concepts now at the undergrad level, because I think this is really important things to understand. What are design controls? What is a username? What our requirements? How do they need to trace to your verification and validation? Is really important concept. And so I'm going through our company's checklist. I don't remember which design review it was but we had a particular design review and it said," Have a risk assessment." And to your point, Devin, this is back in the day where FMEA was cool. That's what we did. I'd never done one before. And my design review was the next day. And I'm like, oh crap, I haven't done this. I went and barricaded myself at my desk and spent probably that whole day, definitely that whole day doing an FMEA style risk assessment to check a box but getting no value out of it whatsoever. And after I did that, man, that's what always frustrated me early in my career is doing things that I'd got no value out of it. And I think this is a lot of people are really confused on risk management and maybe even quality systems. They don't understand the value.
Devin Hubbard: Well, to that point, I think maybe one thing that's really unique to the university setting that, and I hate to generalize, but students don't like doing busy work. Who does? And I think that while we are a required course, so maybe we have the luxury of being able to say," No, you have to do this." I think of my colleagues who have to teach other courses where they've got what students would perceive as minutia. Why am I doing this busy work? And I think that's the category that tends to be thought of when people are thinking of QMS and risk assessment. And that to me fundamentally, the problem with the perception is no, there's a reason we do this. And the reason is obviously because people are using your device and more importantly, you're designing it. You got to be thinking about that stuff. But if you're doing it to check a box, which is again, any students who are listening to this, I'm sorry, but I hate it when my students come to me and say, what do I do to get an A? It's like you're thinking the wrong way, think of this as an experience that you need to use to develop the skills you'll use when you leave here. Anyway, part of the impetus on my end for doing the QMS research is to understand how we can better deliver this as a package to students so it isn't a piece of busy work.
Renee Rogge: Exactly.
Jon Speer: Any other thoughts on that, Renee?
Renee Rogge: No, I think Devin captured them beautifully and that's fine. I agree 100%.
Jon Speer: Yeah. I've worked with enough engineers that engineers are very good at thinking about ways that things can fail and ways things can break and that's good. That's a good skillset to have. The other thing I think engineers are pretty quick to gravitate towards is the failure mode and how it can lead to death. Challenge an engineers sometime, all right, paper cut. And they will in less than three steps, tell you how you can die from a paper cut. A little morbid at times, but we need to think more big picture. And I think we need to do a better job of understanding how to use that critical risk based methodology and thought processes to better inform the design. And I think this is where the industry is behind the times, hugely behind the times, because we're not really applying in a meaningful sort of way risk based methodologies to improve product design.
Devin Hubbard: Well, to be fair, I think industry is probably ahead of general academia on that. If only because industry's practicing it. And don't get me wrong, I guess what I'm saying here is we take our cues from industry or we should be for our students who are going to go that direction. And so the best we can do in many cases is to try and keep up with trends. And so again, at the risk of sounding maybe overly cynical about academia, I think that's something that's important. The fact that you point that out makes me think, well, how far behind am I? I don't know if you feel that way, Renee.
Renee Rogge: Yeah, that's a good point. I like where you were going with Jon, talking about everything leading to death. And I think that's one of the things when we were coming up with this, the case studies we wanted to back away from that and think about ways to be reasonable and realistic and show how it can help you in product design to think about these other options for how things can fail.
Jon Speer: Absolutely.
Devin Hubbard: Yeah. And credit to Renee, because you're the one who figured out, you came up with the case study that we ultimately landed on, which involves a pedicle screw. And it's sort of like, somebody's not going to die from this, I don't think. Again, you can always find a way but I think it's a great example of a device that is medium risk. Like on the spectrum of device classes, we would put it there.
Jon Speer: Yeah, and we'll come back to that here in a moment. I want to take a brief break. I want to remind folks, I'm talking with Renee Rogge. She is a faculty member at Rose- Hulman Institute of Technology and Devin Hubbard, he is faculty member at University of North Carolina and North Carolina State joint biomedical engineering program. And while I'm taking this break, I want to remind folks about Greenlight Guru, the only medical device success platform in the market today. It's been designed specifically and exclusively for medical device professionals by actual medical device professionals, which seems crazy, but it is true. We have workflows and solutions that we manage. Design controls, risk, your documents and records, as well as all of those quality events, things like CAPAs and complaints. And it's all in a single source of truth, a single platform for you to use. If you'd like to learn more about the Greenlight Guru medical device success platform, I would encourage you to go to www.greenlight.guru. We'd love to have a conversation with you, understand your needs and requirements and see if we have solutions to help you. Again, www.greenlight.guru to learn more. All right, so let's talk about this study design. Devin, you talked about pedicle screw and it's sort of Renee's brain brainchild, so to speak. Renee, talk about what is the study that we've alluded to a couple times so far?
Renee Rogge: We tried to find a topic that would be something that would be a low barrier to entrance for faculty and students to get up to speed on very quickly. They don't have 10 weeks to learn about pedicle screws so we tried to pick something very mechanical that has some risk associated with it that we could develop kind of a case study on and also allow students and faculty to interact with the FDA website and use recall notices or mod events to kind of reinforce the things that they may talk about in classes naturally. What we did was we picked on the pedicle screw, we found a recall notice that identified some potential problems that were being seen in an operating room. And so we decided to put our students and our faculty in this scenario where they have a product, a similar competing product on the pedicle market line that they have used the company that has the recall notice to hit part of their 510( k). And so now they're saying," Hey so the premise is, is there something we need to be worried about because there's this recall notice that has been issued?" While your engineers are gathered together and said," Here is our risk matrix, a little section of it for the intraoperative concerns that have been addressed in that recall notice, what do you think kids? Tell us what we should be concerned about here?" That's the background for the study.
Devin Hubbard: And a couple of things to that if it's okay. We spent a lot of time trying to figure out how to make something like this engaging. I think it's easy to make risk boring. I think it's hard to make risk engaging in a way that is educational for the students. Here's where it gets interesting. The way it's structured is that it's kind of like a whodunit. We intentionally leave a lot of for students explore all the options for what could have gone wrong. The mod events that Renee has pulled out will direct student teams down different pathways. For example, the original recall that we use happens to involve a tool. I don't want to give it away because if I have students listening to this, I don't want them to get the answer but it involves sort of a tool that's used in the procedure. And the recall notice involves a strange sound associated with use of the subject medical device and cracking. And so, you could go down the route of what's missing from the risk matrix involving that? And then some of the other mod events point students in different directions. Now the goal of this case study isn't necessarily for students to identify the actual cause of the recall necessarily, it's more to get them to understand how to use the resources that are available to them to kind of think. And so that's where I think it gets kind of fun. What we did, and this was fun too, we actually got a group of students together and were like," Hey, let's test this out." We made a student guide that basically gives them the bare amount of information they need to get started and we basically hit a stopwatch and say," All right, you got 15 minutes to figure things out, ask questions as you need." It's not quite that undirected, but that's in short what it is. And we were surprised at how well our students did. They really, first of all, they got to where we wanted them to, eventually. They were asking the kinds of questions that we were hoping they would ask and they gave us great feedback about how to improve the actual instructions. That's kind of where we are with that. And then we use what we learned from that to create an instructor guide that goes with it. We assumed that most folks don't have a background in pedicle screw design and probably aren't intimately familiar with the recalls and mods that came out with this particular example. And so we created an instructor guide that kind of walks them through the process, has answers to questions they might expect and sort of is meant to give instructors the peace of mind to know they don't have to spend five hours getting up to speed on this. They can walk into class with maybe 20 minutes of prep work and be ready to go.
Jon Speer: Wow. That's cool.
Renee Rogge: With the added bonus that we provided a section of a risk matrix, which I think is helpful to faculty to see that and be able to present it so they're not starting with a blank piece of paper and neither of the students and the students were actually really engaged with that matrix and asking questions about what was incomplete or what was missing? And they were really engaged with that. And we think it's because it wasn't just a blank piece of paper, that they had something to kind of riff off of and talked with each other about, which was really fun.
Devin Hubbard: And thanks to you, Jon, because I think one of the first places we went was you guys. We were like," Hey, can you give us feedback on this risk matrix? Also, do you happen to have an example that we could borrow?" Yeah, thanks for that.
Jon Speer: Well, the experiences that you are sharing, they really mimic and mirror what we often hear from a lot of startups. They know enough to realize that, okay, these are things that I have to do. I cannot ignore. They may or may not have any working experience with risk or design or any of these sorts of things from a regulated point of view but they're always looking for help and guidance and direction and examples and all that sort of thing. Because I think once it clicks, it clicks. Sometimes it takes a moment to. I can imagine that's super rewarding for each of you when you actually see the light bulb go off. I assume that that happens quite often.
Devin Hubbard: Yeah, I think so. I think there's the analogous scenario to seeing that click of understanding with this kind of thing because engineers tend to be drawn to puzzles. And we've given them a puzzle. I would argue it's a pretty fun puzzle to work out. And at the same time, I think it hits all the notes that we're trying to get to. Here's how you use it, now go do it.
Renee Rogge: I think one student actually did use the word fun.
Devin Hubbard: No, you're right. They did.
Renee Rogge: Which is unheard of.
Jon Speer: I'm confident that your students and the rest of the students that have been involved in this study, maybe not all of them, but I'm confident that quite a few of them, once they get into the workforce, they'll be like, oh yeah. They'll have a brighter light bulb go off then. I just know from unrelated topic, I can remember my high school English teacher, I did not like her in high school. She was very rigid about what was important and that sort of thing. But man, when I got to engineering school and I had to write technical writing, that discipline like kicked in. It was like, oh wow, this makes a lot of sense. And now I look back and okay, I didn't appreciate her at that time or that activity at that time. But man, looking back that was so invaluable.
Renee Rogge: I think that's one of the things as faculty and as capstone instructors, we may need to keep in mind too, is the end goal is not perfection and 100% knowledge of all risk. It's really just getting their feet a little more wet, not just dipping a toe, but getting them some experience that they can draw on, like you're talking about Jon, just draw on it later. And I think we have comfortable with that and know that that's going to help them in the long run.
Devin Hubbard: Yeah, I think as design instructors, we face a unique version of that problem too. Our students in many ways, it's their last opportunity to fail catastrophically and not have something really bad happen. And I go as far as to tell that to my students and I think instilling that try stuff and learn from it in the classroom is really important. And I do think that this is providing that sort of fail safe. It's like, okay, this is a simulated scenario, but it was real. This actually happened and we want you guys to figure it out, which is kind of cool.
Jon Speer: Well but that's hard for engineers because engineers do not want to fail anything ever. And man, I do not want to be wrong. And they always think that they are the smartest person in the conversation.
Devin Hubbard: Wait, we're not?
Jon Speer: Shh, that's for others to determine. All right. I think you also did another study or is it the same study? The other study, was this on the EQMS side of things?
Devin Hubbard: Yeah. I was going to ask if you wanted us to tease that information a little bit?
Jon Speer: Yeah, sure. Talk about that too.
Devin Hubbard: I'll set it up and then Renee, I'll sort of pass it to you. When we first started using Greenlight, anecdotally we started noticing at least what we perceived as better understanding and application in our students. And it was like, wow. They seem to understand the link between risk and design control. They're actually using risk assessment to add new features and safety, labels, et cetera, to their device. And they seemed to be just seamlessly adopting it. And there was no special thing that we really changed about our instruction. The question then arose naturally, is this anecdotal and perception? Just us perceiving it? Or can we measure this somehow? And so that's what the goal of the study that we designed is, is to assess quantitatively if there is any difference in performance, student performance and understanding of medical device design and quality systems, that's the result of using Greenlight Guru. And Renee may be frozen. Oh, you're here. You're just holding really still.
Renee Rogge: I'm very still.
Jon Speer: Wow, very pensive.
Devin Hubbard: That's so impressive. Me, Renee and Casey Howard from UC Denver started putting our heads together on how we could study this. And so we designed a quantitative study assessment that we were going to use that we have used and we're currently collecting data on the understanding and performance of students when they use QMS. Renee, if you want to pick up, I don't know what you want to add. Go ahead.
Renee Rogge: And we also say that the concept of the pre- post inventory kind of came out of talking with other universities who were interested in using Greenlight but worried about that cost benefit. You're going to have to change a little bit of how you teach, you're going to have to integrate that and what really wanted to know, is it going to be worth it? And we had a lot of evidence, but like good researchers, we wanted to provide some concrete evidence that truly the time that they do put into teaching and helping students learn about Greenlight will pay off. And so we wanted to use this concept inventory as a way to do that.
Devin Hubbard: Yeah. We inserted a number of controls in this study. Educational studies are always difficult because you're engaging inside of somebody's brain. And so the methodology is pre- selected. And without going into the details, basically what we did is that we did a pre- post study, which in this context means that before students are exposed to an EQMS platform, we asked them a series of questions. And this battery of questions is broken into sort of three sections. There's a section that just collects information about their background and experiences thus far. Have you worked in a company before that uses a QMS platform? Obviously that might change your understanding of it without coming into the classroom. And then we had a section on that's called a concept map. A concept map is used to visually gauge somebody's understanding of a subject. The way it works is that you give somebody a word bank. And in this case we gave them a word bank of terminology that's used in medical device design. And we asked them to draw the connections between those words visually. They write all the words onto a piece of paper and they draw lines and where the lines go they basically write in what it is that connects risk and design controls. And so what you'll find is that for people who understand the subject better, there are more lines. Somebody has a deeper understanding of a concept, they will link things together more completely. There's concept inventory to try and assess their understanding of the subject matter at a whole. Then we have a section that gauges explicitly did this change your understanding? On scale of zero to five kind of thing. And then we also were curious about ethics. We created in collaboration with some others that are ethics experts, some questions that were meant to gauge whether students were really internalizing regulatory and quality systems as a fundamental practice to their ethical decision making. For example, we asked them a question sort of agree, strongly agree, moderately agree or disagree, things like I should sometimes accept a lower standard of device quality, safety, efficacy for the sake of lower costs. We're baiting them a little bit with some of these questions. But again, this study was designed to be done in two parts. Same set of is given to the students before they use Greenlight and the same set is given after they use Greenlight.
Jon Speer: Okay. I'm kind of fascinated by that. I haven't obviously seen the results and I think the study is still ongoing, right?
Renee Rogge: Yeah. We just got the post data for our institutions at the end of the semester. We're still doing a bunch of crunching. Fun story about the word bank that Devin talked about. We didn't initially have a word bank, sometimes concept maps are just put all the things you think of related to this topic. And when we beta tested that with just professionals, it was thousands of different words so we decided to narrow that down. For any listeners who are concept map experts, they know that the word bank isn't necessarily part of it but we had to do that otherwise we were going to end up with some really interesting data to look at.
Devin Hubbard: Yeah, I think that you just added something that's really important. We started off on this track with just three people. And then we were like, we should really get some industry folks who actually, Casey, Renee and I pulled together, actually I think it was mostly Casey, the folks from her industry network to come in and give us, just inform our direction for what we should be assessing. Because I think at the end of the day, the hypothesis is, is Greenlight in the classroom affecting our students' ability when they leave and go get hired. And so we needed to make sure we were asking questions and probing our students' skills and understanding in the areas that are relevant to industry.
Jon Speer: Yeah, absolutely. Well, first and foremost, I'm honored that Greenlight and I have had an opportunity to work with the two of you and the other academic institutions. I'm so appreciative of what you guys have taken and run with. Just the fact that you're doing some scientific research and analysis of some of these complicated topics. And it's just when you first shared that with me some ago, I was a little bit in awe but I didn't really get it. And then as we talked further and I'm like, oh my goodness, this is amazing information. I'm really looking forward to the study when those items are published. And I hope that we can disseminate those to the world. I don't know what your plans are for that, but if and when we're able to do that, I'd love to share that with the world.
Devin Hubbard: Yeah. We just got our data back, so we're going to start analyzing and we are planning to publish it. Obviously it's going to be the front page of Nature, I'm pretty sure.
Jon Speer: Well, I hope it is.
Devin Hubbard: That'd be great. No, in all seriousness, I think first of all, I don't think we would have done a study like this if we didn't think there was something to look at. And to be clear, I hadn't seen this kind of research when we started using Greenlight at all. It didn't even cross my mind. It was like, oh great. We've been looking for an electronic platform to do this. Perfect. There's this Greenlight Guru thing we're going to use it. I think literally after the first semester of using it, it was we need to drill down on like going on here because it was pretty clear that our students were having a much better understanding of things overall and nothing else had really changed. That makes me sound like a really stagnant teacher but what I mean is the material that was really the biggest change that we had made. And so that's what prompted it. And then it turned into this, how do we measure all these things? And it turned out, we're not the only people interested in this. And so all the institutions you listed earlier, they're also interested in this problem. And then it turns out that it's not just those institutions as well. There's a general stigma about quality systems and where do you draw the line for what you teach and what you don't teach in the classroom? I think Greenlight offers something that's a little unique.
Jon Speer: Yeah, I appreciate the opportunity to work with you all. Any final words before we wrap up this episode of the Global Medical Device podcast?
Renee Rogge: Oh, if anyone's interested in seeing the case study or working with us on another case study for risk, we would love to collaborate where we lack new ideas and we like running with stuff. Anyone should feel free to reach out to us if they want to be involved as either a pre- post inventory, as either a partner using Greenlight or as a control. We're also looking for people willing to play along with us.
Jon Speer: Yeah, and I'll also extend the offer for other academic and universities who, you're teaching biomedical engineering either at the undergrad or graduate level, if you'd like to learn more about how you can use Greenlight Guru in the classroom, reach out to me, let me know. I'm pretty easy to get a hold of Jon, J- O- N. speer, S- P- E- E- R @ greenlight. guru. And let me know a little bit about the courses you're teaching and your institution and that sort of thing. I expect that we'll be announcing our 2021, 2022 academic partner program participants. That's quite the mouthful there, but anyway, here very, very soon. Looking forward to continuing the program, but reach out to me, let me know. Speaking of teaching and learning and all that sort of thing, want you all to also to check out something, a wonderful resource that we've recently made available at Greenlight, it's called the Greenlight Guru Academy. Simply go to academy. greenlight. guru. Within the Greenlight Guru Academy, we have a variety of courses on a variety of topics. We have some that are free and that may take you 30 minutes to go through. We also have some premium, some of the best courses I've ever taken myself. Yes, it's going to cost you a few hundred dollars, but man, is it worth it. I particularly took the risk management course and I thought, I knew a lot about risk management and I paid money to go through this course. And man, I learned so much. It's so worth it. Go to academy. greenlight.guru, you can see all the different course offerings, the free ones, the for fee ones and that sort of thing. Just be sure to check that out. Devin Hubbard, faculty at University of North Carolina, North Carolina State joint biomedical engineering program, thank you for all your work and thank you for being on the program today.
Devin Hubbard: Thank you. Yeah, it was an honor.
Jon Speer: And Renee Rogge, faculty at Rose- Hulman Institute of Technology, the Fighting Engineers in beautiful, Terre Haute, Indiana, thank you as well.
Renee Rogge: Thanks, Jon. Thanks for all the support you give us in academics. We appreciate it.
Jon Speer: Absolutely. And folks, thank you for being loyal listeners of the Global Medical Device podcast, the number one podcast in the medical device industry. I forgot to mention there is another podcast you need to check out, GuideWire podcast by Devin Hubbard. Go check that out. Devin, where can people find the GuideWire podcast?
Devin Hubbard: Yeah, any streaming service actually run every one. But GuideWire podcasts, if you go to guidewire.unc.edu, you can check us out and learn more.
Jon Speer: Yeah, it's good stuff. I highly encouraged people to check that out too. But thank you all for listening. Continue to spread the word about the Global Medical Device podcast. And as always, this is your host and founder at Greenlight Guru, Jon Speer.
ABOUT THE GLOBAL MEDICAL DEVICE PODCAST:
![medical_device_podcast]()
The Global Medical Device Podcast powered by Greenlight Guru is where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the world's leading medical device experts and companies.
Like this episode? Subscribe today on iTunes or Spotify.
