Regulators of IEC 62304 have put a lot of energy into normalizing how to handle SOUPs (Software Of Unknown Provenance) for software of classes B and C (software that is in a position to potentially harm people in a non-benign way). The definition says: “Software that is already developed and generally available and that has not been developed for the purpose of being incorporated into the MEDICAL DEVICE (also known as “off the-shelf software”) or software previously developed for which adequate records of the development PROCESSES are not available”. To sum up: everything that hasn’t been built according to the norm.
What I’ve seen in the trenches indicates that this distrust in SOUPs is a bit misplaced: in my projects, carefully chosen libraries contain several dozen times less bugs than home-made code before verification. Why?
- Released libraries are finished software.
- They are used by many more developers than code being called in only one context and thus have a higher probability that bugs have already been found and fixed.
- The rise of open source software along with excellent processes (automated builds, TDD, gitflow with systematic reviews of pull request…) and psychological motivators (the name of developers permanently and publicly attached to every commit incentivizes perfection in code) has dramatically increased the quality of free libraries compared to ten years ago, when 62304 was first released.
But I understand the theoretical need of regulators: if there was no SOUP policy, it would be too easy to pretend that a major part of the code is a SOUP and not apply the regulation at all. I can’t imagine a norm that doesn’t think that what’s coming from outside of its jurisdiction could be better.
Norms are norms and auditors are paid to verify compliance to a norm, not to argue about how well or bad the norm was written. I’ve heard that SOUPs are one of the top favorite areas for auditors to look for defects in your implementation of IEC 62304 (the other one being risk analysis): be warned.
So how do we handle this mandatory and not-so-useful activity? Here are a few hints to maximize productivity.
- You need a list of dependencies and their versions. In some programming environments (nuget, bower, npm…), there is a clear list of these dependencies and their versions (package.config, package.json, bower.config…): try to generate the SOUP list from these files.
Dependencies in a bower.json file - It’s a good idea to take advantage of this list to perform a thorough inventory of licenses and do what’s required to be clear. For example, many open source libraries require your software documentation (typically online help) to quote them. And maybe you’ll find one or two that’s not free for commercial use and that needs to be replaced – the sooner the better.
- 62304 requires specifications for SOUPs, including performance criteria. This is a tricky business: some of your SOUPs are a lot more complex than your medical device (the base library of your favorite language, the OS): you can’t possibility retro-spec them entirely. My preferred approach is to document the requirements of the main behaviors of the library that you actually use – a projection of its features on your special use case.
- You should always try to wrap the external dependencies of your code in, well, wrapper classes. This prevents this external namespace to creep all over your code. It helps to easily change the library with another functionally similar implementation someday. In the context of SOUPs, the public interface of the wrapper makes very clear which part of the SOUP you use, and which part you don’t. This can serve as a boundary to limit your SOUP specification effort.
The SOUP wrapper acts as a Facade and helps limit the specification and unit testing effort to features that are really used. - 62304 requires you to test these requirements. That’s something developers spontaneously do when choosing a library: make sure the library works, test a few edge cases. But you need to do it again every time you upgrade the library. For the latter reason, I strongly suggest unit tests that you can link to the specification (so that they end up in the traceability matrix) and use to test the mandatory performance requirements (for example by using the MaxTime attribute in NUnit). These unit tests will help you make sure the next version of the library works with very little extra effort.
- When they are available, you could run the unit tests of the library itself and use their results as a proof of quality. You will still have to deal with writing your own requirements and linking them to the tests. In practice my teams often have had problems with libraries having a few failed tests related to features we didn’t use, which triggered cumbersome justification; in this case we just skipped the library unit tests.
- You are required to perform a risk analysis of your SOUPs and add mitigation strategies as required. This is theoretically a good idea, but I’ve often found it very difficult to put in practice with general-purpose libraries, because their impact cannot be bound to a single feature. In some cases – databases, ORMs, mappers – almost any features could potentially be compromised. As always with risk analysis, there is a temptation to assess every possible failure mode, which would lead to an overwhelming analysis that never gets finished. My advice here would be to trust your gut feeling and choose a selected handful of risks where the brainpower consumed performing risk analysis will be most valuable. There are less failure modes in SOUPs than in your code; use your time on the risks that really threaten patients. Don’t get stuck in an impossible thorough analysis of everything that could possibly go wrong in things that are more complex than what you produce.
- You are also required to perform a list of known bugs and assess the risk your software incurs because of them. It’s a demanding endeavor: in practice my projects tend to use dozens of libraries, some of them have hundreds of bugs, and others don’t publish bugs at all; when they do, it is often difficult to tell which versions of the library have the bug without testing them. I suggest you don’t waste your time with this before the end of the project because you are likely to upgrade your libraries until then and because more known bugs are likely to be closed with newer versions. The ROI of this activity seems very low. I would be glad if this requirement was stripped out of the next version, or adapted to be more cost-effective.
- Operating Systems are a special kind of SOUP. Of course you don’t want to retro-specify and test what it took your vendor decades of work with thousands of developers. But there is an alternative approach. These days, a lot of emphasis has been put on cybersecurity for medical devices – and this is good, patient data is sacred and hackers are on the brink of cyberwar to get it. You must harden your OS – and maybe brand it along the way. My recommendation would be for you to specify, document and test the hardened OS and not the base OS. This way, the OS spec is really useful and has a realistic scope.
- SOUPs of SOUPs. Developers often ask how they should handle SOUPs of SOUPs – the dependencies of the libraries themselves. Of course you can’t handle all dependencies recursively, you would be overwhelmed. Treat your direct dependencies; their own dependencies are an implementation detail. The tests that verify the requirements you wrote for what you use in the SOUP will exercise the lines of code of the SOUP dependencies that you actually use. Their possible failure would be ways to trigger the failure mode of the level 1 SOUP that you already considered in your risk analysis; you don’t need to analyze them separately.
Whatever the hardships in producing the required documentation, resist the temptation to code for yourself what others have done. Reinventing the wheel is a waste of time. Remember, your goal in agile development is customer feedback and delight, not library writing. The thrill of writing cutting-edge technical code is what I suspect entices many developers into rolling their own version of existing stuff, and not good project governance; this an area where a responsible mindset – adult self-supervision – is of particular importance. Developing with an agile mindset implies going as fast as you can by removing waste; missed opportunities of good reuse are horrendous waste. Your immature code will always more buggy and more poorly designed than a library written by people dedicated to it, maybe working on it full-time for several years, and that has lived in production for several releases in many different contexts. In this regard I think that the writers of 62304 have done a very dangerous job in discouraging people to use reliable libraries and creating an incentive to write brittle home-made code instead, which would have a very negative effect on overall medical device reliability and safety. A few month ago I stumbled upon a concrete example of this : a developer I know decided to write his own XML generation routine to avoid the lengthy, boring and absurd (according to him) process of documenting an off-the-shelf library. Don’t ever do this. SOUPs are good. Always use SOUPs when they make sense. Accept the pointless burden (automating as much as you can) and write the required doc.
Let me take advantage of this tribune to deeply thank all the open source contributors in the world.
Agile Med Dev 
This is a great and very informative post. Something that is still not clear to me around libraries is why and when the library needs to be considered as SOUP. You touched on this in discussing second order libraries, that they’re just “implementational details”. Does this not apply to the bulk of actual code? Can an external library be delegated to an “implementational detail”?
LikeLike
Hi Henry,
Thanks for commenting.
I believe SOUPs are libraries developed outside of a SDLC complying with IEC 62304. In practice: all the libraries and tools that are directly referenced in your medical device codebase (think package.json on the front-end or packages.config in .NET).
“Does this not apply to the bulk of actual code? Can an external library be delegated to an “implementational detail”?”. I don’t think so.
Treating second order libraries as implementation details makes sense in that you still specify and verify the first order library, as well as manage the risks it poses and its published bugs… Suppose a flaw was introduced in a second order library, you would still be able to catch it thanks to all you put in place to manage the first order library.
I’ve heard of projects trying to outsource their entire medical device codebase and treat it as SOUP. It doesn’t work this way. Auditors are not stupid. They can tell the difference between a Github lib with 10k downloads and an ad-hoc medical device codebase or internally shared library. They would understand someone is trying to fool them and might retaliate with very harsh reports.
Please keep in mind that I was only a fellow medical device developer; my advice is not bulletproof. Interpret with caution.
LikeLike
Well that’s a very interesting article!
LikeLike
Thank you for your interesting article, and in general for your awesome blog! I’m new to medical device development, and your blog has helped me a lot. We are considering using web technologies to develop a GUI for the control of a class IIa medical device, with mostly class A, but some B components in the frontend. One thing which scares us off is the large number of open issues on GitHub, and the number of required dependencies for these technologies. Say we go with a stack of Electron + Angular + Bootstrap, we are already in the 10k+ issues to analyze, which we cannot look at individually. I was wondering: would you consider it generally a bad idea to use such technologies, as opposed to say SDKs such as WPF, JavaFX, or QT which are more complete of a package, and have less open issues? I’m hopeful that you have experience in that, since you mention a “package.json” in the article above 🙂 Thanks & have a nice day!
LikeLike
Hi Ayedo,
This is difficult topic because SOUPs is the area of the norm where it is IMHO totally wrong. There are several orders of magnitude less bugs and risks in the libraries you use than in your own code. Please don’t stop using carefully chosen libraries – they increase the safety of the medical device software. You users will thank you.
So what do you do? You have to go a bit schizophrenic here: split product quality and regulatory quality. Work on product quality by picking good libraries, not reinventing the wheel, benefitting from code polishing by being used in thousands of different use cases, saving time for your people so that they can track difficult bugs and rogue risks.
Now you need to handle regulatory quality. Craft a minimal report that will make auditors happy without wasting your time on tasks that do not increase the safety of the medical devices. Here’s my advice (I’m speaking freely since I left the medical devices industry. My opinions here are my own and not linked to the activities of my past or future employers):
• Briefly skim through the list of bugs. Tried to locate areas where they might have an impact on your product. Expand with a risk analysis for a couple of interesting ones. Invest 2 to 8 hours in this, not 2 to 8 months.
• Write a report with the exported list of bugs, peoples, dates, signatures. Auditors will be looking for overlooked activities. Give them something to eat. It’s hard to argue that an activity could have been done better or more thoroughly.
• Don’t do it often. Just before major submissions or releases.
• Maybe you can strengthen the SOUP report in other areas that are more beneficial to patient safety. Most open source libraries come with large suites of automated tests. Use these as specifications (BDD interpretation) and attach the tests runs as proofs of quality or the library.
Some additional thoughts about this:
• The medical devices industry has major issues with cybersecurity. I see hospitals grinding to a halt because of ransomware every week in the news. And what’s one of the top ways to make life difficult for hackers? Up-to-date dependencies. Meaningless activities around SOUPs create a barrier to update. Medical devices end up easier to hack. We must fight this nonsense.
• I worked on a serious product for the financial industry that could perform up to half a billion dollars trades. Wanna know how we handled SOUPs upgrades? Automatic upgrade of all libraries to latest every week-end. Zero documentation. Zero. How did it work? With a crazy test coverage. This is more effective than reading bug lists.
• “SDKs such as WPF, JavaFX, or QT” are not safer. It’s just that the bugs are harder to find – (although WPF has gone open source with .NET 5, see https://github.com/dotnet/wpf).
• I’m surprised that there’s a part of your front-end that’s riskier than the back-end. Maybe you could change your design so that the risky decision is taken at the core of business logic (think domain model of the back-end modeled with DDD or CLEAN architecture) and not in the front-end. This is because your front-end is likely to change more often than your core domain. Making your front-end class A would be a strategic move.
• If you’re struggling with Bootstrap… Are you sure you really need it? These days with css grid and flexbox naked css is a viable option.
Good luck!
LikeLike
SOUP assessment is simply checking that the SOUP you are using is good SOUP and will remain so over the product lifetime. I disagree that it doesn’t add value. Many old libraries have been found to be buggy or unsupported or have security vulnerabilities and I certainly don’t want that in any medical devices that are used with me!
The question really should be is why don’t ALL software development companies do diligence in checking that the SOUP they use is appropriate! See the NPM/left pad debacle: https://www.theregister.com/2016/03/23/npm_left_pad_chaos/
LikeLike
Hi supersh, thanks for engaging.
Agreed, some form of vetting process is required for dependencies, especially for medical devices. The difficult part is to find an efficient one and I doubt IEC 62304 is pointing us in the right direction.
For example, the Angular project on Github has 1700+ open issues. You could spend several years analyzing each bug. They would probably be opened and closed faster than you scrutinize them.
Another example: Linux. It has more than 1 million commits and close to 12 000 contributors. You can’t analyze that, it’s just too big. Yet it’s dependable and runs thousands of mission-critical apps around the world. You can run medical devices on Linux. Definitely safer than rolling your own OS. But you can’t make it fit into a SOUP risk analysis report.
The process I see in non-regulated environments towards SOUPs is the following:
* Analyze the structure of the open source project (number of contributors, well-known organizations, rate of commits) and pick solid ones.
* If possible, don’t rely on public feeds but on feeds vetted by security groups to limit vulnerabilities and supply chain attacks.
* Cover the code with automated tests (library alone and integrated with the rest of the app), which is anyway the only way to be sure that anything works. Incidentally, a good build pipeline will catch the npm package issue mentioned in the npm apocalypse article before it creates any damage.
* I have never heard of organizations reviewing bug lists or retro-specifying libraries if they’re not forced to do so. Because it’s just not effective. Those are the areas of IEC 62304 that should be rewritten, I believe. (Let me clarify that most of the norm is really helpful in avoiding reckless organizations to create a mess that would kill people. I’m just criticizing a portion of the SOUP handling process)
I’m still coding and reviewing code every day. I seldom see bugs in well-chosen libraries (say 1%). When I do the bug is often already patched and an upgrade is enough to fix it. We shouldn’t waste our efforts where there are less bugs and less risks. Just the appropriate amount. And keep energy for the remaining 99% of issues, which are hidden in home-made code with few users and few different use cases. The real goal is to make medical devices safer within constrained resources.
LikeLiked by 1 person
I’ve been looking through many articles, but have not found an answer to my question about the possible need for keeping SOUP items up to date:
Let’s assume you’re in the middle of a project developing Class A software, and you’re using the Angular framework. You started with the then most current version of Angular. But as it’s a multi year project by the end of the project the Angular version you are using will be out of support already.
My gut feeling tells me that I have to update to a supported angular framework before releasing the software as a medical software.
But I haven’t found anything in 62304 that tells me to do so (which makes sense, because for a lot of SOUP you might not have a fixed release cycle or any support at all).
But I’m still wondering: Am I missing something?
Thanks for the article, by the way!
LikeLike
Hi Angelica,
Thanks for engaging and for your support.
As far as I know, nothing in 62304 mentions updating SOUPs. I guess that using outdated, non supported libraries could be classified as a risk. But nothing directly encourages you to keep your dependencies up-to-date.
But it doesn’t mean you should not update them. Number one reason is security. Upgrading fast and often is paramount to any software connected to the internet. Other reasons related to the dev team would be to benefit from the latest features and bug fixes, and keep the codebase attractive for recruitment and retention.
So what should you do in practice? I would suggest writing high-level documentation (requirements and risk analysis) and using automated tests to prove the SOUP works, so as to make the administrative burden of upgrading the SOUPs as light as possible. On the technical side, you can use npm audit to list vulnerabilities (in the front-end), or tools such as dependabot or renovate to automate the creation of pull requests to update the SOUPS (which you could blindly merge provided you have sufficient automated test coverage). Whatever you do, be sure to update all your SOUPs often (weekly if it’ fully automated, quarterly if it’s manual) and pay attention to published vulnerabilities.
Remember the recent log4shell mayhem? You don’t want that unpatched on a medical device more than a few days.
LikeLike