Leveraging AI To Detect False Positives in Vulnerability Reports

Introduction

Have you ever spent hours, or worse days, sifting through your application's vulnerability report? Jumping from project to project, reading lines of code, tracing dependencies to see where and how they are used. If you have, you may find something useful from my experience creating an AI vulnerability assistant. Stick around to learn about what problems I was trying to solve, the solution, and lessons learned from this exercise.

The Problem I Was Trying To Solve

It all started on Monday. We would go through the vulnerability report one-by-one and try to get as much done as we could in the hour we had scheduled. With this limited time, we would only look at Critical and High vulnerabilities. Throughout the week, I would try to find time to address what was left of the Critical and Highs. This often took 4 to 6 hours a week, and I was still ignoring the mediums and lows. This process took time and attention away from other important responsibilities, such as fixing the confirmed vulnerabilities.

The Solution

I noticed that I was already using AI tools to help me go through the vulnerability report, and as a developer, I thought, "How can I automate this?". The solution was to create an AI vulnerability assistant. This assistant reaches out and gets the vulnerability report, goes through each one, clones the repo, and uses the actual code base to check if the vulnerability is valid or a false positive. If it is a false positive, it will close it and comment on why/how it determines it is a false positive.

One thing I did not want was for it to guess or assume anything; if it is not sure, or if it is a real threat, it creates an issue/ticket in the project with its findings or a statement of "Not enough information". These issues are linked to the original vulnerability report for reference.

When the process is completed, it creates a summary and an Excel file and emails the security team. The Excel file contains a timestamp, the project name, vulnerability title, severity, verdict, and the URL to the vulnerability. The reason for this was that I still wanted to know what was closed out and what was left. I wanted to be able to monitor this process to see if and where it failed. I also have the past 90 Excel files stored in an S3 bucket for auditing if needed.

Lessons Learned

What worked well was the strict requirement that if there is not enough information, do not close the vulnerability. I have not seen any vulnerabilities being closed when they shouldn't. Also, SBOM and dependency vulnerabilities were handled well. It could use the code base to see exactly how, where, and if the vulnerability was being used. In my latest report, I had 460 vulnerabilities. The AI assistant closed out 374 of them and confirmed 86. This alone saved over 6 hours of manually checking every single vulnerability.

Some projects use internal libraries, and the current process cannot see into that library and do its assessment. For these, I still need to manually address them. I thought about how to address this and plan to implement a fix in the future. Another issue that I run into is false positives for Passwords. For example, the Postgres connect URL might be postgres://${USER}:${PASSWD}@localhost , and it still gets flagged as a valid password. It should be easy enough to have the AI assistant recognize that it is an environment variable, and I plan on implementing this in the near future.

This tool identifies and closes hundreds of false positives while I sleep. You too can take advantage of AI and should! Implement it into your security processes. Use it to do repetitive tasks and tasks that need some judgment. I invite you to think about how you can use AI to improve your day-to-day.