Repository navigation
Low number of findings compared to commercial tool #572
Description
Hi all,
for OSS compliance and vulnerability reports we use the OSS review toolkit (ORT). The ORT advisor component currently supports querying vulnerability information from VulnerableCode and Sonatype Nexus IQ, which we both use. We host an instance of VulnerableCode and run the importers on a regular schedule.
With this setup in place for about half a year, I did an evaluation of the findings returned by VulnerableCode and Nexus IQ based on the results produced by ORT. The outcome is that the number of findings reported by VulnerableCode is significantly lower than for Nexus IQ, particularly for certain types of packages (NPM, Python, Maven). Find below an excerpt from the results. (The "Packages" column contains the number of packages for which at least one security vulnerability has been reported by one of the systems.)
Type Packages Findings IQ Findings VC
Crate 23 19 17
Gem 22 67 25
Maven 929 2072 1044
NPM 644 1213 48
NuGet 40 42 29
PyPi 68 209 21
All 1729 3639 1184
The projects that have been scanned by ORT to produce these numbers are currently ongoing software development projects. I assume they use a typical set of library dependencies with up-to-date versions.
Now I am trying to investigate the reasons for these differences. What I have tried so far is the following:
- I checked that the importers are actually running successfully and populate the database. At least, I did not see any suspicious logs during the execution. Here are some figures regarding the number of packages in our database generated by the command
SELECT vp."type" pt, COUNT(*)
FROM vulnerabilities_package vp
GROUP BY pt
Maven 21225
NPM 12077
PyPi 13768
Does this look plausible or do we miss relevant data from sources?
- ORT queries VulnerableCode via the Bulk API passing in a list of PURLs for the packages in question. To rule out bugs in the interaction between these tools, I queried the VulnerableCode API manually, but came to similar results.
- I tried to match the packages found by ORT directly in the VulnerableCode database, circumventing the API; but again, I did not find more matches.
So, the question is, do you have any ideas/suggestions what could be the cause for this low number of findings? Is our database corrupt or is VulnerableCode missing important sources of vulnerability information? Any help would be appreciated.
Activity
pombredanne commentedon Oct 26, 2021
@oheger-bosch Thank you ++ for this detailed report!
Could you provide a list of purl/CVEs combos that you found missing in VulnerableCode? (If it is big you can send these zipped privately at pom@nexb.com)
That way we can investigate why this is happening: it could be either a bug or missing data source or both, but it is going to be hard to investigate short of hard data.
oheger-bosch commentedon Oct 26, 2021
@pombredanne I have sent a mail with packages / CVEs that have been reported by Nexus IQ, but not by VulnerableCode.
pombredanne commentedon Oct 29, 2021
I have started checking the differences and that's awesome data. @oheger-bosch Thank you +++
I will report back here when done.
sschuberth commentedon Oct 29, 2021
Thank you for looking at this so timely @pombredanne. It's quite important for us to understand what causes these differences in findings, as we're still planning for a major rollout of VulnerableCode, but this is being blocked by the concerns raised here.
jlarfors commentedon Nov 18, 2021
This might be related, but I found many CVEs without packages... To take a completely random sample, consider
CVE-2020-15222Querying the NVD CVE database we get at least one package (or CPE in this case:
cpe:2.3:a:ory:fosite:*:*:*:*:*:*:*:*), e.g.Checking VulnerableCode, I can find the CVE but it has zero packages:
Is this just because purls and CPEs don't really play well together, or should the data be consistent?
pombredanne commentedon Nov 19, 2021
@jlarfors Thank you for the details. For this CVE-2020-15222 this is different bug
pkg:github.com/ory/fosite@0c9e0f6d654913ad57c507dd9a36631e1858a3e9which seems to be the fix commitpkg:github.com/ory/fosite@0.31.0which is a fixed versionpkg:github.com/ory/fosite@0.30.5which is the first affected versionpkg:github.com/ory/fosite@0.30.6which the affected version in between@Hritik14 ^ :)
pombredanne commentedon Nov 19, 2021
@jlarfors and possibly also other package relationships to be inferred as this is a Go package based on https://github.com/ory/fosite/blob/master/go.mod
sbs2001 commentedon Dec 6, 2021
We're missing the fosite vulnerability since there's no importer for go ecosystem. However this should be fixed by #578
sschuberth commentedon May 6, 2022
Dear all, is there any progress to report regarding this issue?
Hritik14 commentedon May 7, 2022
@sschuberth We are working on improving the data quality by VulnerableCode. A lot of it can be tracked in #597 and the other open tickets. We are also pondering over a project to compare VulnerableCode data to the rest of the world with something like VulnTotal.