GSoC 2022 idea: get test coverage to 95% (stretch goal, 100%)

[terriko]

Possible GSoC idea: get test coverage to 95% (stretch goal, 100%)

• related: GSoC 2022: Start Here #1462
• related: GSoC 2022 Ideas / Brainstorming thread #1379

We're currently hovering around 80% coverage according to codecov: https://codecov.io/gh/intel/cve-bin-tool
Codecov lets you see which code paths have not been tested and you can use it to guide you in writing tests that explore missed code paths.

This would involve:

• writing tests
• removing/refactoring code we no longer need
• possibly improving our test harnesses and CI infrastructure
• probably a lot of pytest-mock

I think you could make good headway on getting to 100% in a 175hr project.

For a 350hr project you might want to plan to go beyond 100% coverage to see if you can improve robustness beyond our current code. I'd suggest planning to use a fuzzer on some of our file inputs to find new bugs, then fix them and add test cases.

Fuzzing starter ideas:

• use an xml fuzzer on the sbom code
• use json/csv/etc fuzzers on the report code
• as you integrate fuzzers, you'd want to fix all issues found and make new test cases to avoid regressions in future

(It's possible I'll be doing some fuzzer exploration before GSoC starts, but the benefit to fuzzing is that there's always the opportunity to find more stuff, so don't let that stop you.)

Hours

175 for a basic "improve coverage" project, 350 for going above and beyond coverage to improve robustness

Difficulty level

• intermediate (possibly doable by an advanced beginner if they get some experience with pytest)

Recommended skills

• pytest, pytest-mock, experience with fuzzers a plus

[terriko]

A note because it came up in gitter: This project is loosely reserved for a paid contributor to be selected through the GSoC 2022 process. (open to anyone over 18 who's willing to put in either 175 or 350 hours of paid work through the program). If you wish to work on this, please apply through that program when it opens on March 7.

Discussion and ideas are fine, and minor test improvements are welcome but pull requests covering big portions of the code base are discouraged at this time because we don't want to interfere with applicants who might want to do this idea. (If you do a PR and get us to 100% coverage, we might not be able to justify paying anyone, and that would be sad.)

There's 100 other issues available, please feel free to solve ones not flagged for gsoc participants!

[terriko]

Done as part of GSoC 2022, thanks @yashugarg !