Refactor LICENSE.md to make it scanner friendly #41095

SamuraiAku · 2021-06-05T06:03:29Z

Most corporate environments will not approve the use of open-source software until all of the licensing terms have been reviewed and approved. Given the huge amounts of open source in use these days, companies rely on automated scanning to make a first pass at evaluation. If the scanning software is able to comprehend the license files, then approval could be granted in just a day or two. If the scanners cannot comprehend the license file then it goes into a queue for review by a human which takes a good deal longer. Commonly this review is required for every new point release that someone at the company wants to use. Given the rapid rate of Julia development, having to go into manual review will impede adoption of the latest and greatest versions of Julia in the corporate world.

Currently Julia's LICENSE.md file cannot be comprehended by the scanners. A good and easy way to tell if the license is scanner friendly is by checking if Github's own automated scanners can detect the license. For further information on Github's automated scanning see:
https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/licensing-a-repository#detecting-a-license

This pull request modifies LICENSE.md to make it scanner friendly. To do that, only the top-level MIT license is listed in the file. Changes to the formatting of license text have been to make it comprehensible to the scanners but the text of the license has not been altered. The licensing for other components have been moved to a new file name named THIRDPARTY.md.

To demonstrate that the reformatted LICENSE.md can be read by license scanners, I have created a new repository that contains only this PR as it's master branch.
https://github.com/SamuraiAku/license_refactor

You can see that under "About" the page says this repository uses the MIT license. That information was automatically filled in by Github's license scanner

Please note that this PR is only one step towards making the code base full scanner friendly. Future PRs can begin to tackle the remaining issues piece by piece.

ViralBShah · 2021-06-06T23:17:00Z

The code in this repository is all MIT licensed (maybe there are some exceptions that I am forgetting), and the third party licenses probably only matter for the binaries. On the downloads page, we can link both the Julia and the third party licenses prominently.

Curious to see what others have to say.

SamuraiAku · 2021-06-07T03:10:50Z

Following the text in license.md, the following source files are covered by a different license than MIT

src/crc32.c : zlib license; credit (although not the license) is given in the source file
src/ccall.cpp starting at line 288 : bsd-3 license
some portion of src/jitlayers.cpp : uiuc
src/disarm.cpp: uiuc license
src/getopt.c: MIT license from another project (so I guess it doesn’t have to be listed in the license file?)
some code from MINGW project for dirname on windows ; multiple licenses listed on homepage; Julia lists MIT
Some other functions from netbsd; bsd-3 license
Some functions from python; Python license
Some code from google for cyclecount; Apache license
everything in base/ryu; Boost license
Everything in base/grisu; except that folder doesn’t exist? Should that line be removed?
base/src/{exp, rem_pio2, hyperbolic}.jl; exp.jl is missing a copyright notice in the file. Portions of the other two files are covered by FDLIBM license

oscardssmith · 2021-06-07T03:52:31Z

exp and all the variants are mit from 1.6 on. I rewrote the entire file from scratch using a different algorithm.

SamuraiAku · 2021-06-07T03:58:30Z

exp and all the variants are mit from 1.6 on. I rewrote the entire file from scratch using a different algorithm.

Good to know. That’s something to be removed from the license files in another PR.

vtjnash · 2021-06-07T16:02:59Z

Yeah, seems to need cleanup. I don't think cyclecount should have ever been listed and grisu is deleted.

SamuraiAku · 2021-06-08T02:33:12Z

I can open another PR for thirdparty cleanup after this one is dispositioned.

ViralBShah · 2021-06-08T13:41:14Z

@StefanKarpinski Any thoughts here? We should probably keep this open a bit longer to give time to folks to comment.

oscardssmith · 2021-06-08T13:48:20Z

Perhaps we should just add it to triage?

StefanKarpinski · 2021-06-08T13:56:58Z

I think it's fairly important to leave some kind of breadcrumb for someone who is looking at the license file indicating that there may be other licensing considerations when using the result of compiling this code. If an extra line at the bottom could link to the THIRDPARTY.md file that would be fine.

SamuraiAku · 2021-06-10T02:51:51Z

I think that will do it. At first adding the pointer to THIRDPARTY.md broke the license scanner until I found a way to make it ignore everything after the license.

licensee/licensee#498

THIRDPARTY.md

StefanKarpinski

This is looking good to me. With cross references between the two files this seems clear and it's good that it helps automatic license detection do its thing.

ViralBShah · 2021-06-16T21:17:31Z

Did triage discuss this?

oscardssmith · 2021-06-16T21:20:39Z

We didn't have triage last week. (So no)

LICENSE.md

JeffBezanson · 2021-06-17T19:15:49Z

Triage says yes.

mbauman · 2021-06-17T19:41:33Z

There doesn't seem to be a definitive idiom for this, but this SoftwareEngineering.SE question its answers suggest some similar ideas — it seems as long as whatever we do here is obvious we're in good stead. IMO most options suggested there are less obvious than this.

SamuraiAku · 2021-06-28T17:06:16Z

@ViralBShah Does this PR need your approval before it can move forward?

ViralBShah · 2021-06-28T18:09:17Z

What's going on in the CI that is preventing the merging? I think this is good to go.

ViralBShah · 2021-06-28T18:10:40Z

LICENSE.md

-
-On some platforms, distributions of Julia contain SSL certificate authority certificates,
-released under the [Mozilla Public License](https://en.wikipedia.org/wiki/Mozilla_Public_License).
+Please see THIRDPARTY.md for license information for other software used in this project.


It would be nice to link the URL of THIRDPARTY.md in here (either now or after merging).

SamuraiAku · 2021-06-28T18:56:40Z

It looks like maybe the version that I made these changes to had other issues. Should I rebase?

ViralBShah · 2021-06-29T04:09:06Z

Sure no harm rebasing

…ered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub).

… doesn't break the Github scanner (and hopefully others)

Co-authored-by: Stefan Karpinski <[email protected]>

Co-authored-by: Jeff Bezanson <[email protected]>

SamuraiAku · 2021-06-29T17:28:40Z

@ViralBShah I’ve rebased and now CI is passing. This should be enough to move forward right?

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub).

…s third party code. Remove reference from THIRDPARTY.md and add standard Julia header to exp.jl. See comments in JuliaLang#41095

* Add LLVM project license text to src/disasm.cpp * Remove mention of third party code from header in src/jitlayers.cpp, since there are no markings of third party code in the actual source code. Update THIRDPARTY.md as well * More precise reference to source files using MUSL code in THIRDPARTY.md * Remove reference to joinpath in Python section of THIRDPARTY.md. The function does not appear to exist * Removed reference to cyclecount from THIRDPARTY.md since it does not apear to exist in the codebase * Remove reference to grisu from THIRDPARTY.md. This code was deleted a while ago * base/special/exp.jl was previously rewritten so that it no longer uses third party code. Remove reference from THIRDPARTY.md and add standard Julia header to exp.jl. See comments in #41095 * Add BSD-3 license to src/flisp * In THIRDPARTY.md, move FEMTOLISP reference to the third party code in src section. This is the proper location for the reference since the code is part of the Julia repository and not an external dependency * In THIRDPARTY.md, change the LIBUNWIND license link to the repository Julia actually pulls the dependency from * In THIRDPARTY.md, change the LIBUV license link to the repository Julia actually pulls the dependency from * Clearly mark references to LLVM open source license as UIUC not BSD-3. They are very similar but are recognized as different licenses. * In THIRDPARTY.md, change the DSFMT license link to the respository Julia actually pulls the code from. * In THIRDPARTY.md, change the MBEDTLS license link to the repository Julia actually pulls from. The license in that repo is Apache 2.0 only, which is the default for the project anyhow, so I removed mention of the alternate GPL licensing option * In THIRDPARTY.md, change the SUITESPARSE license link to the repository Julia pulls the dependency from. * Add missing reference to LIBBLASTRAMPOLINE in THIRDPARTY.md * Add missing reference to LIBWHICH in THIRDPARTY.md * Add missing reference to NGHTTP2 in THIRDPARTY.md * Add hyperlinks between LICENSE.md and THIRDPARTY.md * In THIRDPARTY.md, move LIBWHICH from stdlib dependencies to build tools

SamuraiAku · 2021-08-26T04:52:22Z

@ViralBShah, Should this PR be back ported to 1.6 and 1.7?

ViralBShah · 2021-08-26T13:16:10Z

Yes, let's do that.

SamuraiAku · 2021-09-01T00:38:09Z

@ViralBShah can you add the back port labels then? Thanks.

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub). (cherry picked from commit 161e384)

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub). (cherry picked from commit 8978269)

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub).

* Add LLVM project license text to src/disasm.cpp * Remove mention of third party code from header in src/jitlayers.cpp, since there are no markings of third party code in the actual source code. Update THIRDPARTY.md as well * More precise reference to source files using MUSL code in THIRDPARTY.md * Remove reference to joinpath in Python section of THIRDPARTY.md. The function does not appear to exist * Removed reference to cyclecount from THIRDPARTY.md since it does not apear to exist in the codebase * Remove reference to grisu from THIRDPARTY.md. This code was deleted a while ago * base/special/exp.jl was previously rewritten so that it no longer uses third party code. Remove reference from THIRDPARTY.md and add standard Julia header to exp.jl. See comments in JuliaLang/julia#41095 * Add BSD-3 license to src/flisp * In THIRDPARTY.md, move FEMTOLISP reference to the third party code in src section. This is the proper location for the reference since the code is part of the Julia repository and not an external dependency * In THIRDPARTY.md, change the LIBUNWIND license link to the repository Julia actually pulls the dependency from * In THIRDPARTY.md, change the LIBUV license link to the repository Julia actually pulls the dependency from * Clearly mark references to LLVM open source license as UIUC not BSD-3. They are very similar but are recognized as different licenses. * In THIRDPARTY.md, change the DSFMT license link to the respository Julia actually pulls the code from. * In THIRDPARTY.md, change the MBEDTLS license link to the repository Julia actually pulls from. The license in that repo is Apache 2.0 only, which is the default for the project anyhow, so I removed mention of the alternate GPL licensing option * In THIRDPARTY.md, change the SUITESPARSE license link to the repository Julia pulls the dependency from. * Add missing reference to LIBBLASTRAMPOLINE in THIRDPARTY.md * Add missing reference to LIBWHICH in THIRDPARTY.md * Add missing reference to NGHTTP2 in THIRDPARTY.md * Add hyperlinks between LICENSE.md and THIRDPARTY.md * In THIRDPARTY.md, move LIBWHICH from stdlib dependencies to build tools

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub). (cherry picked from commit 8bc9409)

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub). (cherry picked from commit c312422)

Create THIRDPARTY.md to hold license information for all code not covered by the main MIT license. This format allows for automated scanning and categorization of Julia's license. The licenses were broken up this way because this is the format that many automated license scanners (including GitHub).

* Add LLVM project license text to src/disasm.cpp * Remove mention of third party code from header in src/jitlayers.cpp, since there are no markings of third party code in the actual source code. Update THIRDPARTY.md as well * More precise reference to source files using MUSL code in THIRDPARTY.md * Remove reference to joinpath in Python section of THIRDPARTY.md. The function does not appear to exist * Removed reference to cyclecount from THIRDPARTY.md since it does not apear to exist in the codebase * Remove reference to grisu from THIRDPARTY.md. This code was deleted a while ago * base/special/exp.jl was previously rewritten so that it no longer uses third party code. Remove reference from THIRDPARTY.md and add standard Julia header to exp.jl. See comments in #41095 * Add BSD-3 license to src/flisp * In THIRDPARTY.md, move FEMTOLISP reference to the third party code in src section. This is the proper location for the reference since the code is part of the Julia repository and not an external dependency * In THIRDPARTY.md, change the LIBUNWIND license link to the repository Julia actually pulls the dependency from * In THIRDPARTY.md, change the LIBUV license link to the repository Julia actually pulls the dependency from * Clearly mark references to LLVM open source license as UIUC not BSD-3. They are very similar but are recognized as different licenses. * In THIRDPARTY.md, change the DSFMT license link to the respository Julia actually pulls the code from. * In THIRDPARTY.md, change the MBEDTLS license link to the repository Julia actually pulls from. The license in that repo is Apache 2.0 only, which is the default for the project anyhow, so I removed mention of the alternate GPL licensing option * In THIRDPARTY.md, change the SUITESPARSE license link to the repository Julia pulls the dependency from. * Add missing reference to LIBBLASTRAMPOLINE in THIRDPARTY.md * Add missing reference to LIBWHICH in THIRDPARTY.md * Add missing reference to NGHTTP2 in THIRDPARTY.md * Add hyperlinks between LICENSE.md and THIRDPARTY.md * In THIRDPARTY.md, move LIBWHICH from stdlib dependencies to build tools

DilumAluthge requested review from ViralBShah and StefanKarpinski June 6, 2021 20:32

ViralBShah added the triage This should be discussed on a triage call label Jun 9, 2021

StefanKarpinski reviewed Jun 10, 2021

View reviewed changes

THIRDPARTY.md Outdated Show resolved Hide resolved

StefanKarpinski approved these changes Jun 10, 2021

View reviewed changes

JeffBezanson reviewed Jun 17, 2021

View reviewed changes

LICENSE.md Outdated Show resolved Hide resolved

JeffBezanson removed the triage This should be discussed on a triage call label Jun 17, 2021

ViralBShah approved these changes Jun 28, 2021

View reviewed changes

ViralBShah reviewed Jun 28, 2021

View reviewed changes

SamuraiAku added 3 commits June 29, 2021 08:26

Additional formatting of LICENSE.md to try and get the scanner to work.

32af673

Add pointer to thirdparty license info in LICENSE.md in a format that…

9846354

… doesn't break the Github scanner (and hopefully others)

SamuraiAku and others added 2 commits June 29, 2021 08:26

Update THIRDPARTY.md

5d40dd8

Co-authored-by: Stefan Karpinski <[email protected]>

Update LICENSE.md

9f3f817

Co-authored-by: Jeff Bezanson <[email protected]>

SamuraiAku force-pushed the lic_refactor branch from 3d49c18 to 9f3f817 Compare June 29, 2021 15:28

JeffBezanson merged commit 161e384 into JuliaLang:master Jun 29, 2021

SamuraiAku mentioned this pull request Jul 22, 2021

THIRDPARTY.md cleanup #41674

Merged

ViralBShah added backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Sep 1, 2021

This was referenced Sep 2, 2021

release-1.7: Backports for 1.7-RC1 #41781

Merged

Backports for julia-1.6.3 #41554

Merged

KristofferC removed backport 1.6 Change should be backported to release-1.6 backport 1.7 labels Sep 7, 2021

Uh oh!

Refactor LICENSE.md to make it scanner friendly #41095

Refactor LICENSE.md to make it scanner friendly #41095

Uh oh!

Conversation

SamuraiAku commented Jun 5, 2021

Uh oh!

ViralBShah commented Jun 6, 2021

Uh oh!

SamuraiAku commented Jun 7, 2021

Uh oh!

oscardssmith commented Jun 7, 2021

Uh oh!

SamuraiAku commented Jun 7, 2021

Uh oh!

vtjnash commented Jun 7, 2021

Uh oh!

SamuraiAku commented Jun 8, 2021

Uh oh!

ViralBShah commented Jun 8, 2021

Uh oh!

oscardssmith commented Jun 8, 2021

Uh oh!

StefanKarpinski commented Jun 8, 2021

Uh oh!

SamuraiAku commented Jun 10, 2021

Uh oh!

Uh oh!

StefanKarpinski left a comment

Choose a reason for hiding this comment

Uh oh!

ViralBShah commented Jun 16, 2021

Uh oh!

oscardssmith commented Jun 16, 2021

Uh oh!

Uh oh!

JeffBezanson commented Jun 17, 2021

Uh oh!

mbauman commented Jun 17, 2021

Uh oh!

SamuraiAku commented Jun 28, 2021

Uh oh!

ViralBShah commented Jun 28, 2021

Uh oh!

ViralBShah Jun 28, 2021

Choose a reason for hiding this comment

Uh oh!

SamuraiAku commented Jun 28, 2021

Uh oh!

ViralBShah commented Jun 29, 2021

Uh oh!

SamuraiAku commented Jun 29, 2021

Uh oh!

SamuraiAku commented Aug 26, 2021

Uh oh!

ViralBShah commented Aug 26, 2021

Uh oh!

SamuraiAku commented Sep 1, 2021

Uh oh!

Uh oh!