[MDEP-82] go-offline / resolve-plugins does not resolve all plugin dependencies

[jira-importer]

Arne Degenring opened MDEP-82 and commented

The attached pom.xml is a very simple JAR project, without any direct dependencies or plugin dependencies.

Start with an empty local repository, and run mvn dependency:go-offline on it. Some files get downloaded, but not everything that is needed for the build. If you run "mvn -o package" afterwards, you end up with the following error:

[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] The plugin 'org.apache.maven.plugins:maven-resources-plugin' does not exist or no valid version could be found

Afterwards, even "mvn package" without the "-o" parameter does not work any longer.

---

Affects: 2.0-alpha-4, 2.8, 2.10

Attachments:

• maven-dependency-plugin-2.3.patch (4.75 kB)
• pom.xml (407 bytes)

Issue Links:

• MNG-6169 Packaging plugin bindings version updates

Remote Links:

• GitHub Pull Request #130
  

21 votes, 47 watchers

[jira-importer]

Brian E. Fox (imported) commented

The problem is that the project isn't injected with the plugins from the standard lifecycle. I may have to hard code the standard ones (site,clean,default) until I can figure out how to get at the list directly.

[jira-importer]

Harold Shinsato commented

Being able to go offline so all needed dependencies (including plugin dependencies) will work is something we have a requirement to do for our builds or it will be hard for us to get to use maven. Is there a workaround for this issue that would not require writing custom code?

[jira-importer]

Derek Lewis commented

This issue has been open with no comment for a long time. Is there really no fix or workaround with the problem? I'm required to package a zip of the ~/.m2/repository directory with our software, including all dependencies needed to build from behind a strict corporate firewall. This bug is causing a great deal of grief for me. Is this fixed in maven 3?

[jira-importer]

Devin Reid commented

A workaround I have found for this issue is to run a build with the property 'maven.repo.local' set to a directory in the project base directory and run a build to the install phase. One can then archive the resulting project and move it to environments that have no network access and be able to build successfully with the '-o' option.

I've tested this successfully with Maven 2.2.1 and Maven 3.0.3.

First run:
mvn -Dmaven.repo.local=lib clean install

Now the following command will work just fine.
mvn -Dmaven.repo.local=lib -o clean install

[jira-importer]

Derek Lewis commented

Devin, I've used a similar situation to work around the problem. However, it means having to build everything, which is time consuming, and also means that commands you haven't run (like mvn deploy) won't work, because the deploy plugin wasn't downloaded.

[jira-importer]

Paul Warren commented

Hi Folks,

Ability (or inability) to go-offline is a showstopper for us. We have very large customers who state they have to have offline development.

So naturally this was the goal I looked at first. One problem I have with it is the fact that it resolves top level dependencies only. It doesnt recursively descend through all subsequent transtive dependencies. I am new to maven but as that stands I don't see how it is adequate to actually solve the problem. But maybe I missed some fundamental concept. Anyway, I had a fiddle with the code and made is descend and resolve all transaitive dependencies and this was sufficient for my use case. I was then able to go-offline. I am going to attach the patch in case it helps others or you would like to fold it into the source.

[jira-importer]

Paul Warren commented

The recursive descent patch as promised.

[jira-importer]

Mark commented

This issue is even more annoying for proper two-phase deployments where packages are built on bare build systems without network access and where testing is done after the build on the target platform: it absolutely makes no sense to execute the tests just to prepare the package sources, but the tests won't have all dependencies later on when I don't execute the tests...

Try:

git clone https://github.com/cometd/cometd.git
cd cometd
git checkout 2.4.0
mvn -Dmaven.repo.local=/tmp/maven-repo dependency:go-offline
mvn -Dmaven.repo.local=/tmp/maven-repo -o install

[jira-importer]

Michael Osipov commented

This issue has been auto closed because it has been inactive for a long period of time. If you think this issue still persists, retest your problem with the most recent version of Maven and the affected component, reopen and post your results.

[jira-importer]

Daniel Compton commented

This is still an issue. Running mvn dependency:resolve-plugins as a second step works, but shouldn't be needed.

[jira-importer]

Hervé Boutemy commented

it looks like the first coment from Brian was not understood: "The problem is that the project isn't injected with the plugins from the standard lifecycle. I may have to hard code the standard ones (site,clean,default) until I can figure out how to get at the list directly."
I'll try to explain more in details.

• go-offline downloads everything that is declared in dependencies or plugins: if you did not declare anything, there is nothing to download
• but if you didn't declare anything, what is run when you launch a build? default plugins from standard lifecycle: http://maven.apache.org/ref/current/maven-core/default-bindings.html

But the plugin does not have the logic to detect these defaults (that are dependant on Maven version used)
I don't know if adding these defaults is feasible, or will give expected reault

one workaround, that is in fact a best-practice: please define plugins versions in your pom (or parent), even for default plugins that get "magically" injected by core if you did not tell anything
Your build will be more repeatable (you won't be hit if Maven core changes versions), and go-offline will do the expected job

[jira-importer]

Hervé Boutemy commented

I just tested the pom.xml proposed as test case:

• with Maven 3.0.5+ and maven-depdendency-plugin 2.8, plugins from standard lifecycle are downloaded
• with Maven 2.2.1, plugins from standard lifecycle are not downloaded

IIUC, Maven 3.x fixes the issue, by better detecting effective plugins (ie declared plugins + plugins from standard lifecycle)
Of course, if go-offline is called with one Maven version and the build is done later with another Maven version that has other plugins versions in standard lifecycle, there will be an issue: that's where defining explicitely plugins version in your pom.xml (or parent) is a best practice to avoid relying on Maven defaults then being dependant on precise Maven version used

If nobody objects, I'll close this issue as fixed in 3 days

[jira-importer]

Daniel Abraham commented

I'm running mvn 3.5.2, and I tried the workaround above.  There's two issues, one of which has its own workaround, but the other I haven't been able to work out.

First off, when explicitly depending on the standard lifecycle plugins, go-offline doesn't seem to be fetch transitive dependencies.  The workaround is to list them all explicitly. For example, in addition to listing org.apache.maven.plugins:maven-compiler-plugin:3.6.1, also list what's below (in xml, just colon separated for brevity):

1. org.codehaus.plexus:plexus-utils:jar:2.0.4
2. org.apache.maven:maven-repository-metadata:jar:3.0
3. org.codehaus.plexus:plexus-component-annotations:jar:1.6

The second problem, and one that I don't have a fix for is that org.apache.maven.plugins:maven-compiler-plugin:3.6.1 requires org.apache.maven:maven-repository-metadata:jar:3.0, but org.apache.maven.plugins:maven-resources-plugin:2.6 requires org.apache.maven:maven-repository-metadata:jar:2.0.6. Listing both versions of maven-repository-metadata gets me the 2.0.6 jar & pom, and the 3.0 pom, but no jar. If I list them together using a range I get the same behavior.

Am I missing something here? Is there a workaround? A fix? Thank you in advance.

[jira-importer]

John commented

Using Apache Maven 3.3.9, this error persists. Our build system uses a "provision" step where all dependencies are gathered (we support a lot more than Maven), and a build step in a clean environment with no internet connectivity. 

starting with a clean local repository:

repository jchittum$ pwd
/Users/jchittum/.m2/repository
repository jchittum$ ls
repository jchittum$

using the following POM

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.akamai.devprod</groupId>
<artifactId>pull-tester</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>test-app</name>
<url>https://dev-repos.devprod.akamai.com</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8</version>
<scope>test</scope>
</dependency> 
<dependency>
<groupId>com.spotify</groupId>
<artifactId>docker-client</artifactId>
<version>8.11.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
</dependencies>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
</properties>
</project>

I then run go-offline, which populates a most plugins and dependencies, however, I always run into some that aren't populated. Running mvn -o install

bos-mpoxc:test-pull-app jchittum$ mvn -o install
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building test-app 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ pull-tester ---
[WARNING] The POM for org.apache.maven:maven-core:jar:2.0.6 is missing, no dependency information available
[WARNING] The POM for org.apache.maven:maven-monitor:jar:2.0.6 is missing, no dependency information available
[WARNING] The POM for org.codehaus.plexus:plexus-utils:jar:2.0.5 is missing, no dependency information available
[WARNING] The POM for org.apache.maven.shared:maven-filtering:jar:1.1 is missing, no dependency information available
[WARNING] The POM for org.codehaus.plexus:plexus-interpolation:jar:1.13 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.525 s
[INFO] Finished at: 2018-05-08T07:52:34-05:00
[INFO] Final Memory: 9M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:2.6:resources (default-resources) on project pull-tester: Execution default-resources of goal org.apache.maven.plugins:maven-resources-plugin:2.6:resources failed: Plugin org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.maven:maven-profile:jar:2.0.6, org.apache.maven:maven-repository-metadata:jar:2.0.6, org.apache.maven:maven-plugin-registry:jar:2.0.6, classworlds:classworlds:jar:1.1-alpha-2: Cannot access central (https://repos.akamai.com/public-maven-repos) in offline mode and the artifact org.apache.maven:maven-profile:jar:2.0.6 has not been downloaded from it before. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

I can tell at least the following plugins are not downloaded with a go-offline:

• surefire (and transitive dependencies)
• jar (and its transitive dependencies
• compiler (and its transitive dependencies)
• resources (and its transitive dependencies)

It appears the original bug stating that core plugins for lifecycle phases are not populated. As is, since our build environment is docker based, I'll be setting up a script to manually pull all Apache maintained plugins and populating the initial build environment. 

How about this proposal?

The Maven Dependency Plugin, when using go-offline, should do a greedy pull, and populate all Maven core plugins and their transitive dependencies.

 

[jira-importer]

John commented

To narrow this down to versions.

maven = 3.5.2

mvn dependency:go-offline
mvn -o clean install

Fails with missing poms for transitive dependencies of plugins:

--- jacoco-maven-plugin:0.8.0:prepare-agent (default-prepare-agent) @ my-lib ---

[WARNING] The POM for junit:junit:jar:4.8.2 is missing, no dependency information available

[WARNING] The POM for org.apache.maven.reporting:maven-reporting-impl:jar:2.1 is missing, no dependency information available

[WARNING] The POM for org.jacoco:org.jacoco.agent:jar:runtime:0.8.0 is missing, no dependency information available

[WARNING] The POM for org.jacoco:org.jacoco.core:jar:0.8.0 is missing, no dependency information available

[WARNING] The POM for org.jacoco:org.jacoco.report:jar:0.8.0 is missing, no dependency information available

1. junit is relied upon somewhere in this build without being explicitly listed anywhere.
2. the org.jacoco:org.jacoco.* dependencies are listed explicitly by org.jacoco, however, when resolved, they only pulled the .jar, not the .pom. This led to the later failures.

But I noticed it was defaulting to using a slightly older maven-dependency-plugin, 2.8. So i tried:

mvn org.apache.maven.plugins:maven-dependency-plugin:3.1.0:go-offline

Forcing that version resolved all transitive plugin dependencies properly! However, I ran into further missing dependencies, specifically with surefire. After explicitly adding those, I was able to build successfully.

So, there are several layers of dependency resolutions not occurring, depending on the version of the plugin being used, and how other plugins are being implemented. Definitely no easy answer here. Our company is committed to getting this to work at the moment, so I'll probably continue to be bothersome