Bug: globalExtends no longer works for repositories and autodiscovery with v38

[mikevader]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Bitbucket Server

Please tell us more about your question or problem

We have the following setup:
We have a central configuration we reuse in multiple environments.
The selection which repositories should be processed we extracted into a separat config which is imported with globalExtends.

With v37 this worked quite well. With the update to v38 this stopped working completely. According to the release notes, there is a change with globalExtends but I think we should be unaffected by it. The change switched the precedence so that the globalExtends has lower priority, but we have the configuration only in one place.

A little bit more concrete. We have this renovate configuration:

    "dryRun": "full",
    "extends": [
      "local>devops/renovate-config:base.json5#develop",
      "local>devops/renovate-config:onboarding.json5#develop"
    ],
    "forkProcessing": "enabled",
    "globalExtends": [
      "https://bitbucket.mycompany.com/projects/DEVOPS/repos/renovate-config/raw/onboarded-repositories.json5?at=refs%2Fheads%2Fdevelop"
    ],
    "onboarding": true,
    "requireConfig": "required",
    "packageRules": [
      {
        "matchCategories": [
          "python"
        ],
        "enabled": false
      }
    ]

The globalExtends file onboarded-repositories.json5 looks like this:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "repositories": [
    "RP/renovate-test-repository",
    "RP/renovate-test-unonboarded-repository"
  ]
}

With this configuration, no repository will be processed.
We tried to do it with autodiscovery, and add just the filters to the onboarded-repositories.json5 file, with the result that all repositories are processed, like the content of the globalExtends is ignored.

Could this be a regression or where are we going awry?

Logs (if relevant)

Logs

{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Detected HTTP_PROXY value in env","time":"2024-11-04T10:27:55.975Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Detected HTTPS_PROXY value in env","time":"2024-11-04T10:27:55.977Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Detected NO_PROXY value in env","time":"2024-11-04T10:27:55.977Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Using RE2 regex engine","time":"2024-11-04T10:27:55.979Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Parsing configs","time":"2024-11-04T10:27:55.979Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Checking for config file in /opt/renovate/config.json","time":"2024-11-04T10:27:55.982Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":40,"logContext":"x2C4-qgvwrjEAVLUcybn9","originalConfig":{"autodiscover":false,"dryRun":"full","extends":["local>devops/renovate-config:base.json5#develop","local>devops/renovate-config:onboarding.json5#develop"],"forkProcessing":"enabled","globalExtends":["https://bitbucket.mycompany.com/projects/DEVOPS/repos/renovate-config/raw/onboarded-repositories.json5?at=refs%2Fheads%2Fdevelop"],"onboarding":true,"python":{"enabled":false},"requireConfig":"required"},"migratedConfig":{"autodiscover":false,"dryRun":"full","extends":["local>devops/renovate-config:base.json5#develop","local>devops/renovate-config:onboarding.json5#develop"],"forkProcessing":"enabled","globalExtends":["https://bitbucket.mycompany.com/projects/DEVOPS/repos/renovate-config/raw/onboarded-repositories.json5?at=refs%2Fheads%2Fdevelop...
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Converting GITHUB_COM_TOKEN into a global host rule","time":"2024-11-04T10:27:56.004Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","config":{"autodiscover":false,"dryRun":"full","extends":["local>devops/renovate-config:base.json5#develop","local>devops/renovate-config:onboarding.json5#develop"],"forkProcessing":"enabled","globalExtends":["https://bitbucket.mycompany.com/projects/DEVOPS/repos/renovate-config/raw/onboarded-repositories.json5?at=refs%2Fheads%2Fdevelop"],"onboarding":true,"requireConfig":"required","packageRules":[{"matchCategories":["python"],"enabled":false}]},"msg":"File config","time":"2024-11-04T10:27:56.006Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","config":{},"msg":"CLI config","time":"2024-11-04T10:27:56.007Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","config":{"hostRules":[{"hostType":"github","matchHost":"github.com","token":"***********"}],"baseDir":"/tmp/renovate/","cacheDir":"/tmp/renovate/cache/","platform":"bitbucket-server","endpoint":"https://bitbucket.mycompany.com/","username":"someuser","password":"***********","gitAuthor":"Renovate Bot, Test <[email protected]>","cacheHardTtlMinutes":10080,"cachePrivatePackages":true},"msg":"Env config","time":"2024-11-04T10:27:56.007Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","config":{"autodiscover":false,"dryRun":"full","extends":["local>devops/renovate-config:base.json5#develop","local>devops/renovate-config:onboarding.json5#develop"],"forkProcessing":"enabled","globalExtends":["https://bitbucket.mycompany.com/projects/DEVOPS/repos/renovate-config/raw/onboarded-repositories.json5?at=refs%2Fheads%2Fdevelop"],"onboarding":true,"requireConfig":"required","packageRules":[{"matchCategories":["python"],"enabled":false}],"hostRules":[{"hostType":"github","matchHost":"github.com","token":"***********"}],"baseDir":"/tmp/renovate/","cacheDir":"/tmp/renovate/cache/","platform":"bitbucket-server","endpoint":"https://bitbucket.mycompany.com/","username":"someuser","password":"***********","gitAuthor":"Renovate Bot, Test <[email protected]>","cacheHardTtlMinutes":10080,"cachePrivate...
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Adding trailing slash to endpoint","time":"2024-11-04T10:27:56.007Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Enabling forkProcessing while in non-autodiscover mode","time":"2024-11-04T10:27:56.007Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Enabling onboardingNoDeps while in non-autodiscover mode","time":"2024-11-04T10:27:56.008Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"hostRules: no authentication for bitbucket.mycompany.com","time":"2024-11-04T10:27:56.013Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Found valid git version: 2.47.0","time":"2024-11-04T10:27:56.259Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Setting global hostRules","time":"2024-11-04T10:27:56.337Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Adding token authentication for github.com (hostType=github) to hostRules","time":"2024-11-04T10:27:56.340Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Bitbucket Server version is: 8.19.3","time":"2024-11-04T10:27:56.366Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Using configured gitAuthor (Renovate Bot, Test <[email protected]>)","time":"2024-11-04T10:27:56.366Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Adding password authentication for bitbucket.mycompany.com (hostType=bitbucket-server) to hostRules","time":"2024-11-04T10:27:56.371Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Using configured baseDir: /tmp/renovate/","time":"2024-11-04T10:27:56.372Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Using configured cacheDir: /tmp/renovate/cache/","time":"2024-11-04T10:27:56.372Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Using containerbaseDir: /tmp/renovate/cache/containerbase","time":"2024-11-04T10:27:56.372Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1","time":"2024-11-04T10:27:56.374Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Commits limit = null","time":"2024-11-04T10:27:56.374Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Setting global hostRules","time":"2024-11-04T10:27:56.374Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Adding token authentication for github.com (hostType=github) to hostRules","time":"2024-11-04T10:27:56.376Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Adding password authentication for bitbucket.mycompany.com (hostType=bitbucket-server) to hostRules","time":"2024-11-04T10:27:56.376Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"validatePresets()","time":"2024-11-04T10:27:56.376Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"hostRules: applying Basic authentication for bitbucket.mycompany.com","time":"2024-11-04T10:27:56.378Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":40,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"No repositories found - did you want to run with flag --autodiscover?","time":"2024-11-04T10:27:57.673Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Checking file package cache for expired items","time":"2024-11-04T10:27:57.674Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Error count cleaning up cache: 176","time":"2024-11-04T10:28:15.042Z","v":0}
{"name":"renovate","hostname":"renovate-test-self-service-test-1730716066-6n8fl","pid":9,"level":20,"logContext":"x2C4-qgvwrjEAVLUcybn9","msg":"Deleted 65 of 3552 file cached entries in 17368ms","time":"2024-11-04T10:28:15.042Z","v":0}

[csmuller]

After experimenting with globalExtends in Renovate version 39.235.2, we've seen the following behavior.
Any self-hosted configuration we add to the globalExtends config file seems to be ignored completely by the bot. We tested with a few, like repositories, autodiscover or logContext. If we use these configuration options in the config file directly, they work just fine.
The weird thing is that repository configuraitons, e.g., packageRule are handed down correctly when we put them into globalExtends. So, there's no problem with finding or reading the config file. Also, there are no errors or (debug) infos in the logs that would point out any mistakes.

Is this known behavior, a bug, or did we misconfigure or misunderstand some of the configuraoitn options?

[rarkins]

The cause of this is the way the configs are merged here:

renovate/lib/workers/global/index.ts

Lines 139 to 145 in 5091b14

	config = mergeChildConfig(
	await resolveGlobalExtends(
	config.globalExtends,
	config.ignorePresets,
	),
	config,
	);

The existing config is a full config, including default values. e.g. it has repositories: [], etc. When merging, those config values take precedence over resolved globalExtends values, even if they're defaults.

Here is an example of how to fix repositories specifically: https://github.com/renovatebot/renovate/compare/fix/32307-global-extends-repositories?expand=1

But the problem likely affects most self-hosted options, as you point out.

I think that the "proper" solution is to refactor this logic so that globalExtends is merged against the raw global config prior to merging with defaults.

[rarkins]

Here is the proper fix, but it needs tests added: #35341

Anyone is welcome to fork that branch and take it over if they want this done fast - I unfortunately don't have the time for at least this month

[csmuller]

Hay @rarkins, thanks for looking into this and already coming up with a fix.
We'll try to implement tests for it soon.

[csmuller]

This was resolved in release 40.3.3