MSBuild should handle paths longer than MAX_PATH

[ariccio]

As seen in Fix 260 character file name length limitation, there's quite a bit of support for better longer-than-MAX_PATH-filename handling.

Will Buik's response w/regard to fixing it was:

We understand that this can be a frustrating issue, however, fixing it requires a large and complicated architectural change across different products and features including Visual Studio, TFS, MSBuild and the .NET Framework. Dedicating resources to this work item would come at the expense of many other features and innovation.

I wondered, how big can those architectural changes be (for MSBuild)?

According to Naming Files, Paths, and Namespaces: Maximum Path Length Limitation, (which I have mirrored here ):

The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\\?\" prefix. For example, "\\?\D:\very long path".
[other stuff]

Because you cannot use the "\\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters.

(emphasis mine)

...Which means that wherever MSBuild uses full paths, we can should be able to just prepend "\\?\" to ask for a long path.

Of course, we'd need to rework the existing path-length-workaround hack.

This won't fix the whole ecosystem, but it'll get us just one step closer.

I'd like to fix this myself (it seems simple enough), so in accordance with:

• Contributions must be discussed with the team first, or they will likely be declined. As our process matures and our experience grows, the team expects to take larger contributions.
• Only contributions referencing an approved Issue will be accepted.

...this is the suggested issue.

I'm a native developer at heart, not an experienced C# developer, but this shouldn't require anything crazy.

Direct references to MAX_PATH:

• NativeMethods.cs#L15
• NativeMethodsShared_Tests.cs#L53
• NativeMethodsShared.cs#L318
• NativeMethodsShared.cs#L540
• NativeMethodsShared.cs#L750
• FileState.cs#L293
• ComReference.cs#L460
• TargetsFile_Test.cs#L2041

[AndyGerlicher]

MSBuild team triage: This isn't a change we're looking to take anytime soon as it would be fairly substantial with implications on dependent tasks and tools.

[ariccio]

[...]it would be fairly substantial with implications on dependent tasks and tools.

Could you be a bit more specific?

[ariccio]

I'm still curious as to what the substantial implications of an incremental fix for MSBuild in isolation are, exactly.

[MattWhilden]

In general, you'd have to have all of the other things you're using understand long paths. There's been a number of discussions internally and externally about the MAX_PATH issue and they always eventually boil down to: "If we could rewrite everything we'd be good but we can't so there's an infinite sea of compat risk here :-(".

So... maybe some day?

[gwojan]

Have you looked at AlphaFS?

[MattWhilden]

The key issue here is that it doesn't really matter if MSBUILD is updated. It's not an island. it passes paths to lots of other things (OS apis, tools etc) and updating yourself in a way that doesn't break them is a Hard Problem (tm). Almost certainly not impossible but there's lots of other work with better payoffs.

[ariccio]

Hmm. The way I see it, is that the entire toolchain is not compatible with paths longer than MAX_PATH, and thus by (incrementally) fixing a single component will not lead to toolchain-wide regression. It'll just be broken in a different place - the entire toolchain, when used as a system, will still be broken, but in a slightly smaller way.

It's even worse, by the way, as MAX_PATH was only the "maximum total file path length" for FAT16, and pre-Unicode/pre-NT APIs. Any documentation that says MAX_PATH is the longest possible file path is horribly wrong! MAX_PATH should really have been named MAX_FAT_PATH or MAX_PATH_COMPONENT; any program that was built after Windows NT 3.1 (yes, July 27, 1993, the earliest default use of NTFS that I know of) to use Unicode APIs with the assumption that MAX_PATH is the longest possible file path is, and always was, terminally broken. GCC 4.9-style broken.

I can even find discussions dating more than ten years back, of developers realizing that assuming MAX_PATH is the maximum length of a path is an outdated (ridiculous) assumption.

Any components that have been retroactively "fixed" to ensure compatibility with only paths that are MAX_PATH or shorter - well, that's just crazy.

Also, I can't imagine that we can break the OS APIs themselves by using them in a documented manner.

So yeah, it's a Hard Problem™, and I know that we don't yet have proper time travel, but if we attack it incrementally, we either make small (and effectively non-functional) improvements, or expose bugs/unforeseen design issues in other components.

I say effectively non-functional, because those components that can't handle MAX_PATH are already broken for paths longer than MAX_PATH. I can't imagine that those applications rely internally on the exact full path name that's given to them.

Furthermore, I imagine that OTHER toolchains use MSBuild in ways that are crippled by the MAX_PATH limitations.

...and while new features/better compliance are nice, I think they're counterproductive/add silly complexity if the core components that they're built on are shaky and fail in unexpected ways (e.g. MAX_PATH limitations). Putting it off into the future only makes it harder to fix.

Lastly, let me remind you of the nearly 3000 users who voted on "Fix 260 character file name length limitation" before it was declined. There are still users commenting on the issue with frustration.

Here's an idea

How about you/we go ahead and make the changes to MSBuild, make it available as a "preview" or "technology demonstration", and see what (if anything) breaks.

[MattWhilden]

You seem to understand the issue pretty well. I don't disagree that isn't a frustrating experience some times but they have finite resources and can only take so much risk. I'd be more interested in them making sure cross plat is brought up cleanly but it's up to them. I'm sure they appreciate your enthusiasm and feedback. I'll bow out of this issue for now.. lots of other bugs to track down.

[ariccio]

Fair enough.

[noamkfir]

@ariccio If you have the inclination, I say go for it. Everybody will eventually benefit, especially if more pieces of the toolchain go open source. Saying that it can't be fixed because everything else in every possible toolchain would also have to be fixed immediately is, quite simply, a non-starter and completely unrealistic. The whole point of going open source is to let people scratch their itches, especially when Microsoft doesn't have the resources to do it themselves...

One way to minimize the risk associated with backwards and toolchain compatibility that Microsoft keeps talking about is to ensure that the current limitation remains the default behavior while making it possible to opt-in to longer paths. The opt-in mechanism can be via any relevant configuration mechanism, including a new command line option, a build file setting, a registry setting and/or some other mechanism.

[gingters]

I'm also in.

Especially down the road to xplat, where *NIX operating systems do not care about a 260 char path length at all. IF set at all (this may well be -1 to indicate that there is no limit) it for example defaults to 4096 on x86 Linux. On OS X's HFSP the max path length is unlimited (while the file name is limited to 255).

So, in fact, this issue here is a blocker for xplat.

Yes, there may be other places in VS, TFS and other tools that have problems with this, but as long as we want to simply 'MSBuild mysolution' on commandline or on an CI system like AppVoyer or Travis, and happen to have paths longer than 260 chars there, it should not artificially prevent us from doing so.

I also don't see how removing this limitation would break other stuff.
If other stuff can't handle paths longer than 260 chars, it already IS broken. It won't break more. I would just break elsewhere.

And: Yes. Especially because MSBuild is not an island, but a part in a chain, someone has to start. And MSBuild is the perfect place because it can be used more or less standalone on new DNX projects (because, honestly, Sake is the totally wrong thing for building stuff), and it could be the door-opener to fix the stuff in other places too. Best thing is, that others like the VS or TFS team won't have excuses like 'MSBuild doesn't support that, so why should we change that?' anymore.

When talking about where to spent time, it's mostly the time of the community that will be spend here. So why not simply start here, by removing that artifical limitation, pass longer paths around where necessary and, who knows, maybe much less is broken that is currently being thought off and a lot of stuff simply starts to work out of the box?

[Piedone]

👍

Mediaeval issues like this still prevent us from using proper folder structures and proper file/folder names still. And it's not a corner case: for us all of our builds would fail if not copied to the drive root of the build server!