no_proxy env ignored on 302 redirect

[Gaasmann]

Hello,

I have a weird problem using Requests and proxies.
I have https_proxy and no_proxy env variable for a domain and it seems the no_proxy is ignored on a HTTP 302 redirect target.
What I do is basically:

s = requests.Session()
r = s.get("https://use-the-proxy.com")

The response is 302 Found (Location: https://do-not-use-the-proxy.com) and Requests tries to follow.
Here the problem: even if do-not-use-the-proxy.com is present in no_proxy, Requests still use the proxy for the request following a 302(the 302 "target").
If I directly request https://do-not-use-the-proxy.com, the proxy is not used as expected.

Thanks,

Nico

[Lukasa]

What version of requests are you using?

[Gaasmann]

Hello,
Requests version is 2.7.0.
Thanks

[Lukasa]

Cool, that's a good spot.

Yes, there is a minor bug in resolve_redirects. Specifically, while resolve_redirects attempts to remove proxy information, it cannot actually tell the difference between a proxy that was passed in via the command-line API or from the session and one that was extracted from the environment. This is because resolve_redirects is passed the computed proxies argument, not the user's proxies argument.

With the way the code in requests is structured, this is a very difficult problem to solve. One option is to hang the original proxies kwarg off the Request object: this will allow rebuild_proxies to essentially re-calculate the proxies argument. Another option is to suggest that the NO_PROXY environment variable overrides the user proxies argument for redirects: this is out of line with what we normally do, so I'm inclined to not do this. A third option is to try to do something wacky with storing the original proxies kwarg value and passing it to Session.send so that we can pass it to resolve_redirects, but that seems kind of nutty.

Does anyone else have an opinion on how to go about doing this that doesn't suck as much as the three I have just mentioned? @sigmavirus24?

[Lukasa]

For what it's worth, I'd argue that this is a good indication that proxies (and probably verify and cert) are being handled at the wrong level of abstraction. Arguably, the core logic about deciding which proxy to use (and, by analogy, how to work the TLS) belongs more on Transport Adapters than on Sessions: it's a property of the connection, not a property of the HTTP layer.

We can fix that up, but it requires a breaking change in 3.0.0, which is...less than ideal.

[sigmavirus24]

What if Request/PreparedRequest objects had hidden state about session level settings and per-request level settings (which I think you're referring to as "user" settings) so that we could distinguish the two? Storing there, means allowing the TransportAdapter to resolve things would be (maybe?) a simpler change.

[Lukasa]

I don't entirely know, to be honest. I'm uncomfortable with shoving stuff onto the Request/PreparedRequest object just because that's the convenient thing to do. It feels like fundamentally the wrong abstraction layer. =(

[sigmavirus24]

I know. Unfortunately, we've exposed too much of the internals to users to be able to fix this another way. If we had a better way of wrapping all of this up in some sort of context object for a request that is passed to a transport adapter, that would be much better. Sadly, that might break things for people with custom transport adapters.