Dynamic setting value of TcpProxy.tunneling_config.hostname

[jewertow]

Description:
I have a case where applications running in Kubernetes communicate with external services (outside Kubernetes and in another network) through a dedicated forward proxy. Apps send HTTP CONNECT request and the proxy opens a TCP tunnel. The flow is illustrated below.

I would like to make forward proxy transparent for apps by routing requests from applications via Envoy to forward proxy. This would allow to avoid sending HTTP CONNECT requests from apps. The desired solution is illustrated below.

It's quite easy to achieve the desired flow by creating the following listener:

static_resources:
  listeners:
  - name: egress_gateway_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    listener_filters:
    - name: envoy.filters.listener.tls_inspector
    filter_chains:
    - filters:
      - name: tcp
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          cluster: external_proxy_cluster
          tunneling_config:
            hostname: example.com:443
  clusters:
  - name: external_proxy_cluster
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: proxy_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: external-proxy
                port_value: 3128

The problem is that the tunneling_config.hostname can only be statically defined, so I can't tunnel traffic to multiple destinations having a single listener. One of the potential solutions would be to set hostname based on SNI. TLS Inspector filter provides SNI value in %REQUESTED_SERVER_NAME%, but it's not possible to overwrite Host header.

tunneling_config:
  hostname: example.com:443
  headers_to_add:
  - header:
      key: Host
      value: "%REQUESTED_SERVER_NAME%"
    append_action: OVERWRITE_IF_EXISTS_OR_ADD

So what do you think about to make Host header mutable or to figure out another way to dynamically set tunneling_config.hostname?

Related issues
#13809 #18838

[ramaraochavali]

+1 for this. @lambdai mentioned that should be supported already but it seems it is not. Host header in this case I think should be mutable and we should relax this restriction.

@lambdai @alyssawilk WDYT?

[jewertow]

Another solution could be to to use "%REQUESTED_SERVER_NAME%" placeholder as a value of hostname:

tunneling_config:
  hostname: "%REQUESTED_SERVER_NAME%:443"

WDYT?

[alyssawilk]

yeah 100% this would be useful

[ramaraochavali]

Another solution could be to to use "%REQUESTED_SERVER_NAME%" placeholder as a value of hostname:

Yes. That would also be a good option.

[ramaraochavali]

@jewertow are you working on this?

[jewertow]

@ramaraochavali I didn't start yet, but I'm going to work on this the next week. But first I would like to dive into the implementation of TcpProxy and SNI inspector filter to figure out a better solution than just overwriting the host header. It's a hack rather than a good and long term solution, so I would like to suggest some more elegant API that will somehow inject SNI automatically when SNI inspector is enabled.

What's more, it's also a potential feature for Istio egress gateway, so I have to discuss this idea with the Istio community.
But don't worry, it's a top priority for me, so I will not delay it.

[ramaraochavali]

@jewertow Sure. thanks. This is also high priority for us - so wanted to know :-)

[lambdai]

There is an config verification that header name "host" or ":authority" can not be added or removed.

I comment that line and it works.

So I guess the only thing we need here is to keep maintain the restriction in http router but allow the mutation in tcp proxy tunnel setting

[ramaraochavali]

@lambdai Yes. That is what my original idea for the fix and it makes sense to me.

@jewertow Can we do that instead of thinking about changing/adding additional behaviours to hostname field?

[jewertow]

I know that I suggested this workaround, but to be honest, I really don't like this idea.
Please take a look at this config:

tunneling_config:
  hostname: example.com:443
  headers_to_add:
  - header:
      key: Host
      value: "%REQUESTED_SERVER_NAME%"
    append_action: OVERWRITE_IF_EXISTS_OR_ADD

This configuration is really bad, because:

1. if filter_chain_match.server_names is set, then hostname will never be used, because %REQUESTED_SERVER_NAME% is always present, so setting this value is redundant and completely useless;
2. if filter_chain_match.server_names is not set, then %REQUESTED_SERVER_NAME% may be empty in case of no SNI and it will cause failures, due to empty hostname.

Both cases are tricky and error prone. I don't think that it's a satisfying solution.

I would suggest the following changes in this API:

message TunnelingConfig {
  oneof hostname_config {
    // one of below configs is required
    
    // Static request-target for CONNECT request
    string hostname = 1;
    
    // Automatically sets request-target for CONNECT request using SNI specified in a downstream connection
    // Supports only TLS connections and requires to enable envoy.filters.listener.tls_inspector.
    SniHostname sni_hostname = 2;
  }
  
  // Sets hostname for new upstream connection based on the SNI extracted by TLS Inspector filter from the downstream connection
  // This option assumes that the downstream connection is TLS. In case of plain TCP inbound connection, filter does not establishes connection.
  message SniHostname {
  
    // Optional port used to create the request-target for all CONNECT request, i.e. CONNECT <hostname_from_sni>:<default_port>
    // If default_port is not specified, 443 is used.
    uint32 default_port = 1;
    
    // Optional map of hostnames and ports that specifies which port to use for a specific hostname
    repeated HostnamePortConfig custom_hostname_and_port_configs = 2;
  }
  
  message HostnamePortConfig {
    string hostname = 1;
    uint32 port = 2;
  }
}

Such an API would be backward compatible and cover most cases for tunneling TLS connections.

Examples how would this API be used:

1. Plain TCP with static request-target (already supported):

filter_chains:
- filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      cluster: proxy_cluster
      tunneling_config:
        hostname: www.example.com:80

2. TLS with static request-target (already supported):

filter_chains:
- filter_chain_match:
    server_names:
    - "www.example.com"
  filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      cluster: proxy_cluster
      tunneling_config:
        hostname: www.example.com:443
  listener_filters:
  - name: envoy.filters.listener.tls_inspector
    typed_config:
      "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"

3. TLS with automatically injected hostname and default port 443:

filter_chains:
- filter_chain_match:
    server_names:
    - "www.example.com"
    - "www.google.com"
  filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      cluster: proxy_cluster
      tunneling_config:
        sni_hostname: {}
  listener_filters:
  - name: envoy.filters.listener.tls_inspector
    typed_config:
      "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"

4. TLS with automatically injected hostname and custom port for a specific hostname:

filter_chains:
- filter_chain_match:
    server_names:
    - "www.example.com"
    - "www.google.com"
  filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      cluster: proxy_cluster
      tunneling_config:
        sni_hostname:
          custom_hostname_and_port_configs:
          - hostname: www.example.com
            port: 8443
  listener_filters:
  - name: envoy.filters.listener.tls_inspector
    typed_config:
      "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"

What do you think?