Ingress doesn't work with connection upgrade

We experience issues with one ingress when connection upgrade is present. The request (from a remote "kubectl") is like that:

$ kubectl port-forward -v=8 pod/hello-kubernetes-9c7fdbc84-9hnf8 8080 -n default
...
I0711 14:56:21.497874   12560 round_trippers.go:416] POST https://env-4234220.mycluster.com/api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward
I0711 14:56:21.497913   12560 round_trippers.go:423] Request Headers:
I0711 14:56:21.497928   12560 round_trippers.go:426]     X-Stream-Protocol-Version: portforward.k8s.io
I0711 14:56:21.497937   12560 round_trippers.go:426]     User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
I0711 14:56:21.497954   12560 round_trippers.go:426]     Authorization: Bearer <--token-->
I0711 14:56:21.583626   12560 round_trippers.go:441] Response Status: 400 Bad Request in 85 milliseconds
I0711 14:56:21.583653   12560 round_trippers.go:444] Response Headers:
I0711 14:56:21.583661   12560 round_trippers.go:447]     Content-Length: 139
I0711 14:56:21.583667   12560 round_trippers.go:447]     Connection: keep-alive
I0711 14:56:21.583674   12560 round_trippers.go:447]     Server: openresty
I0711 14:56:21.583683   12560 round_trippers.go:447]     Date: Thu, 11 Jul 2019 14:56:21 GMT
I0711 14:56:21.583692   12560 round_trippers.go:447]     Content-Type: application/json
F0711 14:56:21.583974   12560 helpers.go:114] error: error upgrading connection: Upgrade request required

We use the following traefik ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/secure-backends: "true"
    ingress.kubernetes.io/protocol: https
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  rules:
  - http:
      paths:
      - path: /api
        backend:
          serviceName: kubernetes
          servicePort: 443

for the service

$ kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.244.0.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         10.102.6.10:6443
Session Affinity:  None
Events:            <none>

Looking closer at "ingress-controller", there are errors:

time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}" ForwardURL="https://10.102.6.10:6443"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: Round trip: https://10.102.6.10:6443, code: 400, Length: 139, duration: 3.988031ms"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"

Tried to reproduce it manually with CURL, and indeed, the request via ingress doesn't work:

$ curl -k -v -XPOST  -H 'Connection: upgrade' -H 'Upgrade: Upgrade: SPDY/3.1' -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" -H "Authorization: Bearer <-token->" 'http://env-4234220.mycluster.com/api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward'
* About to connect() to env-4234220.mycluster.com port 80 (#0)
*   Trying 10.102.5.244...
* Connected to env-4234220.mycluster.com (10.102.5.244) port 80 (#0)
> POST /api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward HTTP/1.1
> Host: env-4234220.mycluster.com
> Accept: */*
> Connection: upgrade
> Upgrade: Upgrade: SPDY/3.1
> X-Stream-Protocol-Version: portforward.k8s.io
> User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
> Authorization: Bearer <-token->
> 
< HTTP/1.1 400 Bad Request
< Content-Length: 139
< Content-Type: application/json
< Date: Thu, 11 Jul 2019 17:45:32 GMT
< 
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}
* Connection #0 to host env-4234220.mycluster.com left intact

although the direct apiserver call works perfectly:

$ curl -k -v -XPOST  -H 'Connection: upgrade' -H 'Upgrade: Upgrade: SPDY/3.1' -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" -H "Authorization: Bearer <-token->" 'https://k8sm.env-4234220.mycluster.com:6443/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward'
* About to connect() to k8sm.env-4234220.mycluster.com port 6443 (#0)
*   Trying 10.102.6.10...
* Connected to k8sm.env-4234220.mycluster.com (10.102.6.10) port 6443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=kube-apiserver
* 	start date: Jul 11 12:19:53 2019 GMT
* 	expire date: Jul 10 12:19:53 2020 GMT
* 	common name: kube-apiserver
* 	issuer: CN=kubernetes
> POST /api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward HTTP/1.1
> Host: k8sm.env-4234220.mycluster.com:6443
> Accept: */*
> Connection: upgrade
> Upgrade: Upgrade: SPDY/3.1
> X-Stream-Protocol-Version: portforward.k8s.io
> User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
> Authorization: Bearer <-token->
> 
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: SPDY/3.1
< X-Stream-Protocol-Version: portforward.k8s.io
< Date: Thu, 11 Jul 2019 17:16:50 GMT

Traefik version is v1.7.12.
What could be the issue with the ingress?

Hi @dfateyev, I might be saying something wrong but, when using kubectl port-forward, the requests emitted are not going through the ingress right?

It looks like you are targeting the apiserver of Kubernetes, using a matching/modifying rule of type PathPrefixStrip: /api, but the requests are going on the path /api/api, which is not clear why?

Could you elaborate by providing a reproductible case (we don't know how is Traefik installed and deployed as Ingress controller for example)? A k3s in a docker-compose could be easy to reproduce such a thing.

Thanks!

Hello @dduportal, thanks for the answer, yes, requests cannot pass the ingress properly. It's actual for external "kubectl" connections that are going via ingress (internal connections go to "apiserver" directly and don't require ingress).

It looks like you are targeting the apiserver of Kubernetes, using a matching/modifying rule of type PathPrefixStrip: /api , but the requests are going on the path /api/api , which is not clear why?

We use "/api" as the apiserver ingress path (please kindly check in the first post), so "/api/api" path is formed when we address apiserver via ingress ( http://env-4234220.mycluster.com/api/api/v1 ... ->
https://10.102.6.10:6443/api/v1 ...".
We can use another ingress path (not "/api"), but I think it doesn't cause the issue.

Traefik is installed as DaemonSet this way, and other ingresses work as they should.

The issue reproduction is pretty straightforward to me, with the ingress configuration from the first post:

  • I have sent a connection upgrade request to "apiserver" directly and via ingress, with CURL;
  • In the first case, "apiserver" answered properly, and offered to upgrade request ("HTTP/1.1 101 Switching protocol");
  • In the second case case (via ingress) — I don't see such upgrade request handled in "ingress-controller" logs, looks like it answers "HTTP/1.1 400 Bad request" on any connection upgrade attempt.

Maybe, there are ingress configuration issues (the configuration from the first post), but honestly I have no idea what can be wrong with it.

@dfateyev Why are you stripping the /api prefix in your ingress object?

It would seem that you are encountering issues due to mismatching paths.

Please note: this annotation ingress.kubernetes.io/secure-backends: "true" is not supported by Traefik.

Also, please confirm that you have insecureSkipVerify enabled for traefik, or you have added the rootCA to traefik, as your API server is running over https, and Traefik needs to communicate with it.

Hello @daniel.tomcej,

Why are you stripping the /api prefix in your ingress object?

Because, I use "/api" path to reach "kube-apiserver" from outside. So, I'm stripping this superfluous prefix before passing the request to the apiserver.

Worth to note, that everything works when the connection doesn't require to upgrade protocol. I can use "get nodes", "get pods" without problems.
I also dropped " secure-backends" (which isn't needed for traefilk indeed), but it also didn't change the situation.

I also tried not to drop / modify any prefixes, so the ingress becomes as below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-api
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/protocol: https
spec:
  rules:
  - http:
      paths:
      - path: /api
        backend:
          serviceName: kubernetes
          servicePort: 443

but get the same issue:


F0722 12:56:18.161819   21910 helpers.go:114] error: error upgrading connection: Upgrade request required

This time, there are no stripped parts, rewrites, and a only the basic annotations, but upgrade requests still don't work. Looks like that the request which is need to be upgrade is broken, when it's passed via Ingress.

Any update on this issue? We hit the same issue using 1.7.12

There should be an issue in Traefik connection upgrade handling.
Switched to "ingress-nginx", and never have this issue afterwards.

Hey :slight_smile: Just to make sure I'm not missing something.

Did someone fill a potential issue on github for that?

Hello SantoDE, the initial issue was here.

I think the issue seems to be that the traefik middleware does not handle SPDY upgrade, and will strip `Upgrade:" header in that case. It does handle websocket upgrade, but the issue is that kubectl does not support websocket.

Hi,
It's not a SPDY issue, SPDY works well in v1.7 and in v2.x.

When you used the curl command, there is two problems, first a typo in the Upgrade header (you duplicate Uprade: ) and then, as we're using HTTPS, and Traefik is compatible with HTTP2 (and not the kubernetes api), curl automatically use HTTP2, you need to specify that you really want http1.1 with --http1.1 option.

Normally kubectl should works because all the curl problem are not present, but, the problems you may encounter is "tls" problem. You need to configure correctly https, with the cert and key for the server part, and the cert part for the connection with the backend.

If you want to be sure it's not a certificate issue, you can use the insecureSkipVerify way. You need to configure it on traefik for the connection with the backend
https://docs.traefik.io/routing/overview/#insecureskipverify for v2
https://docs.traefik.io/v1.7/configuration/commons/ for v1.7

and in your kubectl command by using --insecure-skip-tls-verify.

And finally, if it still doesn't work, could you provide us your traefik configuration?

Hi , the issue still effect in traefik v1.7.21.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes
  namespace: default
  annotations:
    ingress.kubernetes.io/protocol: "https"
spec:
  rules:
  - host: kubernetes.test-apiserver.local
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes
          servicePort: 443

When Access the "https://kubernetes.test-apiserver.local/api/v1/namespaces/kube-system/pods/traefik-6bb59c4dcc-wshm6/exec?command=bash&container=traefik&stderr=true&stdin=true&stdout=true&tty=true" url , the request from dashboard like below.

Host: kubernetes.test-apiserver.local
User-Agent: dashboard/v2.0.0-rc5
Content-Length: 0
Authorization: Bearer ae05b4df-fd21-43f7-bdf4-72854211087c
Connection: Upgrade
Upgrade: SPDY/3.1
X-Stream-Protocol-Version: v4.channel.k8s.io
X-Stream-Protocol-Version: v3.channel.k8s.io
X-Stream-Protocol-Version: v2.channel.k8s.io
X-Stream-Protocol-Version: channel.k8s.io

I had set --insecure-skip-tls-verify arg for traefik.
The Response 500 Error for the request.

Is there any solutions.

I tried with your ingress, and it works for me.

Could you provide us your traefik static configuration?