We experience issues with one ingress when connection upgrade is present. The request (from a remote "kubectl") is like that:
$ kubectl port-forward -v=8 pod/hello-kubernetes-9c7fdbc84-9hnf8 8080 -n default
...
I0711 14:56:21.497874 12560 round_trippers.go:416] POST https://env-4234220.mycluster.com/api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward
I0711 14:56:21.497913 12560 round_trippers.go:423] Request Headers:
I0711 14:56:21.497928 12560 round_trippers.go:426] X-Stream-Protocol-Version: portforward.k8s.io
I0711 14:56:21.497937 12560 round_trippers.go:426] User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
I0711 14:56:21.497954 12560 round_trippers.go:426] Authorization: Bearer <--token-->
I0711 14:56:21.583626 12560 round_trippers.go:441] Response Status: 400 Bad Request in 85 milliseconds
I0711 14:56:21.583653 12560 round_trippers.go:444] Response Headers:
I0711 14:56:21.583661 12560 round_trippers.go:447] Content-Length: 139
I0711 14:56:21.583667 12560 round_trippers.go:447] Connection: keep-alive
I0711 14:56:21.583674 12560 round_trippers.go:447] Server: openresty
I0711 14:56:21.583683 12560 round_trippers.go:447] Date: Thu, 11 Jul 2019 14:56:21 GMT
I0711 14:56:21.583692 12560 round_trippers.go:447] Content-Type: application/json
F0711 14:56:21.583974 12560 helpers.go:114] error: error upgrading connection: Upgrade request required
We use the following traefik ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kubernetes-dashboard
namespace: default
annotations:
kubernetes.io/ingress.class: traefik
ingress.kubernetes.io/secure-backends: "true"
ingress.kubernetes.io/protocol: https
traefik.frontend.rule.type: PathPrefixStrip
spec:
rules:
- http:
paths:
- path: /api
backend:
serviceName: kubernetes
servicePort: 443
for the service
$ kubectl describe svc kubernetes
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP: 10.244.0.1
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 10.102.6.10:6443
Session Affinity: None
Events: <none>
Looking closer at "ingress-controller", there are errors:
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}" ForwardURL="https://10.102.6.10:6443"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: Round trip: https://10.102.6.10:6443, code: 400, Length: 139, duration: 3.988031ms"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward/http: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
time="2019-07-11T14:56:21Z" level=debug msg="vulcand/oxy/forward: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"https\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.102.6.10:6443\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Authorization\":[\"Bearer <-token->\"],\"Connection\":[\"upgrade\"],\"Content-Length\":[\"0\"],\"Https-Enabled\":[\"true\"],\"Upgrade\":[\"SPDY/3.1\"],\"User-Agent\":[\"kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5\"],\"X-Forwarded-For\":[\"54.37.80.26\"],\"X-Forwarded-Prefix\":[\"/api\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Host\":[\"env-4234220.mycluster.com\"],\"X-Real-Ip\":[\"54.37.80.26\"],\"X-Remote-Port\":[\"54274\"],\"X-Stream-Protocol-Version\":[\"portforward.k8s.io\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"env-4234220.mycluster.com\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.100.0.101:35974\",\"RequestURI\":\"/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-9hnf8/portforward\",\"TLS\":null}"
Tried to reproduce it manually with CURL, and indeed, the request via ingress doesn't work:
$ curl -k -v -XPOST -H 'Connection: upgrade' -H 'Upgrade: Upgrade: SPDY/3.1' -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" -H "Authorization: Bearer <-token->" 'http://env-4234220.mycluster.com/api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward'
* About to connect() to env-4234220.mycluster.com port 80 (#0)
* Trying 10.102.5.244...
* Connected to env-4234220.mycluster.com (10.102.5.244) port 80 (#0)
> POST /api/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward HTTP/1.1
> Host: env-4234220.mycluster.com
> Accept: */*
> Connection: upgrade
> Upgrade: Upgrade: SPDY/3.1
> X-Stream-Protocol-Version: portforward.k8s.io
> User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
> Authorization: Bearer <-token->
>
< HTTP/1.1 400 Bad Request
< Content-Length: 139
< Content-Type: application/json
< Date: Thu, 11 Jul 2019 17:45:32 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}
* Connection #0 to host env-4234220.mycluster.com left intact
although the direct apiserver call works perfectly:
$ curl -k -v -XPOST -H 'Connection: upgrade' -H 'Upgrade: Upgrade: SPDY/3.1' -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" -H "Authorization: Bearer <-token->" 'https://k8sm.env-4234220.mycluster.com:6443/api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward'
* About to connect() to k8sm.env-4234220.mycluster.com port 6443 (#0)
* Trying 10.102.6.10...
* Connected to k8sm.env-4234220.mycluster.com (10.102.6.10) port 6443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=kube-apiserver
* start date: Jul 11 12:19:53 2019 GMT
* expire date: Jul 10 12:19:53 2020 GMT
* common name: kube-apiserver
* issuer: CN=kubernetes
> POST /api/v1/namespaces/default/pods/hello-kubernetes-9c7fdbc84-h2vd9/portforward HTTP/1.1
> Host: k8sm.env-4234220.mycluster.com:6443
> Accept: */*
> Connection: upgrade
> Upgrade: Upgrade: SPDY/3.1
> X-Stream-Protocol-Version: portforward.k8s.io
> User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5
> Authorization: Bearer <-token->
>
< HTTP/1.1 101 Switching Protocols
< Connection: Upgrade
< Upgrade: SPDY/3.1
< X-Stream-Protocol-Version: portforward.k8s.io
< Date: Thu, 11 Jul 2019 17:16:50 GMT
Traefik version is v1.7.12.
What could be the issue with the ingress?