Traefik 1.7.12 ingress requests ending with a HTTP 504 gateway timeout

mmisztal1980 · June 26, 2019, 8:18am

I’ve provisioned an Azure AKS cluster with advanced networking
I’ve deployed traefik using the HELM cmdline provided below.
I’ve added the following manifest to one of my services which are being exposed to the public internet using the traefik ingress controller with the intention of enabling basic auth:

Ingress manifest template:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-gateway-service
  namespace: "#{Environment.Name}#"
  labels:
    app: my-gateway-service
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/protocol: http
    traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
    traefik.ingress.kubernetes.io/auth-type: "basic"
    traefik.ingress.kubernetes.io/auth-secret: "my-secret-map"
spec:
  rules:
    - host: "#{Environment.Name}#.#{Environment}#.my-host.dk"
      http:
        paths:
          - path: "/my-gateway-service"
            backend:
              serviceName: my-gateway-service
              servicePort: 30084

I’ve started to execute requests using POSTMAN against my endpoint’s liveness probe path: /probez : http://{my-generated-subdomain}.my-host.dk/my-gateway-service/probez

What did you expect to see?

I expected to see 100% of my responses succeeding with HTTP 200

What did you see instead?

Traefik’s frontend rule points to a k8s service which has 3 pods.
1/3 requests is ending with a gateway timeout, other requests are succeeding.

#	HTTP Status Code
1	200
2	200
3	504
4	200
5	200
6	504
7	200
8	200
9	504

etc…

This behavior seems to differ after redeploying traefik or scaling the deployment of the underlying my-gateway-service

Interestingly it appears to be regular in nature.

Output of `traefik version`: (What version of Traefik are you using?)

1.7.12

What is your environment & configuration (arguments, toml, provider, platform, …)?

Traefik deployed using HELM with the following values.yaml overrides:
helm.exe upgrade --namespace kube-system --install --set imageTag=1.7.12,dashboard.enabled=true,debug.enabled=true --wait traefik stable/traefik

If applicable, please paste the log output in DEBUG level (`--log.level=DEBUG` switch)

[
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"0bbda921-d0a7-43e2-81f0-3f1892b6f739\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31452\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: completed ServeHttp on request","time":"2019-06-25T11:25:04Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"0bbda921-d0a7-43e2-81f0-3f1892b6f739\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31452\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: completed ServeHttp on request","time":"2019-06-25T11:25:04Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"0bbda921-d0a7-43e2-81f0-3f1892b6f739\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31452\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: completed ServeHttp on request","time":"2019-06-25T11:25:04Z"},
    {"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Basic auth succeeded","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: begin ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"ForwardURL":{"Scheme":"http","Opaque":"","User":null,"Host":"10.244.2.56:80","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: Forwarding this request to URL","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.56:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: begin ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.56:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: begin ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel.","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"vulcand/oxy/forward/http: Round trip: http://10.244.2.56:80, code: 200, Length: 0, duration: 42.743604ms","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.56:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: completed ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.56:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: completed ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"82383179-6419-4229-a4c8-1651b81d8032\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: completed ServeHttp on request","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:05Z"},
    {"level":"debug","msg":"Basic auth succeeded","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: begin ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"ForwardURL":{"Scheme":"http","Opaque":"","User":null,"Host":"10.244.2.57:80","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: Forwarding this request to URL","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.57:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: begin ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.57:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: begin ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"level":"debug","msg":"Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel.","time":"2019-06-25T11:25:06Z"},
    {"level":"debug","msg":"vulcand/oxy/forward/http: Round trip: http://10.244.2.57:80, code: 200, Length: 0, duration: 1.636961ms","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.57:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: completed ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.2.57:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: completed ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"c197c4b0-7099-4bf4-ad33-9121bac0ea28\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: completed ServeHttp on request","time":"2019-06-25T11:25:06Z"},
    {"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Basic auth succeeded","time":"2019-06-25T11:25:07Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: begin ServeHttp on request","time":"2019-06-25T11:25:07Z"},
    {"ForwardURL":{"Scheme":"http","Opaque":"","User":null,"Host":"10.244.3.55:80","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: Forwarding this request to URL","time":"2019-06-25T11:25:07Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: begin ServeHttp on request","time":"2019-06-25T11:25:07Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: begin ServeHttp on request","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Upstream ResponseWriter of type *pipelining.writerWithoutCloseNotify does not implement http.CloseNotifier. Returning dummy channel.","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:07Z"},
    {"level":"debug","msg":"Received Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:09Z"},
    {"level":"debug","msg":"Skipping Kubernetes event kind *v1.Endpoints","time":"2019-06-25T11:25:09Z"},
    {"level":"debug","msg":"'499 Client Closed Request' caused by: context canceled","time":"2019-06-25T11:25:09Z"},
    {"level":"debug","msg":"vulcand/oxy/forward/http: Round trip: http://10.244.3.55:80, code: 499, Length: 21, duration: 2.023496558s","time":"2019-06-25T11:25:09Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward/http: completed ServeHttp on request","time":"2019-06-25T11:25:09Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"http\",\"Opaque\":\"\",\"User\":null,\"Host\":\"10.244.3.55:80\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/forward: completed ServeHttp on request","time":"2019-06-25T11:25:09Z"},
    {"Request":"{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/probez\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"*/*\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Authorization\":[\"Basic ********************\"],\"Cache-Control\":[\"no-cache\"],\"Connection\":[\"keep-alive\"],\"Postman-Token\":[\"a5cd116a-4b28-4362-aeb8-41f1cf8c2e19\"],\"User-Agent\":[\"PostmanRuntime/7.15.0\"],\"X-Forwarded-Prefix\":[\"/my-gateway-service\"],\"X-WebAuth-User\":[\"schultz\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"f1.infrastructure.my-host.dk\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.244.2.1:31535\",\"RequestURI\":\"/probez\",\"TLS\":null}","level":"debug","msg":"vulcand/oxy/roundrobin/rr: completed ServeHttp on request","time":"2019-06-25T11:25:09Z"}
]

The above log is everything I could obtain at the moment, I’m failing to find evidence that this is a configuration fault, so I’m guessing it might be a bug. Please advise on how I can diagnose this further ?

zespri · June 26, 2019, 9:38am

Try accessing the service without traefik, if you get similar results it would mean that some of the pods that the service is froning are unhealthy.

mmisztal1980 · June 26, 2019, 11:21am

@zespri done that, port-forwarded directly to the underlying service and I was 100% successful

Is there anything else I can try to diagnose this? I fail to see issues in the debug log

daniel.tomcej · June 26, 2019, 4:57pm

Hello @mmisztal1980,

From your log:

Round trip: http://10.244.3.55:80, code: 499, Length: 21, duration: 2.023496558s

Your pod (ending with .55 IP) returned a code 499 after a 2 second request.

Compare this to your log with the pod ending in .57 IP:

Round trip: http://10.244.2.57:80, code: 200, Length: 0, duration: 1.636961ms

The error here is being generated by your backend pod, not by Traefik itself. The 504 Traefik reports reflects the fact that your pod returned an error.

There is content in the failed response (non-zero length), but you may have to check the pod logs (on the .55 pod), or manually curl a few times to confirm what the message is.

The logs clearly show that Traefik was able to forward the request successfully, but that your backend pod returned an unexpected response.

mmisztal1980 · June 26, 2019, 5:50pm

Hi @daniel.tomcej this may have been me cancelling the request - I’ll try to get some more logs. Is there any way to make the logging output more verbose?

I doubt this is a pod error, as all that endpoint does is to return a HTTP 200 (liveness probe)

mmisztal1980 · July 4, 2019, 2:41pm

Hi @daniel.tomcej , sorry about the late response.
I've repeated my test, but this time I wasn't cancelling anything from POSTMAN, and waited for the requests to timeout.
I've conducted 2 test runs, 3 requests each:

Test run 1:

ok
ok
timeout

Test run 2:

ok
ok
timeout

Logs in github gist:

gist.github.com

https://gist.github.com/mmisztal1980/a205b9d3accc787667d46d0613be360b

gistfile1.txt

[
  {
    "level": "debug",
    "msg": "Received Kubernetes event kind *v1.Endpoints",
    "time": "2019-07-04T14:34:55Z"
  },
  {
    "level": "debug",
    "msg": "Skipping Kubernetes event kind *v1.Endpoints",
    "time": "2019-07-04T14:34:55Z"

This file has been truncated. show original

daniel.tomcej · July 4, 2019, 7:06pm

Hello @mmisztal1980,

Similar to your logs initially,

You have 3 pods: with IPs 63, 64, and 65.

Your logs show the IP's 64 and 65 responding properly:

"vulcand/oxy/forward/http: Round trip: http://10.244.2.64:80, code: 200, Length: 0, duration: 1.752466ms"
"vulcand/oxy/forward/http: Round trip: http://10.244.2.65:80, code: 200, Length: 0, duration: 2.234484ms"

But your 3rd pod:

"'504 Gateway Timeout' caused by: dial tcp 10.244.3.63:80: i/o timeout"
"vulcand/oxy/forward/http: Round trip: http://10.244.3.63:80, code: 504, Length: 15, duration: 30.000626097s"

Traefik waited for over 30 seconds for your pod to respond, and it didn't...

There are really 2 issues that may be at play:

You may have a networking issue with the overlay network between your nodes.
Your pod may have issues responding consistently on that path.

Either way, the debug logs show that Traefik is attempting to connect to your pods, but cannot. I would try running an alpine container and try curling your endpoint manually to confirm.

daniel.tomcej · July 4, 2019, 7:09pm

The other thing to note:

Your third pod is not on the same /24 network.:

10.244.3.63
10.244.2.64
10.244.2.65

I don't know if it is related, but it is odd. Especially since the last octet is sequential. You mentioned that you enabled advanced networking in your cluster setup. I would definitely start looking there.

mmisztal1980 · July 4, 2019, 7:35pm

Definitely odd. I’ll start there. Thanks!

mmisztal1980 · July 4, 2019, 9:46pm

@daniel.tomcej just so that I have a clear picture, when traefik routes the request, is it sending it directly to the pods, or is it routing the request though the kubernetes service object?

When doing kubectl port-forward the success rate is 100% - hence the question

daniel.tomcej · July 5, 2019, 10:23pm

@mmisztal1980 You are correct. Traefik connects directly to the pods. It does not use the service IP.

dperezmavro · July 8, 2019, 9:33am

I am facing a similar issue with occassional 504s that appear to happen for no reason. @daniel.tomcej have you got any suggestions (or are there any good guides for this) for how I'd start debuging this ?!

I thought this might have been a stale DNS record issue, but since traefik goes straight to the pods, then I don't think this could be it. As an alternative, is there any way to configure traefik to retry that request ?!

mmisztal1980 · July 8, 2019, 2:35pm

I've examined my AKS cluster in depth, and found that our networking was miss-configured. Basically the template was configured for 100% Azure CNI usage but it was kubenet (default) that was active, and that was causing the issue. After fixing the template and redeploying the cluster, traefik is 100% successful at routing requests to the target service.

Thanks @daniel.tomcej for pointing me in the right direction

daniel.tomcej · July 8, 2019, 3:26pm

Hello @dperezmavro, I would use the same steps:

Enable the debug log, and recreate your 504 situation.
Once you have the debug log, it will show the backend requests that have failed, and what the response from your pod is.

With that information, you can get a clearer picture as to what requests worked, which requests failed, and you can play the "what is different between these two?" game

99% of the time, 504's in kubernetes are due to networking issues.

If you are still encountering issues @dperezmavro, feel free to open a new thread here, and we can work on your issue directly.

Thanks!

dperezmavro · July 9, 2019, 9:43am

@daniel.tomcej thank you very much for your help! I followed your suggestions and I think I am getting closer, I'll now open a new issue to stop spamming this one!

Topic		Replies	Views
504 Gateway timeout errors on Kubernetes Traefik v2 (latest) kubernetes-crd	2	1664	October 6, 2021
Traefik ingress - transport issue Traefik v2 (latest) kubernetes-ingress	2	406	November 7, 2022
Unable to access Traefik Ingress on port 8080 Traefik v2 (latest) kubernetes-ingress	0	302	March 21, 2023
Ingressroute configuration for multiple services - AKS Traefik v2 (latest) middleware	0	132	February 7, 2024
Traefik v1.7.x to Traefik v2.x.x ingress controller migration Traefik v2 (latest) kubernetes-ingress	2	619	February 24, 2020

Traefik 1.7.12 ingress requests ending with a HTTP 504 gateway timeout

What did you expect to see?

What did you see instead?

Output of traefik version: (What version of Traefik are you using?)

What is your environment & configuration (arguments, toml, provider, platform, …)?

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Related Topics

Output of `traefik version`: (What version of Traefik are you using?)

If applicable, please paste the log output in DEBUG level (`--log.level=DEBUG` switch)