Traefik 2.x Unable to Route to service 8080- Specifically Jenkins

Routing with IngressRoute CRD not working but works with Ingress.

I deployed Traefik 2.2.1 using helm.
I additionally applied ingressRoute for the dashboard from the helm/traefik site
I am able to view the dashboard no issues
I created whoami service with ingressRoute and middleware. Traefik routed correctly to service.
I created Jenkins service from helm installation.
I turned off the creation of ingress and instead created ingressRoute as follows:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    traefik.env: mytest
  name: mynameingress
  namespace: mynamespace
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`test.example.com`)
    services:
    - name: myjenkins
      namespace: jenkins
      port: 8080

The router shows up on the dashboard
The jenkins service looks like this

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  labels:
    traefik.env: myselector
  name: mynameingress
  namespace: mynamespace
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: Host(`test.example.com`)
    services:
    - name: myjenkins
      namespace: jenkins
      port: 8080

Under additional arguments for Traefik the following is added as a selector:
additionalArguments:

  • --providers.kubernetescrd.labelSelector=traefik.env=myselector

When I try to http://test.example.com I get a 404 error

When I reinstall Jenkins using helm and add an ingress which looks like this

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
  labels:
    traefik.env: myselector
  name: jenkinsingress
  namespace: jenkins
spec:
  rules:
  - host: test.example.com
    http:
      paths:
      - backend:
          serviceName: myjenkins
          servicePort: 8080

I now see two entries on the Traefik Dashboard under HTTP Routers one from the ingress and one from the ingressRoute.

But now when I try to http://test.example.com the application "Jenkins" comes up...

Some additional information:
The Traefik dashboard displays the routers appropriately. The one with ingress is designated as provider Kubernetes and the IngressRoute router is designated as KubernetesCRD.

The only content in the logs is this

time="2020-07-14T18:18:18Z" level=info msg="Configuration loaded from flags."
W0714 18:34:24.373123       1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1alpha1.IngressRouteTCP ended with: too old resource version: 8046202 (8061646)
W0714 18:34:32.423449       1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1.Secret ended with: too old resource version: 8056179 (8061675)
W0714 18:34:32.423514       1 reflector.go:326] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: watch of *v1.Secret ended with: too old resource version: 8056179 (8061675)

My deployment has the following list of args:

        args:
          - "--global.checknewversion"
          - "--global.sendanonymoususage"
          - "--entryPoints.traefik.address=:9000/tcp"
          - "--entryPoints.web.address=:8000/tcp"
          - "--entryPoints.websecure.address=:8443/tcp"
          - "--api.dashboard=true"
          - "--ping=true"
          - "--providers.kubernetescrd"
          - "--providers.kubernetesingress"
          - "--providers.kubernetescrd.labelSelector=traefik.env=myselector"

My questions:

  1. Is something wrong in my configuration for IngressRoute which prevents it from working without an ingress?
  2. Is an Ingress needed if you use IngressRoute?

Any feedback or assistance would be greatly appreciated.

thanks

Hey, can you update your question to include the definition of your jenkins service, it looks like you copy-pasted wrongly.

Hi --

Here is the yaml for Jenkins Service

apiVersion: v1
kind: Service
metadata:
  labels:
  name: myjenkins
  namespace: jenkins
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: NodePort 

I do not think you can specify namespace in IngressRoute under spec.routes.services.namespace. Where did you get this from?

Another thing I can suggest is to enable debug logging and compare dynamic configurations that come through against what you expect.

I was able to create whomai ingressRoute with namespace and it worked. I am not sure where I picked that up. I was trying several things just to try to get it to work.

I removed the namespace from the section in the ingressRoute and it did not make a difference.

The dynamic configurations all seem to be there in the logs.

I removed the namespace from the section in the ingressRoute and it did not make a difference.

Well, they are in different namespaces, so I would not expect that removing it would make any difference. Try putting them in the same namespace, may be?

The dynamic configurations all seem to be there in the logs.

I suggest going through them property by property and comparing values with the expectations. Often a typo leads to some of them to be dropped resulting in non-working configurations.

Post all the dynamic configurations from the log here, I l'll have a quick look.

I also noticed that you are connecting to jenkins on port 80, but the web entry point is 8000. Can you explain how this is exposed please?

The installation of Traefik was done with the helm chart(https://github.com/containous/traefik-helm-chart/tree/master/traefik). The default setting is set to expose the port:

The below is a portion of the Traefik service:

apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: traefik
spec:
  ports:
  - name: web
    nodePort: 31377
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30455
    port: 443
    protocol: TCP
    targetPort: websecure
  type: LoadBalancer

The targetPort web .. I believe is the entry point "web" defined in the args..
"--entryPoints.web.address=:8000/tcp"
These were all defaults out of the helm deployment.

I am working on getting the logs out shortly..

Here are the logs -- I replaces company specific settings ..

time="2020-07-15T12:56:14Z" level=info msg="Configuration loaded from file: /config/traefik.toml"
time="2020-07-15T12:56:14Z" level=info msg="Traefik version 2.2.5 built on 2020-07-13T16:35:29Z"
time="2020-07-15T12:56:14Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"traefik\":{\"address\":\":9000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}},\"web\":{\"address\":\":8000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{\"insecure\":true},\"http\":{}},\"websecure\":{\"address\":\":8443/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":10000000000},\"respondingTimeouts\":{\"idleTimeout\":180000000000}},\"forwardedHeaders\":{},\"http\":{}}},\"providers\":{\"providersThrottleDuration\":2000000000,\"kubernetesCRD\":{\"labelSelector\":\"traefik.env=myselector\"}},\"api\":{\"dashboard\":true},\"ping\":{\"entryPoint\":\"traefik\"},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}}}"
time="2020-07-15T12:56:14Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/contributing/data-collection/\n"
time="2020-07-15T12:56:14Z" level=info msg="Starting provider aggregator.ProviderAggregator {}"
time="2020-07-15T12:56:14Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2020-07-15T12:56:14Z" level=debug msg="Start TCP Server" entryPointName=web
time="2020-07-15T12:56:14Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2020-07-15T12:56:14Z" level=info msg="Starting provider *traefik.Provider {}"
time="2020-07-15T12:56:14Z" level=info msg="Starting provider *crd.Provider {\"labelSelector\":\"traefik.env=myselector\"}"
time="2020-07-15T12:56:14Z" level=debug msg="Using label selector: \"traefik.env=myselector\"" providerName=kubernetescrd
time="2020-07-15T12:56:14Z" level=info msg="label selector is: \"traefik.env=myselector\"" providerName=kubernetescrd
time="2020-07-15T12:56:14Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2020-07-15T12:56:14Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"ping\":{\"entryPoints\":[\"traefik\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2020-07-15T12:56:14Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=traefik routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-15T12:56:14Z" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=traefik
time="2020-07-15T12:56:14Z" level=debug msg="No default certificate, generating one"
time="2020-07-15T12:56:15Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"jenkins-myjenkins-894292368dfb49c2d8d5\":{\"entryPoints\":[\"web\"],\"service\":\"jenkins-myjenkins-894292368dfb49c2d8d5\",\"rule\":\"Host(`myjenkins.example.com`)\"}},\"services\":{\"jenkins-myjenkins-894292368dfb49c2d8d5\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://10.30.8.132:8080\"}],\"passHostHeader\":true}}},\"middlewares\":{\"notused-stripprefix\":{\"stripPrefix\":{\"prefixes\":[\"/jenkins\"]}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-07-15T12:56:15Z" level=debug msg="Added outgoing tracing middleware ping@internal" entryPointName=traefik routerName=ping@internal middlewareName=tracing middlewareType=TracingForwarder
time="2020-07-15T12:56:15Z" level=debug msg="Creating middleware" entryPointName=traefik middlewareName=traefik-internal-recovery middlewareType=Recovery
time="2020-07-15T12:56:15Z" level=debug msg="Creating middleware" routerName=jenkins-myjenkins-894292368dfb49c2d8d5@kubernetescrd middlewareName=pipelining middlewareType=Pipelining entryPointName=web serviceName=jenkins-myjenkins-894292368dfb49c2d8d5
time="2020-07-15T12:56:15Z" level=debug msg="Creating load-balancer" routerName=jenkins-myjenkins-894292368dfb49c2d8d5@kubernetescrd entryPointName=web serviceName=jenkins-myjenkins-894292368dfb49c2d8d5
time="2020-07-15T12:56:15Z" level=debug msg="Creating server 0 http://10.30.8.132:8080" serviceName=jenkins-myjenkins-894292368dfb49c2d8d5 serverName=0 routerName=jenkins-myjenkins-894292368dfb49c2d8d5@kubernetescrd entryPointName=web
time="2020-07-15T12:56:15Z" level=debug msg="Added outgoing tracing middleware jenkins-myjenkins-894292368dfb49c2d8d5" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=jenkins-myjenkins-894292368dfb49c2d8d5@kubernetescrd
time="2020-07-15T12:56:15Z" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery
time="2020-07-15T12:56:15Z" level=debug msg="No default certificate, generating one"
time="2020-07-15T12:56:15Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd
time="2020-07-15T12:56:15Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetescrd

Your logs say that the host rule is myjenkins.example.com, you are querying test.example.com. test.example.com is not part on any dynamic configuration from your logs. This would be the reason why it's not working...

Re: helm chart: thank you for explaining that, it makes sense, just out of curiosity, what cloud provider are you using?

The discrepancy .. is because I tried to mask the company info ... provider is AWS

I saw this link that says ingress is a valid option for Traefik https://containo.us/blog/traefik-2-2-ingress/

Yet on this page --> https://docs.traefik.io/migration/v1-to-v2/.
It says convert Ingress to Traefik IngressRoute resources.

It maybe the documentation is not caught up with the changes ... If I can get Ingress to work instead of IngressRoute I may go that way.

It just concerns me why I can get Ingress Route to work for some services and not others? I would prefer to be consistent.

If anyone has experienced this or has gotten further I would really like to hear your thoughts ... It makes me wary of using something like this for production without a clear pattern for implementation.

Yes, it sounds like that's what it is.

If I can get Ingress to work instead of IngressRoute I may go that way.

FWIW I have everything working with IngressRoutes, so I know it is possible. There is also a full Let's Encrypt example in the docos that works, you may try starting with it, and do incremental changes until you break/fix it, if you like.

It just concerns me why I can get Ingress Route to work for some services and not others?

Just to exclude this issue we just had with random 404s can you try with traefik 2.2.1 specifically and see if it still does not work - just to make sure that you are not hitting that issue.

If anyone ... gotten further I would really like to hear your thoughts

Never had any problem with IngressRoutes, they are working as I would expect them.

It makes me wary of using something like this for production without a clear pattern for implementation.

Totally understandable. We must be missing something. It would be easier to troubleshoot if you obfuscate your things consistently. It is very difficult to help when you do it differently in every message.

Some more thoughts:

  • Try putting your deployment, service and IngressRoute in the same namespace
  • Your service does not seen to have a selector, could you please double check
  • The dynamic config from the logs reports IP/PORT traefik is using to connect to your pod. Try to reach this IP/PORT form the traefik pod manually with curl and see if it's reachable.

@grumbo I'm facing same issue with IngressRoute. So, I switched to Ingress and followed official link but I'm getting below error in logs.

E0726 17:01:08.006765       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0726 17:01:12.311744       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0726 17:01:23.452737       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0726 17:01:39.526007       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0726 17:02:16.043578       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:traefik-ingress-controller" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

I'm using Kubernetes v1.18 and Traefik v2.3.
Please help.

Your rbac is off. Probably it's not for 2.3 but for earlier versions. Need to bring it up to date.

@zespri I have shared all the details here. Could you please take a look?