How to configure middleware with kubernetes-ingress? (middleware "xyz" does not exist)

Hi there,

I'm using Traefik 2.2.0-rc3 on a new k8s cluster, basically set up as documented on docs.traefik.io. All in all it works, but I can't figure out how to get middlewares working.

I have also tried the kubernetes-crd setup, where adding middlewares wasn't a problem, but I prefer the kubernetes-ingress way for its smaller deployment manifests.

Adding the router.middlewares annotation in the config shown below only let to an error message:

middleware "traefik-basic-auth@kubernetescrd" does not exist

kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: traefik-dashboard
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: traefik-basic-auth@kubernetescrd
    traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
spec:
  rules:
    - host: traefik.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: traefik
              servicePort: 8080

Here is the rest of my deployment file that might be relevant:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: traefik
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 10
      containers:
        - name: traefik
          image: traefik:v2.2
          args:
            - --api.insecure
            - --accesslog
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            - --providers.kubernetesingress
            #- --providers.kubernetescrd TODO is this required to get the middleware running?
            - --certificatesresolvers.letsencrypt.acme.tlschallenge
            - --certificatesresolvers.letsencrypt.acme.email=webmaster@example.com
            - --certificatesresolvers.letsencrypt.acme.storage=acme.json
            # Please note that this is the staging Let's Encrypt server.
            # Once you get things working, you should remove that whole line altogether.
            - --certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: web
              hostPort: 80
              containerPort: 80
            - name: websecure
              hostPort: 443
              containerPort: 443
            - name: admin
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: LoadBalancer
  selector:
    app: traefik
  ports:
    - name: web
      protocol: TCP
      port: 80
      targetPort: 80
    - name: websecure
      protocol: TCP
      port: 443
      targetPort: 443
    - name: admin
      protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: Secret
metadata:
  name: traefik-basic-auth-secret
data:
  users: <redacted>
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-basic-auth
spec:
  basicAuth:
    realm: Traefik
    secret: traefik-basic-auth-secret
    removeHeader: true

I guess @kubernetescrd might be wrong. According to these docs it should be the provider namespace, but I'm not sure what is the correct value in my case.

I have a similar situation. I would like to keep a minimal configuration using an Ingress, but I need a Middleware (stripPrefix).

Using Ingress CRD works perfectly, but when I try middleware definition a plain Ingress, like:

apiVersion: apiextensions.k8s.io/v1beta1
metadata:
  name: middlewares.traefik.containo.us
spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced
---
kind: Middleware
apiVersion: traefik.containo.us/v1alpha1
metadata:
  name: path-prefix
spec:
  stripPrefix:
    prefixes:
      - /play
---
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
  name: traefik-basic
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web
    traefik.ingress.kubernetes.io/router.middlewares: path-prefix@kubernatescrd

There is no way to make it working. I get something similar in the log:

msg="middleware \"path-prefix@kubernatescrd\" does not exist" entryPointName=web

My container configuration is similar.

      containers:
      - name: traefik
        image: traefik:v2.2
        args:
          - --log.level=DEBUG
          - --api
          - --api.insecure
          - --entrypoints.web.address=:80
          - --entrypoints.web.http.middlewares=default-striper@kubernetescrd
          - --providers.kubernetesingress
         # - --providers.kubernetescrd ??

If you went through that, please let me know.

S.

I've stumbled across that one myself. But fortunately a look at the dashboard helped me. If you define the middleware via the annotations, you have to put the Kubernetes namespace in front of it. That looks like this

traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-secureheaders@kubernetescrd

My middleware is called "secureheaders" and the namespace is "ingress-traefik"

This should definitely be better documented, because it is very confusing. But at least there is the possibility to access middlewares across the namespace.

1 Like

Thank you @p7k , your reply is very useful and I totally agree there is an "hole"in the documentation.

Could you clarify how can do you define other resources.
For example, if use "default" namespace, shouldn't my definition be

traefik.ingress.kubernetes.io/router.middlewares: default-path-prefix@kubernetescrd

(Don't bother of me keeping mispelling k8s, replacing randomly 'e' with 'a')

Moreover I do you configure configure your traefik CLI in the container?
Could you point me a complete working example snippet?

Because after I make few changes. Dashboard keep on showing:

ERRORS
middleware "default-path-prefix@kubernetescrd" does not exist 

Thanks a lot

Yes, that's right. In your case, the middleware would be called: default-path-prefix@kubernetescrd

But to be honest. So far I could only access the middlewares via annotations in an ingress object. I also tried using the CLI arguments, because I wanted to configure a default middleware for my entry point.

- --entrypoints.traefik.Address=:8080
- --entrypoints.traefik.http.middlewares=ingress-traefik-traefik-api-auth@kubernetescrd

But in this case I got the same error message as you. I really think this is a bug in Traefik.

Someone else has the same problem here: 2.2 can't see Kubernetes defined entryPoint middlewares?

So I really don't think that this function works properly yet.

Thanks @p7k .
It is not clear to me whether your solution is working for you or not.

Watching my configuration in my first answer, apart from the prefix / provider syntax in "dynamic middleware" configuration in the Ingress definition, how far I am from your solution?
How do you configure traefik in the container in the pod?

S.

I'm not quite sure what you mean. You are trying to define a middleware in a normal k8s ingress object (talk about the kubernetesingress provider). In this case only the kubernetes namespace prefix is missing.

If you want to define the middleware as default in an entrypoint (using the CLI arguments), I don't think this works yet. I haven't got it working yet either.

Example for an ingress object with kubernetesingress provider:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: secureheaders
  namespace: ingress-traefik
spec:
  headers:
    stsSeconds: 15768000
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: ingress-traefik-secureheaders@kubernetescrd
    name: <ingress-name>
    namespace: <namespace>
[...]

This example works for me.

I finally get it working!
Thanks a lot for your suggetions @p7k

1 Like