Isn't it possible to refer a middleware created in kubernetes-crd from CLI?

I have the following configuration, but the middleware defined in kubernetescrd provider namespace cannot be found by Traefik when the Traefik is starting.
I see in the logs "middleware test-headerx@kubernetescrd does not exist".

It works only if the middleware was defined in a dynamic configuration file and would be referred with @file attribute. Is this intended or a bug? To me, it should not matter if it was defined via CRD or dynamic config file. Sounds more like a bug to me.

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: testns
  name: traefik-ingress-controller-internal
  labels:
    app: traefik-ingress-controller-internal
spec:
  replicas: 1
  revisionHistoryLimit: 2
  minReadySeconds: 6
  selector:
    matchLabels:
      app: traefik-ingress-controller-internal
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: traefik-ingress-controller-internal
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      tolerations:
        - key: node-role.kubernetes.io/ingress-internal
          effect: NoSchedule
          operator: Exists
        - key: node-role.kubernetes.io/ingress-internal
          effect: NoExecute
          operator: Exists
      containers:
        - name: traefik
          image: traefik:v2.2
          args:
            - --log.level=DEBUG
            - --api
            - --accesslog
            - --entrypoints.web.address=:8000
            - --entrypoints.web.forwardedheaders.insecure=true
            - --providers.kubernetescrd
            - --providers.kubernetescrd.namespaces=testns
            - --ping
            - --entryPoints.ping.address=:8082
            - --ping.entryPoint=ping
            - --log=true
            - --api.dashboard=true
            - --api.insecure=true
            - --providers.kubernetescrd.ingressClass=internal
            - --metrics.prometheus
            - --metrics.prometheus.entryPoint=metrics
            - --entryPoints.metrics.address=:10254
            - --entryPoints.web.transport.lifeCycle.graceTimeOut=60
            - --entryPoints.web.transport.lifeCycle.requestAcceptGraceTimeout=3
#            - isn't it possible? I see in the logs "middleware test-headerx@kubernetescrd does not exist"
            - --entryPoints.web.http.middlewares=test-headerx
          env:
            - name: GOMAXPROCS
              value: "2"
          resources:
            requests:
              cpu: "200m"
              memory: "256Mi"
            limits:
              memory: "1Gi"
          ports:
            - name: web
              containerPort: 8000
            - name: ping
              containerPort: 8082
            - name: admin
              containerPort: 8080
            - name: metrics
              containerPort: 10254
          livenessProbe:
            failureThreshold: 4
            httpGet:
              path: /ping
              port: 8082
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 2
            httpGet:
              path: /ping
              port: 8082
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-ingress-controller-internal
  namespace: testns
  labels:
    app: traefik-ingress-controller-internal
spec:
  ports:
    - name: web
      port: 80
      targetPort: 8000
    - name: metrics
      port: 10254
  selector:
    app: traefik-ingress-controller-internal
  type: ClusterIP


---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-headerx
  namespace: testns
spec:
  headers:
    customRequestHeaders:
      X-Script-Name: "test"
    customResponseHeaders:
      X-Custom-Response-Header: "value"

It is certainly possible, there's just a minor mistake in your configuration. However, I think I might enhance our documentation on that, as it might not be obvious.

If you want to reference a middleware from the CRD provider, the middleware's name is prefixed with the namespace of the middleware object.

Therefore, in your case, the reference needs to be --entryPoints.web.http.middlewares= testns-test-headerx@kubernetescrd.

Do you see the difference?

Thanks, that made the trick. :upside_down_face: