Unexpected routing behavior when mixing File and Docker providers

I have a legacy Tomcat application that exposes an API via SOAP. This application is running on bare metal, not Dockerized. We'll say it lives at tomcat.mydomain.com.

I wrote an API in Python using FastAPI and Zeep to act as a proxy between the legacy SOAP API to define some methods and return responses to the potential consumers as serialized JSON rather than XML. This API is running on a container, so I had a need to mix File and Docker providers. I wanted this API to be accessed at tomcat.mydomain.com/api/v1/. Normally without the use of an LB for path interception, the Tomcat application will serve up its own version of a 4xx error when trying to access /api or other non-existent endpoints.

Here was my original dynamic_config.toml file:

[http]
  [http.services]
    [http.services.tomcat-web.loadBalancer]
      [[http.services.tomcat-web.loadBalancer.servers]]
        url = "http://172.16.88.10:8169"
      [http.services.tomcat-web.loadBalancer.healthCheck]
        path = "/ping"
        interval = "10s"
        timeout = "3s"

  [http.middlewares]
    [http.middlewares.redirect.redirectscheme]
      scheme = "https"

  [http.routers]
    [http.routers.tomcat]
      entryPoints = ["web"]
      rule = "Host(`tomcat.mydomain.com`)"
      service = "tomcat-web"
      middlewares = ["redirect"]

    [http.routers.tomcatssl]
      entryPoints = ["web-secure"]
      rule = "Host(`tomcat.mydomain.com`)"
      service = "tomcat-web"
      [http.routers.tomcatssl.tls]

[tls]
  [[tls.certificates]]
    certFile = "/etc/traefik/certs/cert.pem"
    keyFile = "/etc/traefik/certs/key.pem"

And here was my Docker compose file:

version: '3.3'

services:
    traefik:
        image: "traefik:v2.0"
        container_name: "traefik"
        command:
            - --log.level=INFO
            - --log.format=json
            - --api.insecure=true
            - --api.dashboard=true
            - --providers.docker=true
            - --providers.docker.exposedByDefault=false
            - --providers.file=true
            - --providers.file.filename=/etc/traefik/dynamic_config.toml
            - --entrypoints.traefik.address=:8269
            - --entrypoints.web.address=:8080
            - --entrypoints.web-secure.address=:8443
        ports:
            - "8080:8080"
            - "8443:8443"
            - "8269:8269"
        volumes:
            - ./cert.pem:/etc/traefik/certs/cert.pem
            - ./key.pem:/etc/traefik/certs/key.pem
            - ./dynamic_config.toml:/etc/traefik/dynamic_config.toml:ro
            - /var/run/docker.sock:/var/run/docker.sock

    tomcatapi:
        image: "tomcatapi"
        container_name: "tomcatapi"
        environment:
            - TOMCAT_SOAP_USER=REDACTED
            - TOMCAT_SOAP_PASSWORD=REDACTED
        volumes:
            - ./api_keys.json:/keys/api_keys.json:ro
        labels:
            - "traefik.enable=true"
            - "traefik.http.routers.tomcatapi.rule=Host(`tomcat.mydomain.com`) && PathPrefix(`/api/v1/`)"
            - "traefik.http.routers.tomcatapi.entrypoints=web-secure"

What did you expect to see?

Requests to tomcat.mydomain.com/api/v1/foo/bar to be routed to the API running on the Docker container.

What did you see instead?

Requests to tomcat.mydomain.com/api/v1/foo/bar not being caught by Traefik and instead being routed to the baremetal Tomcat application and thus receiving 4xx errors generated by the Tomcat application.

After banging my head against my desk for a few hours, on a hunch I decided to try to define a Router via the File provider that would point to the Docker service definition for the API container, and for some reason it worked. So, I was only able to get expected behavior when I added the following to dynamic_config.toml:

    [http.routers.soapapi]
      entryPoints = ["web-secure"]
      rule = "Host(`tomcat.mydomain.com`) && PathPrefix(`/api/v1/`)"
      service = "tomcatapi-traefik@docker"
      [http.routers.soapapi.tls]

And modified my docker-compose file to the following:

        labels:
            - "traefik.enable=true"
            - "traefik.http.routers.tomcatapi.rule=Host(`tomcatapi-traefik`)"
            - "traefik.http.routers.tomcatapi.entrypoints=web-secure"

Output of traefik version: (What version of Traefik are you using?)

Version:      2.0.1
Codename:     montdor
Go version:   go1.13.1
Built:        2019-09-26T16:18:03Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Configurations have been provided above.

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Debug logs can be provided upon request. I opted not to provide them yet because it would take some scrubbing on my end to hide production details, and honestly they're not that interesting. They only show Traefik receiving requests for tomcat.mydomain.com/api/v1/foo/bar and routing them to the tomcat.mydomain.com service. At first I thought it might have been an issue with setting the Priority, but I really don't understand why I had to place the router pointing to the Docker service in the file, rather than Traefik just doing what I thought it was supposed to do the first time. They're both configured to listen to the same entrypoint, so this is really confusing to me.

I do get an error on startup with the modified configuration:

{"entryPointName":"web-secure","level":"error","msg":"the service \"tomcatapi-traefik@docker\" does not exist","routerName":"soapapi@file","time":"2019-10-09T18:56:35Z"}

However requests are still routed properly to the Dockerized API after docker-compose up -d.

Hello,

as the entry point is web-secure, the TLS configuration is missing in the tomcatapi router on labels

    tomcatapi:
        image: "tomcatapi"
        container_name: "tomcatapi"
        environment:
            - TOMCAT_SOAP_USER=REDACTED
            - TOMCAT_SOAP_PASSWORD=REDACTED
        volumes:
            - ./api_keys.json:/keys/api_keys.json:ro
        labels:
            - "traefik.enable=true"
            - "traefik.http.routers.tomcatapi.rule=Host(`tomcat.mydomain.com`) && PathPrefix(`/api/v1/`)"
            - "traefik.http.routers.tomcatapi.entrypoints=web-secure"
            - "traefik.http.routers.tomcatapi.tls=true"

version: '3.3'

services:
    traefik:
        image: "traefik:v2.0.1"
        container_name: "traefik"
        command:
            - --log.level=INFO
            - --log.format=json
            - --api.insecure=true
            - --api.dashboard=true
            - --providers.docker=true
            - --providers.docker.exposedByDefault=false
            - --providers.file=true
            - --providers.file.filename=/etc/traefik/dynamic_config.toml
            - --entrypoints.traefik.address=:8269
            - --entrypoints.web.address=:8080
            - --entrypoints.web-secure.address=:8443
        ports:
            - "8080:8080"
            - "8443:8443"
            - "8269:8269"
        volumes:
            - ./dynamic_config.toml:/etc/traefik/dynamic_config.toml:ro
            - /var/run/docker.sock:/var/run/docker.sock

    tomcatapi:
        image: containous/whoami:v1.4.0
        container_name: "tomcatapi"
        labels:
            - "traefik.enable=true"
            - "traefik.http.routers.tomcatapi.rule=Host(`tomcat.mydomain.com`) && PathPrefix(`/api/v1/`)"
            - "traefik.http.routers.tomcatapi.entrypoints=web-secure"
            - "traefik.http.routers.tomcatapi.tls=true"
[http]
  [http.services]
    [http.services.tomcat-web.loadBalancer]
      [[http.services.tomcat-web.loadBalancer.servers]]
        url = "https://www.google.com"

  [http.middlewares]
    # [http.middlewares.redirect.redirectscheme]
    #   scheme = "https"
    [http.middlewares.redirect.redirectregex]
      regex = "^(http:\\/\\/)?([\\w\\._-]+)(:\\d+)?(.*)$"
      replacement = "http://${2}:8443${4}"

  [http.routers]
    [http.routers.tomcat]
      entryPoints = ["web"]
      rule = "Host(`tomcat.mydomain.com`)"
      service = "tomcat-web"
      middlewares = ["redirect"]

    [http.routers.tomcatssl]
      entryPoints = ["web-secure"]
      rule = "Host(`tomcat.mydomain.com`)"
      service = "tomcat-web"
      [http.routers.tomcatssl.tls]
curl -k -H Host:tomcat.mydomain.com https://127.0.0.1:8443/api/v1/
# call the whoami
curl -k -H Host:tomcat.mydomain.com https://127.0.0.1:8443
# call google