Help migrating to v2.0 wildcard ACME and non-existent middleware

Hi,

I'm also relatively new to Traefik v2 and still trying to get my head around how to migrate my v1.x config. Following the docs for v2, I've created a new YAML file for my Traefik container and added a bunch of new labels to my Docker Compose file.

Initially I had a problem with declaring a common middleware across my containers which defines a bunch of HTTPS headers, redirecting HTTP-to-HTTPS and securing the /api endpoint which I've since resolved. I use the DNS-01 ACME challenge for LE certificate generation and have provided my DNSimple OAUTH token and API base URL as env vars to my Traefik container.

Now I have these in place, my Traefik container outputs a couple of errors I just can't seem to get my head around resolving.

  1. level=error msg="Unable to obtain ACME certificate for domains \"vacuum.sub.domain.com\": unable to generate a certificate for the domains ... : acme: Error -> One or more domains had a problem:\n vacuum.sub.domain.com acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: No TXT record found at _acme-challenge.sub.domain.com, url: \n" providerName=default.acme rule="Host(vacuum.sub.domain.com)" routerName=vacuum

  2. time="2019-09-25T05:16:17+02:00" level=error msg="middleware \"default@docker\" does not exist" entryPointName=web-secure routerName=vacuum@docker

Would really appreciate a bit of help understanding how I can resolve these and simplify my config.

log:
  level: "ERROR"
api:
  dashboard: true
http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
certificatesResolvers:
  default:
    acme:
      email: service-certs@sub.domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

docker-compose.yml

My goal here is to serve the API endpoint externally with basic auth for now, and publish all services as children of the sub-domain.

Hello there,

do you know about the difference between static and dynamic configuration in traefik?

Still getting me head around that and I think I might be best to use a static configuration for the middleware I'm calling 'default' which has the SSH header options.

So in my mind, I would create a file provider in traefik.yml like this:

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    filename: middleware.yml

that references something like middleware.yml and move all my relevant middleware rules there like so:

http:
  middlewares:
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https

Then in Docker Compose, if I've got this correct, I would reference the middleware as:

- "traefik.http.routers.vacuum.middlewares=default@file"

Have I got that right?

Yep, that's sounds about right to me. Try that, and if you still get errors you cannot figure out, post here again!

I think you can also specify middleware via labels as per here but I personally would try to do what you are doing, because to me the yaml looks nicer than the corresponding labels.

Indeed, I can just see easily 20 labels per container, not so fun to maintain when if there is an option to declare a common set somewhere else as a directive.

Will try and see what turns up next :slight_smile:

Going further down the rabbit hole it seems instead of getting out!

Now getting:

time="2019-09-25T11:06:06+02:00" level=error msg="cannot create middleware: multi-types middleware not supported, consider declaring two different pieces of middleware instead" entryPointName=web routerName=vacuum@docker

and

time="2019-09-25T11:06:07+02:00" level=error msg="cannot create middleware: multi-types middleware not supported, consider declaring two different pieces of middleware instead" routerName=vacuum@docker entryPointName=web-secure

Also getting this

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] [*.domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n[domain.com] time limit exceeded: last error: NS ns3.dnsimple.com. returned NXDOMAIN for _acme-challenge.domain.com.\n" providerName=default.acme
time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.domain.com., value: ...\n[domain.com] [domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n" providerName=default.acme

My middleware.yml looks like this:

http:
  middlewares:
    default:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true
      redirectScheme:
        scheme: https

traefik.yml is now this:

log:
  level: "ERROR"
api:
  dashboard: true
http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    filename: /etc/traefik/middleware.yml
entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"
certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

and my updated Docker compose file now looks like this:

version: '3.7'
services:

  octopus:
    container_name: "proxy"
    image: traefik:2.0
    restart: unless-stopped
    networks:
      - frontend
      - internal
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /proxy/config/:/etc/traefik
    env_file: ./proxy.env
    depends_on:
      - vacuum
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.octopus.rule=Host(`octopus.sub.domain.com`)"
      - "traefik.http.routers.octopus.entrypoints=web, web-secure"
      - "traefik.http.routers.octopus.tls.certresolver=default"
      - "traefik.http.routers.octopus.tls.domains[0].main=domain.com"
      - "traefik.http.routers.octopus.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.octopus.service=octopus"
      - "traefik.http.routers.octopus.middlewares=default@file"
      - "traefik.http.services.octopus.loadbalancer.server.port=8080"
      - "traefik.http.services.octopus.loadbalancer.server.scheme=http"
  
  vacuum:
    container_name: "deluge"
    image: linuxserver/deluge
    restart: unless-stopped
    networks:
      - frontend
    ports:
      - "58846:58846"
      - "58946:58946"
      - "58946:58946/udp"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /deluge/config:/config
      - /staging/downloads:/downloads
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.vacuum.rule=Host(`vacuum.sub.domain.com`)"
      - "traefik.http.routers.vacuum.entrypoints=web, web-secure"
      - "traefik.http.routers.vacuum.tls.certresolver=default"
      - "traefik.http.routers.vacuum.tls.domains[0].main=domain.com"
      - "traefik.http.routers.vacuum.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.vacuum.service=vacuum"
      - "traefik.http.routers.vacuum.middlewares=default@file"
      - "traefik.http.services.vacuum.loadbalancer.server.port=8112"
      - "traefik.http.services.vacuum.loadbalancer.server.scheme=http"

Definitely stuck right now and not sure where to go with this :confused:

Note the domain I'm using is just an example, the domain I'm actually using reports this error.

You can not create multi-types middleware, consider declaring two different pieces of middleware instead:

http:
  middlewares:

    head:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true

    foobar:
      redirectScheme:
        scheme: https
labels:
- "traefik.http.routers.vacuum.middlewares=foobar@file,headers@file"

You can also create a chain:

http:
  middlewares:

    head:
      headers:
        sslRedirect: true
        sslHost: sub.domain.com
        stsSeconds: 315360000
        stsIncludeSubdomains: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        frameDeny: true

    foobar:
      redirectScheme:
        scheme: https

    mychain:
      chain:
        middlewares:
          - head
          - foobar
labels:
- "traefik.http.routers.vacuum.middlewares=mychain@file"
1 Like

Looks like I'm getting somewhere @ldez

I'm now getting the default Traefik self-signed root cert and a number of my endpoints are just returning a 404 with nothing to suggest there's a problem with fetching the cert

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] [*.domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n[domain.com] time limit exceeded: last error: NS ns3.dnsimple.com. returned NXDOMAIN for _acme-challenge.domain.com.\n" providerName=default.acme

Non eXistent DOMAIN, so your domain (domain.com) don't exist or the A record is missing

time="2019-09-25T11:07:16+02:00" level=error msg="Unable to obtain ACME certificate for domains \"domain.com,*.domain.com\" : unable to generate a certificate for the domains [domain.com *.domain.com]: acme: Error -> One or more domains had a problem:\n[*.domain.com] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.domain.com., value: ...\n[domain.com] [domain.com] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/domain.com/records: 400 Zone record already exists\n" providerName=default.acme

You have to clean your TXT records

also you cannot mix dynamic configuration and static configuration in the same file:

traefik.toml:

log:
  level: "ERROR"

api:
  dashboard: true

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    directory: /etc/traefik/config/

entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"

certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0

/etc/traefik/config/api.yml

http:
  routers:
    api:
      rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
      service: api@internal
      middlewares:
        - auth
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"

also move /etc/traefik/middleware.yml to /etc/traefik/config/middleware.yml

/proxy/config/
├── traefik.yml
└── config
    ├── api.yml
    └── middleware.yml
/etc/traefik
├── traefik.yml
└── config
    ├── api.yml
    └── middleware.yml

OK, that last point is really useful to know!! Thx :+1:

Seems I've accidentally hit the LE rate limit so I'll have to wait a bit or move to an alternate domain to continue validating any other issues.

I recommend to use the Let's Encrypt staging server during the test to avoid rate limit:

certificatesResolvers:
  sample:
    acme:
      # ...
      caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      # ...

https://docs.traefik.io/v2.0/https/acme/#caserver

The middlewares can only be define in the dynamic configuration.

dynamic configuration:

  • routers
  • middlewares
  • services
  • tls

documentation:

static configuration:

  • information about the providers to used
  • logger definition
  • ...

documentation:

OK, so besides the invalid cert error now on some container endpoints, a few also just come back as a vanilla 404 from Traefik, it seems the redirect to SSL isn't being followed from the middlewares rule

You have mixed several concept, so I will try to show you the way:

version: '3.7'
services:

  octopus:
    container_name: "proxy"
    image: traefik:2.0
    restart: unless-stopped
    networks:
      - frontend
      - internal
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - /proxy/config/:/etc/traefik
    env_file: ./proxy.env
    depends_on:
      - vacuum
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.dashboard.rule=Host(`octopus.sub.domain.com`)"
      - "traefik.http.routers.dashboard.entrypoints=web-secure"
      - "traefik.http.routers.dashboard.tls.certresolver=default"
      - "traefik.http.routers.dashboard.tls.domains[0].main=domain.com"
      - "traefik.http.routers.dashboard.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.dashboard.service: api@internal"
      - "traefik.http.routers.dashboard.middlewares=auth@file"
  
  vacuum:
    container_name: "deluge"
    image: linuxserver/deluge
    restart: unless-stopped
    networks:
      - frontend
    ports:
      - "58846:58846"
      - "58946:58946"
      - "58946:58946/udp"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /deluge/config:/config
      - /staging/downloads:/downloads
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=frontend"
      - "traefik.http.routers.vacuum.rule=Host(`vacuum.sub.domain.com`)"
      - "traefik.http.routers.vacuum.entrypoints=web-secure"
      - "traefik.http.routers.vacuum.tls.certresolver=default"
      - "traefik.http.routers.vacuum.tls.domains[0].main=domain.com"
      - "traefik.http.routers.vacuum.tls.domains[0].sans=*.domain.com"
      - "traefik.http.routers.vacuum.service=vacuum"
      - "traefik.http.routers.vacuum.middlewares=auth@file"
      - "traefik.http.services.vacuum.loadbalancer.server.port=8112"
/proxy/config/
├── traefik.yml
└── dynamic
    ├── auth.yml
    └── globalredirect.yml
/etc/traefik
├── traefik.yml
└── dynamic
    ├── auth.yml
    └── globalredirect.yml
# /etc/traefik/traefik.yaml
## static configuration

log:
  level: "ERROR"

api: {}

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    watch: true
  file:
    directory: /etc/traefik/dynamic/

entrypoints:
  web:
    address: ":80"
  web-secure:
    address: ":443"

certificatesResolvers:
  default:
    acme:
      email: service-certs@domain.com
      storage: acme.json
      caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: dnsimple
        delayBeforeCheck: 0
# /etc/traefik/dynamic/auth.yaml
## dynamic configuration

# basic auth

http:
  middlewares:
    auth:
      basicAuth:
        users:
        - "redacted:redacted"
## dynamic configuration
# /etc/traefik/dynamic/globalredirect.yml

# create a global redirection to https

http:
  middlewares:
    redirecttohttps:
      redirectScheme:
        scheme: https
  
  routers:
    redirecttohttps:
      rule: "HostRegexp(`{host:.+}`)"
      entryPoints:
      - web
      middlewares:
      - redirecttohttps
      service: "noop"

  services:
    # noop service, the URL will be never called
    noop:
      loadBalancer:
        servers:
        - url: "http://192.168.0.1"
1 Like

This makes much more sense now. Sorry, I wasn't really able to glean these concepts very well from existing documentation, as good as it is.

@ldez - much progress made. I can now reach all my endpoints but I'm now getting a bunch of LE errors for DNS-01 challenge.

time="2019-09-26T00:46:44+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] [*.replicant.vaulted.cloud] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/vaulted.cloud/records: 400 Zone record already exists\n" providerName=default.acme
time="2019-09-26T00:46:53+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] failed to initiate challenge: acme: error: 400 :: POST :: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10441876/qgHu7w :: urn:ietf:params:acme:error:malformed :: Unable to update challenge :: authorization must be pending, url: \n[replicant.vaulted.cloud] [replicant.vaulted.cloud] acme: error presenting token: dnsimple: API call failed: POST https://api.dnsimple.com/v2/74286/zones/vaulted.cloud/records: 400 Zone record already exists\n" providerName=default.acme
time="2019-09-26T00:47:47+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns2.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: nQ8U3r94qlYs92bmRUUv0avdrNG71cp8zjC_qPtuujs]: \n" providerName=default.acme
time="2019-09-26T00:48:12+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] time limit exceeded: last error: NS ns3.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: RoB9OYDpQ8-alOnDH2h8YvnzQKSp2sblQjnZaWMMWJE]: \n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns1.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: ULYdn9s_Ad9UmewOqT5FjBW3_lxJwsC7NjdhHgrOZo4]: \n" providerName=default.acme
time="2019-09-26T00:48:12+02:00" level=error msg="Unable to obtain ACME certificate for domains \"replicant.vaulted.cloud,*.replicant.vaulted.cloud\" : unable to generate a certificate for the domains [replicant.vaulted.cloud *.replicant.vaulted.cloud]: acme: Error -> One or more domains had a problem:\n[*.replicant.vaulted.cloud] time limit exceeded: last error: NS ns2.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: td6zPBqMjOcKfQ-2MdHY8pv5iD0kVqDP1YMe-jf67oQ]: \n[replicant.vaulted.cloud] time limit exceeded: last error: NS ns4.dnsimple.com. did not return the expected TXT record [fqdn: _acme-challenge.replicant.vaulted.cloud., value: Ljx833nFv1eLb8eujx1G1ImUR8wXf_0rIHETTXEZm9U]: \n" providerName=default.acme

I'm using the staging ACME endpoint and this is a domain that exists and was working for DNS-01 challenge previously in Traefik 1.7

You have to clean your TXT records in the DNSimple admin panel.

@ldez I have 3 TXT records currently in this zone, none for _acme-challenge.replicant.vaulted.cloud

Other records are

vaulted.cloud. 3600 IN TXT "v=spf1 ... ~all"
vaulted.cloud. 3600 IN TXT "google-site-verification=..."
google._domainkey.vaulted.cloud. 3600 IN TXT ...

I can also see a bunch of TXT record deletes in my audit trail in DNSimple

Record Delete TXT _acme-challenge.replicant.vaulted.cloud ULYdn9s_Ad9UmewOqT5FjBW3_lxJwsC7NjdhHgrOZo4 redacted Sep 25, 2019 22:46:42
Record Delete TXT _acme-challenge.replicant.vaulted.cloud nQ8U3r94qlYs92bmRUUv0avdrNG71cp8zjC_qPtuujs redacted Sep 25, 2019 22:46:42
Record Delete TXT _acme-challenge.replicant.vaulted.cloud Ljx833nFv1eLb8eujx1G1ImUR8wXf_0rIHETTXEZm9U redacted Sep 25, 2019 22:46:42

I think it's creating a duplicate _acme-challenge TXT record, one for the root domain (I only have A records for the apex, not CNAME's) and then attempting to create another for *.replicant.vaulted.cloud

I see https://github.com/containous/traefik/issues/5317 is already tracking this

The code about DNSimple in the same between v1 and v2.

So could you re-check that similar config works with the v1.7?