[fixed] ACME config seems broken in beta1

EDIT

If you're just coming to this thread do these three things:

  1. Delete the contents of your acme.json file.
  2. Make sure to set certResolver on your routers
  3. The acme section of the file config has changed to certificatesResolvers and should be updated.

I've upgraded to beta1 and all of my ACME configuration has died, this is my setup:

[certificateResolvers.cloudflare.acme]
  email = "email@gmail.com"
  storage = "/acme.json"
  [certificateResolvers.cloudflare.acme.dnsChallenge]
    provider = "cloudflare"
    resolvers = ["1.1.1.1:53", "1.0.0.1:53"]

In addition, I have a traefik.env file I'm loading into the traefik container which defines the following (note that the docs here[0], differ from the lego docs here[1], so I've included both:

  • CLOUDFLARE_API_KEY
  • CLOUDFLARE_EMAIL
  • CF_API_EMAIL
  • CF_API_KEY

I'm not getting any error messages in my Traefik log other than that Traefik is using the default self-signed cert for TLS on connections to all my containers.

Anyone spot what I'm messing up here?

It also appears that the reference doc[2], does not match what is currently stated in the Let's Encrypt docs section[3]. The LE section shows the new [certificateResolvers] field while the reference still has the [acme] section.

[0] - https://docs.traefik.io/v2.0/https/acme/#providers
[1] - https://go-acme.github.io/lego/dns/cloudflare/#credentials
[2] - https://docs.traefik.io/v2.0/https/acme/#configuration-examples
[3] - https://docs.traefik.io/v2.0/reference/static-configuration/file/

1 Like

You have to set a certResolver inside the tls section of your routers.

https://docs.traefik.io/v2.0/routing/routers/#certresolver

  • CLOUDFLARE_API_KEY is an alias to CF_API_EMAIL
  • CLOUDFLARE_EMAIL is an alias to CF_API_KEY
1 Like

hi,

I have a similar issue too but with tlsChallenge after I updated via the guideline https://docs.traefik.io/v2.0/user-guides/crd-acme/. The error message is
Unable to obtain ACME certificate for domains \"app.test.com\": cannot get ACME client ACME challenge not specified, please select TLS or HTTP or DNS Challenge"

Then I changed to use tls secret name with a self-signed certificate and another error show up

Error configuring TLS: secret test/test-tls-secret does not exist" providerName=kubernetescrd ingress=ingressroutetls namespace=test

Not sure if I missed sth in the config. I checked that the secret already existed in test namespace

There is some more context here: https://github.com/containous/traefik/pull/5121#pullrequestreview-264478971

But I am still not able to get it run.

hi @dduportal-test,

My deployment yml is

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: test
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0-alpine
          args:
            - --global.sendAnonymousUsage=false
            - --api
            - --log.level=DEBUG
            - --entrypoints.http-tls.Address=:4443
            - --providers.kubernetescrd
            - --providers.kubernetescrd.namespaces=test
            - --certificatesresolvers.test.acme.tlsChallenge
            - --certificatesresolvers.test.acme.email=ngan@test.com
            - --certificatesresolvers.test.acme.storage=acme.json
          ports:
            - name: http
              containerPort: 8000
            - name: http-tls
              containerPort: 4443

I turned on the debug mode and see this line

Static configuration loaded ...
"certificatesResolvers":{
    "test":{
    "acme":{
        "caServer":"https://acme-v02.api.letsencrypt.org/directory",
        "storage":"acme.json",
        "keyType":"RSA4096"}}}

seems it cannot parse the email and tlsChallenge

@ngan.huynh

try that:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  namespace: test
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.0-alpine
          args:
            - --global.sendAnonymousUsage=false
            - --api=true
            - --log.level=DEBUG
            - --entrypoints.http-tls.Address=:443
            - --providers.kubernetescrd.namespaces=test
            - --certificatesresolvers.test.acme.tlsChallenge=true
            - --certificatesresolvers.test.acme.email=ngan@test.com
            - --certificatesresolvers.test.acme.storage=acme.json
            - --certificatesresolvers.basic.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
          ports:
            - name: http
              containerPort: 8000
            - name: http-tls
              containerPort: 443

To use the TLS challenge you have to use the 443.

In the docker-compose file that I put in the issue, I forget to add =true, it's the joy of the docker-compose file parsing :smile:

version: "3.3"

services:

  traefik:
    image: "traefik:v2.0.0-beta1"
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --api.dashboard=true
      - --providers.docker=true
      - --global.sendAnonymousUsage=true
      # - --log.level=DEBUG
      - --certificatesresolvers.basic.acme.email=postmaster@mydomain.com
      - --certificatesresolvers.basic.acme.storage=/config/acme.json
      - --certificatesresolvers.basic.acme.tlsChallenge=true
      - --certificatesresolvers.basic.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    labels:
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https@docker"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    volumes:
      - "./acme.json:/config/acme.json"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: containous/whoami
    labels:
      - "traefik.http.routers.whoami.rule=Host(`mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=basic"

it works. thank you.

I have another issue with tls using secret name. Not sure if I should create a new one?

Unfortunately, no.

In the above example, I had my resolver cloudflare set, I then added this certresolver to my static and docker-compose config.

traefik.toml

  [http.routers.synology-https]
      entryPoints = ["https", "http"]
      middlewares = [ "compress", "basic-auth"]
      rule = "Host(`synology.sub.domain.com`)"
      service = "synology"
      [http.routers.synology-https.tls]
        certresolver = "cloudflare"

docker-compose.yml

    ...
      - "traefik.http.routers.pihole.tls"
      - "traefik.http.routers.pihole.tls.certresolver=cloudflare"
    ...

But I'm getting this error:

traefik                 | time="2019-07-21T12:21:03Z" level=error msg="the router synology-https uses a non-existent resolver: cloudflare"

And it seems it's not even possible to specify the resolver within docker-compose? As there are no error messages stating that the docker services "can't find a resolver'.

certresolver should also appear somewhere in here: https://docs.traefik.io/v2.0/reference/dynamic-configuration/docker/

1 Like

It works with docker-compose.

The doc is not up to date see https://github.com/containous/traefik/pull/5132

Could you provide your docker-compose file.

I am still getting the following error:

[certificatesResolvers.basic.acme]
  email = "my@email.com"
  storage = "/etc/traefik/acme/acme.json"

  [certificatesResolvers.basic.acme.httpChallenge]
    entryPoint = "web"
time="2019-07-21T08:54:09Z" level=error msg="the router jenkins_https uses a non-existent resolver: basic"

This is currently undocumented and there is no example how this would look in file based config:

--certificatesResolvers.sample.acme.httpChallenge=true

Still not working: the router jenkins_https uses a non-existent resolver: basic

--certificatesResolvers.sample.acme.httpChallenge=true is not needed because you must provide an entryPoint (--certificatesResolvers.sample.acme.httpChallenge.entrypoint=foobar)

Could you provide all elements (docker-compose.yml, traefik.toml, etc.)

Thanks for having a look - appreciated!

traefik.toml
[global]
  checkNewVersion = false
  sendAnonymousUsage = false

[entryPoints]
  [entryPoints.web]
    address = ":80"

    [entryPoints.web.proxyProtocol]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    [entryPoints.web.forwardedHeaders]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

  [entryPoints.web_secure]
    address = ":443"

    [entryPoints.web_secure.proxyProtocol]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    [entryPoints.web_secure.forwardedHeaders]
      trustedIPs = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

# beta
# [certificatesResolvers.basic.acme]
#   email = "bofh@REDACTED.nl"
#   storage = "/etc/traefik/acme/acme.json"

#   [certificatesResolvers.basic.acme.httpChallenge]
#     entryPoint = "web"

[acme]
  email = "bofh@REDACTED.nl"
  storage = "/etc/traefik/acme/acme.json"
  onHostRule = true # dynamic generation based on the Host() & HostSNI() matchers
  [acme.httpChallenge]
    entryPoint = "web"

[providers]
  [providers.docker]
    endpoint = "unix:///var/run/docker.sock"
    exposedByDefault = false
    defaultRule = "Host(`{{ normalize .Name }}.srv.REDACTED.nl`)"
    network = "services"

  [providers.file]
    filename = "/etc/traefik/dynamic_conf.toml"

(note: I switched back to alpha8 for now)

dynamic_conf.toml
[http]
  [http.middlewares]
    [http.middlewares.compression.Compress]

    # cloudflare whitelist
    [http.middlewares.cloudflare_whitelist.ipWhiteList]
      sourceRange = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    # https redirect
    [http.middlewares.https_redirect.redirectscheme]
      scheme = "https"
      permanent = true

And then I use terraform to spin-up a docker container with the following labels:

  labels {
    traefik.enable         = "true"
    traefik.docker.network = "services"

    # http-router
    traefik.http.routers.jenkins_http.entryPoints = "web"
    traefik.http.routers.jenkins_http.rule        = "Host(`jenkins.REDACTED.nl`)"
    traefik.http.routers.jenkins_http.middlewares = "cloudflare_whitelist@file,https_redirect@file"

    # https-router
    traefik.http.routers.jenkins_https.entryPoints      = "web_secure"
    traefik.http.routers.jenkins_https.rule             = "Host(`jenkins.REDACTED.nl`)"
    traefik.http.routers.jenkins_https.tls              = "true"
    #traefik.http.routers.jenkins_https.tls.certresolver = "basic"
    traefik.http.routers.jenkins_https.middlewares      = "cloudflare_whitelist@file,compression@file"
  }

(also here I am currently commenting it out because I reverted back to alpha8).

I am not running traefik itself with any docker labels.

The following example works:

docker-compose.yml
version: "3.3"

services:

  traefik:
    image: "traefik:v2.0.0-beta1"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "./acme.json:/etc/traefik/acme/acme.json"
      - "./dynamic_conf.toml:/etc/traefik/dynamic_conf.toml"
      - "./traefik.toml:/traefik.toml"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami:
    image: containous/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=services"
        # http-router
      - "traefik.http.routers.jenkins_http.entryPoints=web"
      - "traefik.http.routers.jenkins_http.rule=Host(`jenkins.REDACTED.nl`)"
      - "traefik.http.routers.jenkins_http.middlewares=cloudflare_whitelist@file,https_redirect@file"
        # https-router
      - "traefik.http.routers.jenkins_https.entryPoints=web_secure"
      - "traefik.http.routers.jenkins_https.rule=Host(`jenkins.REDACTED.nl`)"
      - "traefik.http.routers.jenkins_https.tls.certresolver=basic"
      - "traefik.http.routers.jenkins_https.middlewares=cloudflare_whitelist@file,compression@file"

traefik.toml
[global]
  checkNewVersion = false
  sendAnonymousUsage = false

[log]
  level="DEBUG"

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.web_secure]
    address = ":443"

# beta
[certificatesResolvers.basic.acme]
  email = "bofh@REDACTED.nl"
  storage = "/etc/traefik/acme/acme.json"
  caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

  [certificatesResolvers.basic.acme.httpChallenge]
    entryPoint = "web"

[providers]
  [providers.docker]
    exposedByDefault = false
    defaultRule = "Host(`{{ normalize .Name }}.srv.REDACTED.nl`)"

  [providers.file]
    filename = "/etc/traefik/dynamic_conf.toml"
dynamic_conf.toml
[http]
  [http.middlewares]
    [http.middlewares.compression.compress]

    # cloudflare whitelist
    [http.middlewares.cloudflare_whitelist.ipWhiteList]
      sourceRange = ["173.245.48.0/20", "103.21.244.0/22", "103.22.200.0/22", "103.31.4.0/22", "141.101.64.0/18", "108.162.192.0/18", "190.93.240.0/20", "188.114.96.0/20", "197.234.240.0/22", "198.41.128.0/17", "162.158.0.0/15", "104.16.0.0/12", "172.64.0.0/13", "131.0.72.0/22", "2400:cb00::/32", "2606:4700::/32", "2803:f800::/32", "2405:b500::/32", "2405:8100::/32", "2a06:98c0::/29", "2c0f:f248::/32"]

    # https redirect
    [http.middlewares.https_redirect.redirectscheme]
      scheme = "https"
      permanent = true

Still not working same error as reported by OP - its just not finding "certificatesResolvers.base". Maybe traefik only works with docker-compose and not terraform.

All good, though will stick to alpha and wait for final release. Appreciate the help.

Traefik don't see difference between a docker-compose or terraform, the final release will have the same behavior, so you have to check your terraform configuration.

I've gotten mine working again this morning. My problem was twofold:

  1. I was using certificateResolvers instead of certifcatesResolvers (arguably the former sounds more natural in English :sweat_smile: )
  2. As in my pihole example above, I had a label on my containers for .tlsand .tls.certresolver=cloudflare and only needed the latter.

Hi all,

@lifeofguenter do you persist the storage of your certificates during the update to beta ?
I got the exact same issue as you, and my problem was solved by removing the acme.json containing the certificates issued by the alpha version. If you remove those before updating, you get no errors, and the certificates are issued again by the beta.

I managed to reproduce this behavior with the following compose file :

[details="docker-compose.yml"]

version: "3.3"

services:

  traefik-beta:
    image: "traefik:v2.0.0-beta1"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--global.sendanonymoususage=false"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge=true"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myhttpchallenge.acme.email=postmaster@mydomain.com"
      - "--certificatesresolvers.myhttpchallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "127.0.0.1:8080:8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami-beta:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls.certresolver=myhttpchallenge"

  traefik-alpha:
    image: "traefik:v2.0.0-alpha8"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api=true"
      - "--global.sendanonymoususage=false"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--acme.email=postmaster@mydomain.com"
      - "--acme.storage=/letsencrypt/acme.json"
      - "--acme.httpchallenge=true"
      - "--acme.httpchallenge.entrypoint=web"
      - "--acme.acmelogging=true"
      - "--acme.onHostRule=true"
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
      - "127.0.0.1:8080:8080"
    volumes:
      - "./letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  whoami-alpha:
    image: "containous/whoami"
    container_name: "simple-service"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami.entrypoints=websecure"
      - "traefik.http.routers.whoami.tls=true"

then issuing :

  • docker-compose up -d traefik-alpha whoami-alpha
  • wait for the cert to be issued
  • docker-compose down
  • docker-compose up -d traefik-beta whoami-beta
  • you get the error : level=error msg="the router whoami uses a non-existent resolver: myhttpchallenge"
  • docker-compose down
  • rm -r letsencrypt
  • docker-compose up -d traefik-beta whoami-beta
  • the certs are now issued properly

@ldez Do you think this is worth opening an issue ?

Nice evening.

2 Likes

We changed the format of the acme.json file.
Then it's required to remove the acme.json file from the alpha stage.

It's the expected behavior.

2 Likes