Cannot retrieve the ACME challenge for token

Traefik Traefik v2 (latest)
,
1

Hi there.

Hoping someone can help me understand what's going on with my setup.

I've set up traefik in the following manner (Much based on this guide).

On the surface It's running great, new certs get applied to the domains no problem. But I've been seeing traefik logs getting spammed with Error getting challenge for token, and can't help but think this will be an issue later on upon renewal? :

Logs

time="2020-02-13T11:57:04+01:00" level=error msg="Error getting challenge for token retrying in 15.837365632s" providerName=le.acme
time="2020-02-13T11:57:14+01:00" level=error msg="Error getting challenge for token retrying in 16.545427742s" providerName=le.acme
time="2020-02-13T11:57:15+01:00" level=error msg="Error getting challenge for token retrying in 12.724799482s" providerName=le.acme
time="2020-02-13T11:57:15+01:00" level=error msg="Error getting challenge for token retrying in 22.361676958s" providerName=le.acme
time="2020-02-13T11:57:20+01:00" level=error msg="Error getting challenge for token retrying in 34.017080431s" providerName=le.acme
time="2020-02-13T11:57:28+01:00" level=error msg="Error getting challenge for token retrying in 35.331453297s" providerName=le.acme
time="2020-02-13T11:57:30+01:00" level=error msg="Error getting challenge for token retrying in 40.608267972s" providerName=le.acme
time="2020-02-13T11:57:37+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:57:54+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:58:03+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:58:11+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T12:13:09+01:00" level=error msg="Error while Peeking first byte: read tcp 172.19.0.4:80->80.86.142.130:51370: read: connection reset by peer"

docker-compose.yml

version: '3'
services:
  traefik:
    image: traefik:v2.1.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log/traefik:/var/log
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/rules:/rules:ro
      - ./data/certs/:/certs
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`redacted.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:pwdhash"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`redacted.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=le"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

traefik.yml

api:
  dashboard: true

log:
  filePath: "/var/log/traefik.log"
  level: INFO

metrics:
  influxDB:
    address: http://redacted.com:8086
    protocol: http
    database: "telegraf"
    username: "telegraf"
    password: "pwd"

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: /rules
    watch: true

certificatesResolvers:
  le:
    acme:
      email: user@redacted.com
      storage: acme.json
      httpChallenge:
        entryPoint: http

rules/tls.yml
wildcard cert from comodo added here is the one I'm using for my main domain and several subdomains.

tls:
  certificates:
    - certFile: /certs/redacted.com.crt
      keyFile: /certs/redacted.com.key
  options:
    TLSv13:
      minVersion: VersionTLS13
      cipherSuites:
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
      sniStrict: true
        
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      sniStrict: true

rules/middlewares.yml

http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "XXX.XXX.XXX.XXX/XX"
        - "172.0.0.0/8"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

A site's docker-compose.yml

version: '3.3'
services:
  wordpress:
    image: wordpress:5-fpm
    container_name: redacted_com-wp
    restart: always
    networks:
      - proxy
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./config/php.ini:/usr/local/etc/php/conf.d/custom.ini
      - /srv/redacted.com:/var/www/html

  nginx:
    image: nginx
    container_name: redacted_com
    depends_on:
      - wordpress
    restart: always
    networks:
      - proxy
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /srv/redacted.com:/var/www/html
      - ./config/nginx.conf:/etc/nginx/conf.d/default.conf
      - /var/log/nginx:/var/log/nginx
    labels:
      - "docker.group=redacted.com"
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.middlewares.redirect.redirectregex.regex=^http://(.*)"
      - "traefik.http.middlewares.redirect.redirectregex.replacement=https://$$1"
      - "traefik.http.routers.redacted_com.entrypoints=http"
      - "traefik.http.routers.redacted_com.rule=Host(`redacted.com`, `www.redacted.com`)"
      - "traefik.http.routers.redacted_com.middlewares=https-redirect@file"
      - "traefik.http.routers.redacted_com-secure.entrypoints=https"
      - "traefik.http.routers.redacted_com-secure.rule=Host(`redacted.com`, `www.redacted.com`)"
      - "traefik.http.routers.redacted_com-secure.tls=true"
      - "traefik.http.routers.redacted_com-secure.tls.certresolver=le"

networks:
  proxy:
    external: true

Port 80 is open in my firewall. I do run fail2ban on my site's access_log and a couple of other services but would not think that could be the issue?

Can anybody see what I'm doing wrong?

2

first, I think theses two lines are not needed:

since you have this one:

(also you don't use the redirect@docker middleware anywhere)

For your main problem, I do not understand why you have to use LE if you already have a comodo cert...
but since it seems that you need a LE cert for this domain, let's analyse your config:
your redirect middleware try to redirect all traffic from http to https, even the one from letsencrypt to verify your token. and it fails because the connexion is not secure.

3

Thanks for replying :slight_smile:

I realize I should've been more specific. I added those regex lines for a couple of domains because i wanted to redirect from http(s)://www.domain.com to https://domain.com. But some other domains have a dns-provider which has a built-in www redirect option. So it's convenient to use labels.

My main domain has a comodo wildcard cert but i have other domains that I want to use with LE. But, I see now that I have the le cert resolver in my traefik docker labels, that's wrong.

Hm, what's the redirect@docker middleware?

My main traefik docker-compose now looks like this:

version: '3'
services:
  traefik:
    image: traefik:v2.1.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log/traefik:/var/log
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/rules:/rules:ro
      - ./data/certs/:/certs
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`redacted.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:pwdhash"
        #      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
        #      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`redacted.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
        #      - "traefik.http.routers.traefik-secure.tls.certresolver=le"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

After a restart I'm watching the logs closely, everything seems to work.
Though doing a curl -i http://maintraefikdomain.com time-outs with a 502 Bad Gateway.

EDIT: Still got multiple "Error getting challenge for token..." I figured as much, since I got a 502.

4

I'm wondering if this could be related to ipv6 issue? I had not disabled ipv6 on this box but just did that now. Crossing fingers that this is indeed the issue here.

EDIT: That was not it :confused:

Also it's a bit confusing but I think I recall reading somewhere that a https redirect would not affect acme certificate resolver because it uses some special underlying route or something. Maybe someone can clarify?

5

Still at a loss here. Been googling like crazy. Hoping some wizard would come by and point out what is going on.

I came across one issue post that is quite similar to mine, they were also using another wildcard certificate for the main domain.

bilde

6

I'm still seeing this :confused: Anybody ever seen this before?

7

I just got this today when I restarted my Traefik 1.7 instance.

8

I am having this issue with Traefik 2.3 - Not sure why.

9

I've notice something similar in the logs of a trafik proxy I have. I've got those logs :

level=error msg="Error getting challenge for token retrying in 14.819788117s" providerName=acme
level=error msg="Error getting challenge for token retrying in 28.76650848s" providerName=acme
level=error msg="Cannot retrieve the ACME challenge for redacted.com (token "xGNUUrCBUk8n-kvfAgqfxrlR6zvNJ-MXyxOtf9GEH9A"): cannot find challenge for token "xGNUUrCBUk8n-kvfAgqfxrlR6zvNJ-MXyxOtf9GEH9A" (redacted.com)" providerName=acme

But the thing is that certificate is still valid until july 21st, more than 30 days from now. It shoudn't try to renew it. Debug logs hasn't been more helpfull.

At arround that time, I do see in access_logs requests for /.well-known/acme-challenge/(...) ending in 404.

I've also tried to change to tlsChallenge, and still see requests for challenges.

10

I found out that the old hosting provider still had the site configured. They were trying to renew the certificate on their end, so that was the errors I was getting were.

2 Likes
11

Sorry for reviving an old thread, but I have the exact same problem as described in the first post and I can't find a solution. There are about 100 of these lines per day.

Seemingly, everything works, but my log is getting filled with "Cannot retrieve the ACME challenge for" and "Error getting challenge for token".

Here is a description of my system:

I am using a middleware to redirect all traffic for myhost.com to https
I am also using a middleware to strip "www."
My certs are correctly generated for both www.myhost.com and myhost.com
I am using the httpchallenge for letsencrypt. Even though it's not logical to use httpchallenge if everything is redirected to https, as far as I understand it should still work (and obviously did work at least once to generate the certs).

I checked that there are no other apps trying to generate a cert for this domain (as described by nicolasross).

my dynamic configuration as set via docker labels:

      #router definition. one letsencrypt cert will be generated for both domains
      - traefik.http.routers.ddd-varnish.rule=Host(`myhost.com`, `www.myhost.com`)
      - traefik.http.routers.ddd-varnish.tls=true
      - traefik.http.routers.ddd-varnish.tls.certresolver=myresolver
      - traefik.http.routers.ddd-varnish.middlewares=stripWww@file
      #define another router to handle http connections
      - traefik.http.routers.ddd-varnish-http.rule=Host(`myhost.com`, `www.myhost.com`)
      - traefik.http.routers.ddd-varnish-http.middlewares=toHttps@file

My middlewares in the file myDynamicConf.toml:

[http.middlewares]
  [http.middlewares.toHttps.redirectScheme]
    scheme="https"
    permanent="true"

  [http.middlewares.stripWww.redirectRegex]
    regex = "^https?://(?:www\\.)?(.+)"
    replacement = "https://${1}"
    permanent = true

My config.toml (relevant parts only):

[entryPoints]
  [entryPoints.web]
    address = ":80"

  [entryPoints.websecure]
    address = ":443"

[certificatesResolvers.myresolver.acme]
  email = "redacted"
  storage = "acme.json"

  # use letsencrypt staging during testing
  #caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

  [certificatesResolvers.myresolver.acme.httpChallenge]
    # used during the challenge
    entryPoint = "web"

On my traefik dashboard there is a router acme-http@internal with the rule PathPrefix(/.well-known/acme-challenge/). I can visit this http webpage with my browser and I am not being upgraded to https, which means whatever is hiding behind this url should be reachable by letsencrypt. But I do get a 404 error. Not sure if a website directory should open or not.

My certificate expires in february, so I don't think traefik is trying to renew it right now.

As I said, everything works, but I want to know why those log entries are being created and I'm afraid my renewal may be compromised. Can anyone explain these log entries? Thank you very much.

12

Share your full Traefik static and dynamic config, and docker-compose.yml if used.

13

My Traefik docker-compose.yml:

version: "3.6"

services:

  traefik:
    #build: "."
    image: "traefik:2.10"
    container_name: traefik
    hostname: traefik
    domainname: mydomain.com
    networks:
      - myNetwork

    labels:
      - traefik.http.routers.dashboard.rule=Host(`traefik.mydomain.com`)
      - traefik.http.routers.dashboard.service=api@internal
      - traefik.http.routers.dashboard.middlewares=auth
      - traefik.http.middlewares.auth.basicauth.users=redacted:redacted.

    ports:
    - "80:80"
    - "443:443"
    #dashboard
    - "8080:8080"

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/config.toml:/etc/traefik/traefik.toml
      - $PWD/myDynamicConf:/myDynamicConf
      - $PWD/acme.json:/acme.json
    environment:
      - TZ=Redacted
    restart: always

networks:
  #this is the network name, but it gets prefixed with the parent-directory name unless overridden by name property (docker-compose >3.5)
  myNetwork:
    name: myNetwork
    ipam:
      config:
        #this subnet has an ip range of 192.168.4.1 to 192.168.4.255
        - subnet: 192.168.4.1/24

Traefik config.toml:

#main traefik configuration file

[entryPoints]
  [entryPoints.web]
    address = ":80"
#this is done as a middleware, dont use this below
#    [entryPoints.web.http]
#      [entryPoints.web.http.redirections]
#        [entryPoints.web.http.redirections.entryPoint]
#          to = "websecure"
#          scheme = "https"

  [entryPoints.websecure]
    address = ":443"

[certificatesResolvers.myresolver.acme]
  email = "redacted"
  storage = "acme.json"

  # use letsencrypt staging during testing
  #caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"

  [certificatesResolvers.myresolver.acme.httpChallenge]
    # used during the challenge
    entryPoint = "web"

[api]
  dashboard = true


#generate dynamic configuration based on docker labels. make sure to bind mount /var/run/docker.sock:/var/run/docker.sock
[providers.docker]

#define some common dynamic configuration (middlewares) in a separate file too
[providers.file]
  directory = "/myDynamicConf/"
  watch = true

Traefik dynamic conf:

[http.middlewares]
  [http.middlewares.toHttps.redirectScheme]
    scheme="https"
    permanent="true"

  [http.middlewares.stripWww.redirectRegex]
    regex = "^https?://(?:www\\.)?(.+)"
    replacement = "https://${1}"
    permanent = true

My container docker-compose yml (Varnish cache that caches stuff for multiple websites, each in its own container):

version: "3.6"

services:

  varnish:
#    build: "."
    image: "varnish:7.3"
    container_name: varnish
    hostname: varnish
    networks:
      - myNetwork

    labels:
      #router definition. one letsencrypt cert will be generated for both domains
      - traefik.http.routers.mydomain-varnish.rule=Host(`mydomain.com`, `www.mydomain.com`)
      - traefik.http.routers.mydomain-varnish.tls=true
      - traefik.http.routers.mydomain-varnish.tls.certresolver=myresolver
      - traefik.http.routers.mydomain-varnish.middlewares=stripWww@file
      #define another router to handle http connections
      - traefik.http.routers.mydomain-varnish-http.rule=Host(`mydomain.com`, `www.mydomain.com`)
      - traefik.http.routers.mydomain-varnish-http.middlewares=toHttps@file

      #router definition. one letsencrypt cert will be generated for both domains
      - traefik.http.routers.mydomain2-varnish.rule=Host(`mydomain2.si`, `www.mydomain2.si`)
      - traefik.http.routers.mydomain2-varnish.tls=true
      - traefik.http.routers.mydomain2-varnish.tls.certresolver=myresolver
      - traefik.http.routers.mydomain2-varnish.middlewares=stripWww@file
      #define another router to handle http connections
      - traefik.http.routers.mydomain2-varnish-http.rule=Host(`mydomain2.si`, `www.mydomain2.si`)
      - traefik.http.routers.mydomain2-varnish-http.middlewares=toHttps@file

    volumes:
      - $PWD/myvcls:/etc/varnish/
    environment:
     - TZ=Redacted
    restart: always
    command: "-I /etc/varnish/cli-command-file.txt"

networks:
  myNetwork:
    external: true
14

May I recommend to clean up your setup?

Simply place the http-to-https redirect globally on entrypoint, to save a lot of labels.

Enable TLS globally on entrypoint. You enable it on router without specifying an entrypoint, so you enable it on port 80, too.

Use .rule=Host() || Host() to be v3 compatible, which is coming up soon.

Get rid of the redundant TOML format. Okay, that’s just a personal preference :smile:

Check and compare to simple Traefik example for a working setup.

15

I don't want to enable http-to-https redirection globally but per service. I also prefer using docker labels to set up container-specific behavior.

You mentioned redundancy in my config.toml, which part is redundant?

You mentioned "You enable it on router without specifying an entrypoint, so you enable it on port 80, too." I enable the http-to-https middleware on my http router, which is according to my wishes and works as expected. Can you please give me extra info if/why this is the cause of the error, so I understand?

So right now I'm still not sure what the cause of the error is and what I should change. Thanks for your help though, I appreciate it

16

With redundancy I mean the TOML syntax:

Try using router…entrypoints=websecure when using TLS on the router, to not enable TLS on web. Not sure if that is the issue.