Cannot retrieve the ACME challenge for token

Hi there.

Hoping someone can help me understand what's going on with my setup.

I've set up traefik in the following manner (Much based on this guide).

On the surface It's running great, new certs get applied to the domains no problem. But I've been seeing traefik logs getting spammed with Error getting challenge for token, and can't help but think this will be an issue later on upon renewal? :

Logs

time="2020-02-13T11:57:04+01:00" level=error msg="Error getting challenge for token retrying in 15.837365632s" providerName=le.acme
time="2020-02-13T11:57:14+01:00" level=error msg="Error getting challenge for token retrying in 16.545427742s" providerName=le.acme
time="2020-02-13T11:57:15+01:00" level=error msg="Error getting challenge for token retrying in 12.724799482s" providerName=le.acme
time="2020-02-13T11:57:15+01:00" level=error msg="Error getting challenge for token retrying in 22.361676958s" providerName=le.acme
time="2020-02-13T11:57:20+01:00" level=error msg="Error getting challenge for token retrying in 34.017080431s" providerName=le.acme
time="2020-02-13T11:57:28+01:00" level=error msg="Error getting challenge for token retrying in 35.331453297s" providerName=le.acme
time="2020-02-13T11:57:30+01:00" level=error msg="Error getting challenge for token retrying in 40.608267972s" providerName=le.acme
time="2020-02-13T11:57:37+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:57:54+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:58:03+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T11:58:11+01:00" level=error msg="Cannot retrieve the ACME challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA: cannot find challenge for token DLaCCdSqDLgbK9q3SQ9b8-bTHr6nj1Q5sYfd9QdnoBmA" providerName=le.acme
time="2020-02-13T12:13:09+01:00" level=error msg="Error while Peeking first byte: read tcp 172.19.0.4:80->80.86.142.130:51370: read: connection reset by peer"

docker-compose.yml

version: '3'
services:
  traefik:
    image: traefik:v2.1.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log/traefik:/var/log
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/rules:/rules:ro
      - ./data/certs/:/certs
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`redacted.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:pwdhash"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`redacted.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=le"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

traefik.yml

api:
  dashboard: true

log:
  filePath: "/var/log/traefik.log"
  level: INFO

metrics:
  influxDB:
    address: http://redacted.com:8086
    protocol: http
    database: "telegraf"
    username: "telegraf"
    password: "pwd"

entryPoints:
  http:
    address: ":80"
  https:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: /rules
    watch: true

certificatesResolvers:
  le:
    acme:
      email: user@redacted.com
      storage: acme.json
      httpChallenge:
        entryPoint: http

rules/tls.yml
wildcard cert from comodo added here is the one I'm using for my main domain and several subdomains.

tls:
  certificates:
    - certFile: /certs/redacted.com.crt
      keyFile: /certs/redacted.com.key
  options:
    TLSv13:
      minVersion: VersionTLS13
      cipherSuites:
        - TLS_AES_128_GCM_SHA256
        - TLS_AES_256_GCM_SHA384
        - TLS_CHACHA20_POLY1305_SHA256
      sniStrict: true
        
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      sniStrict: true

rules/middlewares.yml

http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true

    default-whitelist:
      ipWhiteList:
        sourceRange:
        - "XXX.XXX.XXX.XXX/XX"
        - "172.0.0.0/8"

    secured:
      chain:
        middlewares:
        - default-whitelist
        - default-headers

A site's docker-compose.yml

version: '3.3'
services:
  wordpress:
    image: wordpress:5-fpm
    container_name: redacted_com-wp
    restart: always
    networks:
      - proxy
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./config/php.ini:/usr/local/etc/php/conf.d/custom.ini
      - /srv/redacted.com:/var/www/html

  nginx:
    image: nginx
    container_name: redacted_com
    depends_on:
      - wordpress
    restart: always
    networks:
      - proxy
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /srv/redacted.com:/var/www/html
      - ./config/nginx.conf:/etc/nginx/conf.d/default.conf
      - /var/log/nginx:/var/log/nginx
    labels:
      - "docker.group=redacted.com"
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.middlewares.redirect.redirectregex.regex=^http://(.*)"
      - "traefik.http.middlewares.redirect.redirectregex.replacement=https://$$1"
      - "traefik.http.routers.redacted_com.entrypoints=http"
      - "traefik.http.routers.redacted_com.rule=Host(`redacted.com`, `www.redacted.com`)"
      - "traefik.http.routers.redacted_com.middlewares=https-redirect@file"
      - "traefik.http.routers.redacted_com-secure.entrypoints=https"
      - "traefik.http.routers.redacted_com-secure.rule=Host(`redacted.com`, `www.redacted.com`)"
      - "traefik.http.routers.redacted_com-secure.tls=true"
      - "traefik.http.routers.redacted_com-secure.tls.certresolver=le"

networks:
  proxy:
    external: true

Port 80 is open in my firewall. I do run fail2ban on my site's access_log and a couple of other services but would not think that could be the issue?

Can anybody see what I'm doing wrong?

first, I think theses two lines are not needed:

since you have this one:

(also you don't use the redirect@docker middleware anywhere)

For your main problem, I do not understand why you have to use LE if you already have a comodo cert...
but since it seems that you need a LE cert for this domain, let's analyse your config:
your redirect middleware try to redirect all traffic from http to https, even the one from letsencrypt to verify your token. and it fails because the connexion is not secure.

Thanks for replying :slight_smile:

I realize I should've been more specific. I added those regex lines for a couple of domains because i wanted to redirect from http(s)://www.domain.com to https://domain.com. But some other domains have a dns-provider which has a built-in www redirect option. So it's convenient to use labels.

My main domain has a comodo wildcard cert but i have other domains that I want to use with LE. But, I see now that I have the le cert resolver in my traefik docker labels, that's wrong.

Hm, what's the redirect@docker middleware?

My main traefik docker-compose now looks like this:

version: '3'
services:
  traefik:
    image: traefik:v2.1.4
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log/traefik:/var/log
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
      - ./data/rules:/rules:ro
      - ./data/certs/:/certs
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`redacted.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=user:pwdhash"
        #      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
        #      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`redacted.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
        #      - "traefik.http.routers.traefik-secure.tls.certresolver=le"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

After a restart I'm watching the logs closely, everything seems to work.
Though doing a curl -i http://maintraefikdomain.com time-outs with a 502 Bad Gateway.

EDIT: Still got multiple "Error getting challenge for token..." I figured as much, since I got a 502.

I'm wondering if this could be related to ipv6 issue? I had not disabled ipv6 on this box but just did that now. Crossing fingers that this is indeed the issue here.

EDIT: That was not it :confused:

Also it's a bit confusing but I think I recall reading somewhere that a https redirect would not affect acme certificate resolver because it uses some special underlying route or something. Maybe someone can clarify?

Still at a loss here. Been googling like crazy. Hoping some wizard would come by and point out what is going on.

I came across one issue post that is quite similar to mine, they were also using another wildcard certificate for the main domain.

bilde

I'm still seeing this :confused: Anybody ever seen this before?

I just got this today when I restarted my Traefik 1.7 instance.