How to configure Traefik 2 with TLS - Traefik 2 & TLS 101

HTTPS (& TCP over TLS) for everyone!

There are hundreds of reasons why I love being a developer (besides memories of sleepless nights trying to fix a video game that nobody except myself would ever play).

Being a developer gives you superpowers — you can solve any kind of problems. Yes, especially if they don’t involve real-life practical situations.

But these superpowers are sometimes hindered by tedious configuration work that expects you to master yet another arcane language assembled with heaps of words you’ve never seen before. Such a barrier can be encountered when dealing with HTTPS and its certificates.

Luckily for us, Traefik tends to lower this kind of hurdle and makes sure that there are easy ways of securely connecting your developments to the outside world.

The Goal for Today

The challenge we’ll accept is the following — You have an HTTP service exposed through Traefik, and you want Traefik to deal with the HTTPS burden (TLS termination), leaving your pristine service unspoiled by mundane technical details.

We’ll assume you have a basic understanding of Traefik on Docker and that you’re familiar with its configuration (if not, it’s time to read Traefik 2 & Docker 101).

During this article, we’ll use my pet demo docker-compose file: it enables the docker provider and launches a my-app application that allows us to test any request.

version: "3"
services:
  traefik:
    image: "traefik:v2.0"
    command:
      - --entrypoints.web.address=:80
      - --providers.docker=true
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

my-app:
image: containous/whoami:v1.3.0

Getting Things Ready

First things first, let’s make sure our setup can handle HTTPS traffic on the default port (:443), and that Traefik listens to this port thanks to an entrypoint we’ll name web-secure.

version: "3"
services:
  traefik:
    image: "traefik:v2.0"
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.web-secure.address=:443 #Declares the web-secure entrypoint in Traefik
      - --providers.docker=true
    ports:
      - "80:80"
      - "443:443" #Docker sends requests on port 443 to Traefik on port 443
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

my-app:
image: containous/whoami:v1.3.0

To avoid confusion, let’s state the obvious — We haven’t yet configured anything but enabled requests on 443 to be handled by Traffic. So, no certificate management yet!

General Concepts

Ultimately, in Traefik, you configure HTTPS on the router level. While defining routes, you decide whether they are HTTP routes or HTTPS routes (by default, they are HTTP routes).

First, let’s expose our my-app service on HTTP so that it handles requests on domain example.com.

version: "3"
services:
  # ...
  my-app:
    image: containous/whoami:v1.3.0
    labels:
      - "traefik.http.routers.my-app.rule=Host(`example.com`)"

And now, see what it takes to make this route HTTPS only!

version: "3"
services:
  # ...
  my-app:
    image: containous/whoami:v1.3.0
    labels:
      - "traefik.http.routers.my-app.rule=Host(`example.com`)"
      - "traefik.http.routers.my-app.tls=true"

There, by adding the tls option to the route, we’ve made it HTTPS.

The only unanswered question left is, “Where does Traefik get its certificates from?” And the answer is, “Either from a collection of certificates you own and have configured or from a fully automatic mechanism that gets them for you.”

Let’s see these solutions in action!

Option 1 — Certificates You Own

The least magical of the two options involves creating a configuration file.

Say you already own a certificate for a domain (or a collection of certificates for different domains) and that you are then the proud holder of files to claim your ownership of the said domain.

To have Traefik make a claim on your behalf, you’ll have to give it access to the certificate files. Let’s do this.

Add a Configuration File for Certificates.

version: "3"
services:
  traefik:
    image: "traefik:v2.0"
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.web-secure.address=:443
      - --providers.docker=true
      - --providers.file.directory=/configuration/
      - --providers.file.watch=true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/home/username/traefik/configuration/:/configuration/"

Traefik runs with many providers beyond Docker (i.e., Kubernetes, Rancher, Marathon), and here we chose to add plain old configuration files (--providers.file) in the configuration/ directory (and we’ll automatically reload changes with --providers.file.watch=true). We’ll use a configuration file to declare our certificates.

Add the Certificates to the Configuration File

# in files/certificates.toml
[[tls.certificates]] #first certificate
   certFile = “/path/to/example-com.cert” 
   keyFile = “/path/to/example-com.key”
[[tls.certificates]] #second certificate
   certFile = “/path/to/other.cert” 
   keyFile = “/path/to/other.key”

and so on

Now that we have our TOML configuration file available (thanks to the enabled file provider), we can fill in certificates in the [[tls.certificates]]section.

Enjoy!

This is all there is to do. When dealing with an HTTPS route, Traefik goes through your default certificate store to find a matching certificate.

Specifying a Default Certificate?

If no valid certificate is found, Traefik serves a default auto-signed certificate. But if needed, you can customize the default certificate like so:

[tls.stores]
  [tls.stores.default]
   [tls.stores.default.defaultCertificate] 
     certFile = “path/to/cert.crt” 
     keyFile = “path/to/cert.key”

Additional Thoughts

Even though the configuration is straightforward, it is your responsibility, as the administrator, to configure / renew your certificates when they expire. If you don’t like such constraints, keep reading!

Option 2 — Dynamic / Automatic Certificates

Having to manage (buy/install/renew) your certificates is a process you might not enjoy (I don’t). If so, you’ll be interested in the automatic certificate generation embedded in Traefik (thanks to Let’s Encrypt).

Long story short, you can start Traefik with no other configuration than your Let’s Encrypt account, and Traefik automatically negotiates (get/renew/configure) certificates for you — No extra step.

Certificate Resolvers.

We saw that you can configure a router to use TLS (--traefik.http.routers.router-name.tls=true). As a consequence, we saw that Traefik would go through your certificate list to find a suitable match for the domain at hand (and if not would use a default certificate).

For automatic certificate generation, you can add a certificate resolver to your TLS options. A certificate resolver is responsible for retrieving certificates.

Here, let’s define a certificate resolver that works with your Let’s Encrypt account!

services:
  traefik:
    image: "traefik:v2.0"
    command:
      - --entrypoints.websecure.address=:443
      # ...
      - --certificatesresolvers.le.acme.email=my@email.com
      - --certificatesresolvers.le.acme.storage=/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
      # ...

As you can read, we defined a certificate resolver named le of type acme. Then, we provided an email (your Let’s Encrypt account), the storage file (for certificates it retrieves), and the challenge for certificate negotiation(here tlschallenge, just because it’s the most concise configuration option for the sake of the example).

From now on, Traefik is fully equipped to generate certificates for you!

Using the Certificate Resolver.

If you remember correctly (I’m sure you do!), we enabled TLS on our router like so:

version: "3"
services:
  # ...
  my-app:
    image: containous/whoami:v1.3.0
    labels:
      - "traefik.http.routers.my-app.rule=Host(`example.com`)"
      - "traefik.http.routers.my-app.tls=true"

Now, to enable our certificate resolver and have it automatically generate certificates (when needed), we’ll add it to the TLS configuration, like so:

version: "3"
services:
  # ...
  my-app:
    image: containous/whoami:v1.3.0
    labels:
      - "traefik.http.routers.my-app.rule=Host(`example.com`)"
      - "traefik.http.routers.my-app.tls=true"
      - "traefik.http.routers.my-app.tls.certresolver=le"

Now, if your certificate store doesn’t yet have a valid certificate for example.com, the le certificate resolver will transparently negotiate one for you — it’s that simple.

Multiple Certificate Resolvers?

With certificate resolvers, you can configure different challenges.

Below is an example that shows how to configure two CertResolvers that leverage Let’s Encrypt, one using the dnsChallenge, the other using the tlsChallenge.

[certificatesResolvers.resolver-digital-ocean.acme]
  # ... 
  [certificatesResolvers.resolver-digital-ocean.acme.dnsChallenge]
    provider = "digitalocean"
    delayBeforeCheck = 0
[certificatesResolvers.tls-challenge-resolver.acme]
  # ...
  [certificatesResolvers.tls-challenge-resolver.acme.tlsChallenge]

Later on, you’ll be able to use one or the other on your routers.

# in routers.toml
[http.routers]
  [http.routers.https-route]
    rule = "Host(`my.domain`)"
    [http.routers.https-route.tls]
      certResolver = "resolver-digital-ocean"
[http.routers.https-route-2]
    rule = "Host(`other.domain`)"
    [http.routers.https-route-2.tls]
      certResolver = "tls-challenge-resolver"

In the above example (that uses the file provider), we’ve asked Traefik to generate certificates for my.domain using the dnsChallenge (with digital ocean) and to generate certificates for other.domain using the TLSChallenge.

And you’ve guessed it already — Traefik supports DNS challenge for different DNS providers, at the same time!

Wildcard and Let’s Encrypt?

Instead of generating a certificate for each subdomain, you can choose to generate wildcard certificates!

[http.routers]
  [http.routers.router-example]
    rule = "Host(`something.my.domain`)"
    [http.routers.router-example.tls]
      certResolver = "my-resolver"
      [[http.routers.router-example.tls.domains]]
        main = "my.domain"
        sans = "*.my.domain"

In the above example, we’ve configured Traefik to generate a wildcard certificate for *.my.domain.

If we had omitted the .tls.domains section, Traefik would have used the host (here something.my.domain) defined in the Host rule to generate a certificate.

What About TCP & TLS?

If you want to configure TLS with TCP, then good news: nothing changes, you’ll configure the same tls option, but this time on your tcp router.

version: "3"
services:
  # ...
  my-tcp-app:
    image: containous/whoamitcp:v1.0.0
    labels:
      - "traefik.tcp.routers.my-tcp-app.rule=HostSNI(`tcp-example.com`)"
      - "traefik.tcp.routers.my-tcp-app.tls=true"

What About Pass-Through?

Sometimes your services handle TLS by themselves. In such cases, Traefik mustn’t terminate the TLS connection but forward the request “as is” to these services. To configure this passthrough, you’ll need to configure a TCP router (even if your service handles HTTPS).

version: "3"
services:
  # ...
  my-tcp-app:
    image: containous/whoamitcp:v1.0.0
    labels:
      - "traefik.tcp.routers.my-tcp-app.rule=HostSNI(`tcp-example.com`)"
      - "traefik.tcp.routers.my-tcp-app.tls.passthrough=true"

Questions? Where to Go Next?

Hopefully, this article sheds light on how to configure Traefik 2 with TLS.

If there are missing use cases or still unanswered questions, let me know in the comments or on the community forum!

In the meantime — Happy Traefik!


This is a companion discussion topic for the original entry at https://containo.us/blog/traefik-2-tls-101-23b4fbee81f1/
1 Like

Nice tutorial but I still have troubles with my own static glob certificates. I have condensed your tutorial in the following config file:

entryPoints:
  https:
    address: ":443"
tls:
  certificates:
    -
      certFile: /certs/fullchain.pem
      keyFile:  /certs/privkey.pem    
  stores:
    default:
      defaultCertificate:
        certFile: /certs/fullchain.pem
        keyFile:  /certs/privkey.pem

and docker-compose label section:

labels:
  - traefik.http.routers.whoami1.rule=Host("whoami1.MYDOMAIN")
  - traefik.http.routers.whoami1.tls=true
  - traefik.docker.network=traefik
  - traefik.port=80

The certificates are valid * certificates for MYDOMAIN but I still get the self-signed certificate instead of my own. How do I force traefik to use my certificate ?

I figured out myself. In fact the tls certificate part must go into a separate file and provided by a file provider. This is extremely confusing but it is for sure me who haven't understood the logic.

1 Like

Awesome Tutorial!!!
But when I use my own TLS certificate, I get the error:

time="2020-05-12T09:03:31Z" level=debug msg="http: TLS handshake error from 192.168.1.1:51423: remote error: tls: unknown ce
time="2020-05-12T09:03:53Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"G
time="2020-05-12T09:03:53Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://1
time="2020-05-12T09:03:53Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\"
time="2020-05-12T09:04:42Z" level=debug msg="http2: received GOAWAY [FrameHeader GOAWAY len=8], starting graceful shutdown"

I have the certificates put in the dynamic file:

tls:
  stores:                                                                                                                        default:                                                                                                                       defaultCertificate:                                                                                                            certFile: /etc/traefik/certs/whoami.cert                                                                                     keyFile: /etc/traefik/certs/whoami.key                                                                                 certificates:
    - certFile: /etc/traefik/certs/whoami.cert
      keyFile: /etc/traefik/certs/whoami.key
      stores:                                                                                                                        - default 

But the problem comes at the time of accessing the web in the browser, after accepting the risk that it is a self signed certificate, the page is not displayed. It's as if it doesn't load.
I'm always waiting on this page

1 Like

@vi8a this is totally normal, because you are using a selfsigned certificate.
Your browser does not trust this certificate or the one who issued it (CA = Certificate Authority).

You have 2 ways to go:

  1. you choose "Aceptar el riesgo y continuar" (because you know the one who made the cert)
  2. or you use a certificate from a trusted issuer (like, LetsEncrypt or other official CAs)

There would be a third one but we don't go into that now ;o)

Please read this https://en.wikipedia.org/wiki/Public_key_infrastructure and you will gain information about certificates and public key infrastructure.

1 Like

May you cover the use case of traefik using step-ca (Docker Hub) like CA acme provisioner?

Thanks in advance.

dear god, can you please elaborate more on what exactly you did here?

My traefik config has been broken for 6 months because of this exact problem.

I am not god (who does not exist) but I will reply anyway.
In my case, not sure it is the same for you, the error was that I had the certificate parts in the main traefik.yml config file while it should go in the dynamic part. If you use files it reads something like the following:

# /config/aaa.yml
tls:
  certificates:
    -
      certFile: /certs/fullchain.pem
      keyFile:  /certs/privkey.pem    

and in the main config file you have something like:

# traefik.yml
providers:
  file:
    directory: /config
    watch: true

and nothing about certs in traefik.yml

hope this helps

thanks @multiscan for your godly answer!
And for figuring out the issue and explaining it in the first place.

Just to clarify why that happens and the real motivation behind it: Traefik has a very strong separation of what is considered static configuration, things that won't change during its runtime no matter what and then the dynamic configuration which is everything that composes your routing configuration and is subject to constant changes and updates that Traefik should be able to recognise and adapt without requiring a restart of the process.

All dynamic configuration in Traefik is expected to come from the provider itself, and when there is no good alternative to declare stuff on the provider in use, you can always rely in the good old File provider to load those values, the best example being certificates!

To exemplify that we can compare two different providers, in Kubernetes CRD you can specify a secret name that is holding the certificate for your router on the IngressRoute configuration directly. This is not possible in the Docker provider, but the responsibility to load certificates remains on the dynamic config side, so in the providers, which means another provider will be required to declare it.

@douglasdtm sanity check

version: "3.9"
services:
  traefik:
    image: traefik:v2.9
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik:/etc/traefik
    restart: unless-stopped

/etc/traefik/conf/traefik.yml

entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: web-secure
          scheme: https
    proxyProtocol:
      trustedIPs:
        - "10.8.0.0/20"
  web-secure:
    address: :443
    forwardedHeaders:
      trustedIPs:
        - "192.168.88.0/30"
    http:
      tls:
        domains:
          - main: "domain.com"
            sans:
              - "*.domain.com"
      middlewares:
    proxyProtocol:
      trustedIPs:
        - "10.8.0.0/20"
  traefik:
    address: "10.6.6.99:8080"
    
ping: {}

providers:
  docker:
    endpoint: unix:///var/run/docker.sock
    watch: true 
    exposedByDefault: false
  file:
    directory: /etc/traefik/conf
    watch: true

api:
  dashboard: true 
  insecure: true 

serversTransport:
  insecureSkipVerify: false 

metrics:
  prometheus:
    entryPoint: traefik 

pilot:
  dashboard: false 
  
log:
  level: DEBUG
  filePath: "/var/log/traefik/log-file.log"
  format: json
accessLog:
  filePath: "/var/log/traefik/access.log"
  filters:
    statusCodes:
      - "400-600"

/etc/traefik/conf/jellyfin.yml

http:
  routers:
    router-jellyfin:
      rule: Host(`jel.domain.com`)
      service: service-jellyfin
  services:
    service-jellyfin:
      loadBalancer:
        servers:
          - url: "http://10.10.10.10:8096"
tls:
  certificates:
    - 
      certFile: /etc/traefik/conf/ssl/t.cert
      keyFile: /etc/traefik/conf/ssl/t.key

/etc/traefik/conf/main.yml

http:
  middlewares:
    secureHeaders:
      headers:
        sslRedirect: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000 
    compress:
      compress: {}


If I docker compose up this, it completely ignores my certificates, and loads the default docker self signed cert.

Is the certificate self generated / self signed?

Can you show it with:

 openssl x509 -in ./traefik/conf/ssl/t.cert -text

It would also help to confirm the problem if you can share logs in debug mode while hitting that router rule.

If you run in Docker network host mode, you only need proxyProtocol when you have another node/container like a load balancer in front of Traefik.

Alternatively to full host mode you can only expose the required ports in host mode.

docker-compose.yml:

version: '3.9'

services:
  traefik:
    image: traefik:v2.9
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    …