HTTP -> HTTPS for a specific URL, only?

[TrafikJam]

@ldez The way you separated things is starting to make more sense. I've copied your example, but I'm still getting a 404 unfortunately:

"POST /path/123 HTTP/1.1" 404 165 "-" "-" 644 "services_redirect@docker" 

That's the result for posting to "http://www.URL.com/path/123" - while if I post to "https://services.URL.com/456" it does work:

"POST /456 HTTP/1.1" 200 2 "-" "-" 645 "services@docker"

I still don't see how the HTTPS service will be able to send back as HTTP.

[ldez]

• Traefik dashboard: https://traefik.localhost
• https://services.example.localhost -> OK
• http://www.example.localhost/path/123 -> https://services.example.localhost/456 -> ok

Note to redirect you have to use the redirectregex middleware.

https://docs.traefik.io/v2.1/middlewares/redirectregex/

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

  whoami:
    image: containous/whoami:v1.4.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
     traefik.enable: true

      # router for services.example.localhost on the entrypoint websecure
      # https://services.example.localhost
     traefik.http.routers.app.rule: Host(`services.example.localhost`)
     traefik.http.routers.app.entrypoints: websecure
     traefik.http.routers.app.middlewares: comp
     traefik.http.routers.app.tls: true

     traefik.http.services.app_svc.loadbalancer.server.port: 2001

      # router for http://www.example.localhost on the entrypoint web, always redirect.
      # http://www.example.localhost/path/123 -> https://services.example.localhost/456
     traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path/123`)
     traefik.http.routers.app_redirect.entrypoints: web
     traefik.http.routers.app_redirect.middlewares: redirect

      # Middlewares defintions

     traefik.http.middlewares.comp.compress: true
      
     traefik.http.middlewares.redirect.redirectregex.regex: ^http://www\.example\.localhost/path/123
     traefik.http.middlewares.redirect.redirectregex.replacement: https://services.example.localhost/456

This example uses self-signed certificates only to be simple, it also works with Let's Encrypt.

Also I used different syntax for labels (without -, and with : instead of =), but this don't change the behavior.

$ curl -k https://services.example.localhost

Hostname: 865d693c538a
IP: 127.0.0.1
IP: 172.23.0.3
RemoteAddr: 172.23.0.2:36172
GET / HTTP/1.1
Host: services.example.localhost
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: services.example.localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ee94c8d56a29
X-Real-Ip: 172.23.0.1

$ curl -I http://www.example.localhost/path/123

HTTP/1.1 307 Temporary Redirect
Location: https://services.example.localhost/456
Date: Wed, 11 Mar 2020 09:39:00 GMT
Content-Length: 18
Content-Type: text/plain; charset=utf-8

$ curl -Lk http://www.example.localhost/path/123

Hostname: 865d693c538a
IP: 127.0.0.1
IP: 172.23.0.3
RemoteAddr: 172.23.0.2:36172
GET /456 HTTP/1.1
Host: services.example.localhost
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: services.example.localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ee94c8d56a29
X-Real-Ip: 172.23.0.1

[TrafikJam]

I asked a few times and I don't think there's a straight answer. I'm using redirectregex.replacement, like you are... But if this is a redirect as in 301, then it will not work for me. What works is what nginx calls a "rewrite." A 307 could also potentially work for me. The API going to the replacement URL does not allow redirection.

This is what is coming through:
https://developer.paypal.com/docs/ipn/
So I cannot test with Curl, but I do test with the IPN simulator. PayPal does not allow anyone to change their old IPN URLs.

[ldez]

You can add traefik.http.middlewares.redirect.redirectregex.permanent: true to get a permanent redirect.

You will have a 308 on GET and a 301 with the other HTTP methods (POST, HEAD, ...)

https://docs.traefik.io/v2.1/middlewares/redirectscheme/#permanent

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

  whoami:
    image: containous/whoami:v1.4.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
     traefik.enable: true

     # router for services.example.localhost on the entrypoint websecure
     # https://services.example.localhost
     traefik.http.routers.app.rule: Host(`services.example.localhost`)
     traefik.http.routers.app.entrypoints: websecure
     traefik.http.routers.app.middlewares: comp
     traefik.http.routers.app.tls: true

     traefik.http.services.app_svc.loadbalancer.server.port: 2001

     # router for http://www.example.localhost on the entrypoint web, always redirect.
     # http://www.example.localhost/path/123 -> https://services.example.localhost/456
     traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path/123`)
     traefik.http.routers.app_redirect.entrypoints: web
     traefik.http.routers.app_redirect.middlewares: redirect

     # Middlewares defintions

     traefik.http.middlewares.comp.compress: true

     traefik.http.middlewares.redirect.redirectregex.regex: ^http://www\.example\.localhost/path/123
     traefik.http.middlewares.redirect.redirectregex.replacement: https://services.example.localhost/456
     traefik.http.middlewares.redirect.redirectregex.permanent: true

$ curl -I http://www.example.localhost/path/123

HTTP/1.1 308 Permanent Redirect
Location: https://services.example.localhost/456
Date: Wed, 11 Mar 2020 09:44:58 GMT
Content-Length: 18
Content-Type: text/plain; charset=utf-8

[TrafikJam]

Yes, so as I said, a 301 will break it. Do you understand my question about terminology? Is a rewrite the same in Traefik-speak as it is in Nginx-speak? It either has to be a 307 or a rewrite as far as I know... This solution seems so much simpler using other methods around the web:

How to change Paypal IPN URL when migrating Civicrm onto a new domain?

[ldez]

A redirect in Traefik is a kind of rewrite in Nginx.

The 2 products have different approaches so there is no one by one conversion.

You must try to understand how Traefik works instead of trying to do a one by one conversion.

[TrafikJam]

I think Traefik needs to convert their special language into the lingua franca... Perhaps it has and I'm not sufficiently educated. I've obviously put A LOT of effort into this... Can you please see my above link and let me know if it is possible in Traefik? If so, then what would it be called?

[ldez]

I don't know Paypal IPN, from the link you sent, I think you're just looking for a temporary redirect on POST.

The redirect middleware will send a 307 on POST and a 301 on GET.

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

  whoami:
    image: containous/whoami:v1.4.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
      traefik.enable: true

      # router for services.example.localhost on the entrypoint websecure
      # https://services.example.localhost
      traefik.http.routers.app.rule: Host(`services.example.localhost`)
      traefik.http.routers.app.entrypoints: websecure
      traefik.http.routers.app.middlewares: comp
      traefik.http.routers.app.tls: true

      traefik.http.services.app_svc.loadbalancer.server.port: 2001

      # router for http://www.example.localhost on the entrypoint web, always redirect.
      traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path`)
      traefik.http.routers.app_redirect.entrypoints: web
      traefik.http.routers.app_redirect.middlewares: redirect

      # Middlewares defintions

      traefik.http.middlewares.comp.compress: true

      traefik.http.middlewares.redirect.redirectregex.regex: ^http://[^/]+/path/(.+)$$
      traefik.http.middlewares.redirect.redirectregex.replacement: https://services.example.localhost/$${1}

$ curl -v -X POST -d "test"  http://www.example.localhost/path/123

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1:80...
* TCP_NODELAY set
* Connected to www.example.localhost (::1) port 80 (#0)
> POST /path/123 HTTP/1.1
> Host: www.example.localhost
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 4
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 4 out of 4 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 307 Temporary Redirect
< Location: https://services.example.localhost/123
< Date: Wed, 11 Mar 2020 10:37:53 GMT
< Content-Length: 18
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host www.example.localhost left intact

$ curl -L -k -X POST -d "test"  http://www.example.localhost/path/123

Hostname: 38159b0aa900
IP: 127.0.0.1
IP: 172.23.0.3
RemoteAddr: 172.23.0.2:40620
POST /123 HTTP/1.1
Host: services.example.localhost
User-Agent: curl/7.68.0
Content-Length: 4
Accept: */*
Accept-Encoding: gzip
Content-Type: application/x-www-form-urlencoded
X-Forwarded-For: 172.23.0.1
X-Forwarded-Host: services.example.localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ee94c8d56a29
X-Real-Ip: 172.23.0.1

[TrafikJam]

@ldez
OK, now I see a 307 or 308 depending upon if I use "permanent" or not. But it still does not work... If I use replacepathregex middleware, then I get a 404.

So, I'm thinking with redirectregex: how does it get the reply? Isn't it replying to the HTTP remote with SSL? Because on the way in it gets translated into an HTTP URL by redirectregex, but then on the way out the server is running on HTTPS - so doesn't it reply using SSL? Does that make sense? The remote is HTTP only and my server is replying with HTTPS because of the above config.

So: how would I make redirectregex.replacement reply without SSL?

[ldez]

you have to use the redirectregex: https://docs.traefik.io/v2.1/middlewares/redirectregex/

The replacepathregex is only to handle path: https://docs.traefik.io/v2.1/middlewares/replacepathregex/

[TrafikJam]

OK, I am... And the message seems to get in, but not out. My hypothesis is because I need to send back as HTTP, not HTTPS. Hopefully I described the thought process in my last reply.

[ldez]

Your goals change after each answer...

Please describe in detail your goals (be clear and explicit) and I will try to answer.

So: how would I make redirectregex.replacement reply without SSL?

A redirect is just a redirect, so if you want to handle no TLS connection you have to handle a router without TLS.

[TrafikJam]

I described it several times including here:

HTTP -> HTTPS for a specific URL, only?

As I said, "traefik needs to remove SSL somehow" (on the reply).

Would you mind helping me using your above example?
The only criteria I've left out is that I'd like the global SSL redirect (your standard one is what I used) to work again.

[ldez]

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

  whoami:
    image: containous/whoami:v1.4.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
      traefik.enable: true

      # https://services.example.localhost
      traefik.http.routers.app_secure.rule: Host(`services.example.localhost`)
      traefik.http.routers.app_secure.entrypoints: websecure
      traefik.http.routers.app_secure.middlewares: comp
      traefik.http.routers.app_secure.tls: true

      # http://services.example.localhost
      traefik.http.routers.app.rule: Host(`services.example.localhost`)
      traefik.http.routers.app.entrypoints: web
      traefik.http.routers.app.middlewares: comp
      traefik.http.routers.app.tls: true

      traefik.http.services.app_svc.loadbalancer.server.port: 2001

      # router for http://www.example.localhost on the entrypoint web, always redirect.
      traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path`)
      traefik.http.routers.app_redirect.entrypoints: web
      traefik.http.routers.app_redirect.middlewares: redirect

      # Middlewares defintions

      traefik.http.middlewares.comp.compress: true

      traefik.http.middlewares.redirect.redirectregex.regex: ^http://[^/]+/path/(.+)$$
      traefik.http.middlewares.redirect.redirectregex.replacement: http://services.example.localhost/$${1}

[ldez]

With the global redirect:

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # global HTTP to HTTPS redirection
      traefik.http.routers.http_catchall.rule: hostregexp(`{host:.+}`)
      traefik.http.routers.http_catchall.entrypoints: web
      traefik.http.routers.http_catchall.middlewares: redirect_https
      traefik.http.routers.http_catchall.priority: 1

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

      # Middlewares defintions
      traefik.http.middlewares.comp.compress: true
      traefik.http.middlewares.redirect_https.redirectscheme.scheme: https

  whoami:
    image: containous/whoami:v1.4.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
      traefik.enable: true

      # https://services.example.localhost
      traefik.http.routers.app_secure.rule: Host(`services.example.localhost`)
      traefik.http.routers.app_secure.entrypoints: websecure
      traefik.http.routers.app_secure.middlewares: comp
      traefik.http.routers.app_secure.tls: true

      # http://services.example.localhost
      traefik.http.routers.app.rule: Host(`services.example.localhost`)
      traefik.http.routers.app.entrypoints: web
      traefik.http.routers.app.middlewares: comp

      traefik.http.services.app_svc.loadbalancer.server.port: 2001

      # router for http://www.example.localhost on the entrypoint web, always redirect.
      traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path`)
      traefik.http.routers.app_redirect.entrypoints: web
      traefik.http.routers.app_redirect.middlewares: redirect

      # Middlewares defintions

      traefik.http.middlewares.redirect.redirectregex.regex: ^http://[^/]+/path/(.+)$$
      traefik.http.middlewares.redirect.redirectregex.replacement: http://services.example.localhost/$${1}

  whoami2:
    image: containous/whoami:v1.4.0
    labels:
      traefik.enable: true

      traefik.http.routers.app2.rule: Host(`app2.example.localhost`)
      traefik.http.routers.app2.entrypoints: websecure
      traefik.http.routers.app2.middlewares: comp
      traefik.http.routers.app2.tls: true

[TrafikJam]

Mine has this difference (https, not http, for the replacement):

-"traefik.http.middlewares.redirect.redirectregex.regex=^http://www.URL.com/path/123"
-"traefik.http.middlewares.redirect.redirectregex.replacement=https://services.URL.com/456"

I see... So what in your example causes the reply to be over HTTP instead of HTTPS?
OK, I've implemented your solution and it has not helped. I get a 307 from Traefik, but my internal service is not sending anything back.

[ldez]

A redirect is done by the client (browser, curl, ...), not by Traefik (it's the standard HTTP behavior, see RFC):
When a request comes to an endpoint to redirect, Traefik send a response to the client to inform him to do a request on another location
So the client (browser, curl, ...) calls the new location by it-self.

If have a global redirect (or redirect only for a domain) from HTTP to HTTPS

ex: http://services.example.localhost -> https://services.example.localhost

So it's impossible to call http://services.example.localhost directly: you will be redirected.

To do that you have to create a router with a rule on the entrypoint web that only match your excluded request.

version: '3.7'

services:

  traefik:
    image: traefik:v2.1.6
    command:
      - --log.level=INFO
      - --api
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      traefik.enable: true

      # global HTTP to HTTPS redirection
      traefik.http.routers.http_catchall.rule: hostregexp(`{host:.+}`)
      traefik.http.routers.http_catchall.entrypoints: web
      traefik.http.routers.http_catchall.middlewares: redirect_https
      traefik.http.routers.http_catchall.priority: 1

      # Dashboard
      traefik.http.routers.traefik.rule: Host(`traefik.localhost`)
      traefik.http.routers.traefik.entrypoints: websecure
      traefik.http.routers.traefik.service: api@internal
      traefik.http.routers.traefik.tls: true

      # Middlewares defintions
      traefik.http.middlewares.comp.compress: true
      traefik.http.middlewares.redirect_https.redirectscheme.scheme: https

  whoami:
    image: containous/whoami:v1.5.0
    command:
      # It tells whoami to start listening on 2001 instead of 80
      - --port=2001
    labels:
      traefik.enable: true

      # https://services.example.localhost
      traefik.http.routers.app_secure.rule: Host(`services.example.localhost`)
      traefik.http.routers.app_secure.entrypoints: websecure
      traefik.http.routers.app_secure.middlewares: comp
      traefik.http.routers.app_secure.tls: true

      # http://services.example.localhost/foo
      traefik.http.routers.app.rule: Host(`services.example.localhost`) && PathPrefix(`/foo`)
      traefik.http.routers.app.entrypoints: web
      traefik.http.routers.app.middlewares: strip,comp

      traefik.http.services.app_svc.loadbalancer.server.port: 2001

      # router for http://www.example.localhost on the entrypoint web, always redirect.
      traefik.http.routers.app_redirect.rule: Host(`www.example.localhost`) && PathPrefix(`/path`)
      traefik.http.routers.app_redirect.entrypoints: web
      traefik.http.routers.app_redirect.middlewares: redirect

      # Middlewares defintions

      traefik.http.middlewares.strip.stripprefix.prefixes: /foo
      traefik.http.middlewares.redirect.redirectregex.regex: ^http://[^/]+/path/(.+)$$
      traefik.http.middlewares.redirect.redirectregex.replacement: http://services.example.localhost/foo/$${1}

  whoami2:
    image: containous/whoami:v1.4.0
    labels:
      traefik.enable: true

      traefik.http.routers.app2.rule: Host(`app2.example.localhost`)
      traefik.http.routers.app2.entrypoints: websecure
      traefik.http.routers.app2.middlewares: comp
      traefik.http.routers.app2.tls: true

[TrafikJam]

I've implemented your solution and it didn't help. Same as before: I see a POST with a 307 on the regex URL. But the communication can't happen - and I have the same hypothesis as before: how is your latest example causing SSL to be removed on the reply?

[ldez]

Could you create simple and detailed (with all your cases) graph to explain your goal?

ex: (the graph related to my latest example)

http://services.example.localhost
|_ (redirect)-> https://services.example.localhost
   |_ your app

http://services.example.localhost/foo
|_ your app

http://www.example.localhost/path/123 [POST]
|_ (redirect 307)->  http://services.example.localhost/foo/123 [POST]
  |_ (strip)-> http://services.example.localhost/123 [POST]
     |_ your app

https://www.example.localhost/path/123
|_ 404

[TrafikJam]

@ldez

@ldez

http://www.example.localhost/path/123 [POST] |_ (redirect 307)-> https://services.example.localhost/foo/123 [POST] |_ (strip)-> https://services.example.localhost/123 [POST] |_ your app

I'm unsure of the correct formatting, but after this is "redirected," then it needs to be able to answer. Obviously it will answer using SSL. However, the remote service is expecting HTTP because it was talking to an HTTP service (that was redirected to https://services.example.localhost/123). Does that make sense?

1. Request comes in to http://www.URL.com/path/123
2. Traefik (silently) replaces it with https://services.URL.com/456
3. https://services.URL.com/456 receives the requests
4. https://services.URL.com/456 answers, but the remote doesn't understand because it is answering with SSL, which expecting none.