Traefik access logs get real source IP

Traefik Traefik v1
1

Hi guys, I have the following setup:
HAProxy (Layer 4) --> Traefik Cluster in kubernetes deployed using the daemonset.

And I just did enable the accessLog to get the source IPs of each request, so I went to HAProxy configuration and enabled the option forwardfor and configured traefik logs like this:

    [accessLog]
      filePath = "/logs/access.log"
      format = "json"
      bufferingSize = 100
      [accessLog.fields]
        defaultMode = "keep"
        [accessLog.fields.headers]
          defaultMode = "keep"
          [accessLog.fields.headers.names]
            "X-Forwarded-For" = "keep"

But I can't see the client source address in the logs, this is what I get:

{
	"BackendAddr": "10.244.5.23:9000",
	"BackendName": "https_main.domain.xyz",
	"BackendURL": {
		"Scheme": "http",
		"Opaque": "",
		"User": null,
		"Host": "10.244.5.23:9000",
		"Path": "",
		"RawPath": "",
		"ForceQuery": false,
		"RawQuery": "",
		"Fragment": ""
	},
	"ClientAddr": "192.168.128.4:34326",
	"ClientHost": "192.168.128.4",
	"ClientPort": "34326",
	"ClientUsername": "-",
	"DownstreamContentSize": 2450,
	"DownstreamStatus": 200,
	"DownstreamStatusLine": "200 OK",
	"Duration": 5536065,
	"FrontendName": "https_main.domain.xyz",
	"OriginContentSize": 2450,
	"OriginDuration": 5153418,
	"OriginStatus": 200,
	"OriginStatusLine": "200 OK",
	"Overhead": 382647,
	"RequestAddr": "main.domain.xyz",
	"RequestContentSize": 0,
	"RequestCount": 39,
	"RequestHost": "main.domain.xyz",
	"RequestLine": "GET / HTTP/1.0",
	"RequestMethod": "GET",
	"RequestPath": "/",
	"RequestPort": "-",
	"RequestProtocol": "HTTP/1.0",
	"RetryAttempts": 0,
	"StartLocal": "2019-08-14T10:10:29.989624291Z",
	"StartUTC": "2019-08-14T10:10:29.989624291Z",
	"downstream_Accept-Ranges": "bytes",
	"downstream_Cache-Control": "public, max-age=0",
	"downstream_Content-Type": "text/html; charset=UTF-8",
	"downstream_Date": "Wed, 14 Aug 2019 10:10:29 GMT",
	"downstream_Etag": "W/\"992-16c6b837cb8\"",
	"downstream_Last-Modified": "Wed, 07 Aug 2019 09:57:55 GMT",
	"downstream_Vary": "Accept-Encoding",
	"downstream_X-Powered-By": "Express",
	"level": "info",
	"msg": "",
	"origin_Accept-Ranges": "bytes",
	"origin_Cache-Control": "public, max-age=0",
	"origin_Content-Type": "text/html; charset=UTF-8",
	"origin_Date": "Wed, 14 Aug 2019 10:10:29 GMT",
	"origin_Etag": "W/\"992-16c6b837cb8\"",
	"origin_Last-Modified": "Wed, 07 Aug 2019 09:57:55 GMT",
	"origin_Vary": "Accept-Encoding",
	"origin_X-Powered-By": "Express",
	"request_Accept": "*/*",
	"request_Connection": "Keep-Alive",
	"request_User-Agent": "ApacheBench/2.3",
	"time": "2019-08-14T10:10:29Z"
}

So the ClientAddr is the HAProxy's IP which is probably normal, but I expected to get also the X-Forwarded-For header.

Am I missing anything?

Thanks.

2

After a chat with @dduportal in Slack, I got the following conclusions (please correct me if I am wrong).

First of all I can not see the x-forwarded-for header because I am using HAProxy layer 4 so it is not possible to send headers (https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-option%20forwardfor).

So basically the alternatives are the following:

I am still unsure in which alternative I will take but whenever I take one I'll update this post with the results.

Thanks for the help.

3

HAProxy and Traefik support the Proxy Protocol, which allows you to preserve the client IP through network hops that would otherwise lose it (such as when using HAProxy as a l4 proxy in front of Traefik). This is what you want to configure:

HAProxy
Traefik

1 Like
4

Thank you so much @ReillyProcentive!!

I spent a lot of hours with this issue and was about to give up then I tried the Proxy protocol and... boom it just works!

Thanks.

5

Updated link for Traefik: https://docs.traefik.io/routing/entrypoints/#proxyprotocol

6

Sorry to bring up a dead thread, but we are using a AWS network load balancer in-front of Traefik and have the following configurations (using Helm chart). Is it possible to add the X-Real-IP or forwarded ip address into our access logs? Currently we are only getting the AWS network load balancers private ip.

proxyProtocol:
    enabled: false
    # trustedIPs is required when enabled
    trustedIPs: []
    # - 10.0.0.0/8
forwardedHeaders:
    enabled: true
    # trustedIPs is required when enabled
    trustedIPs:
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12

Then for access logs:

    enabled: true
    ## Path to the access logs file. If not provided, Traefik defaults it to stdout.
    # filePath: ""
    format: common  # choices are: common, json
    ## for JSON logging, finer-grained control over what is logged. Fields can be
    ## retained or dropped, and request headers can be retained, dropped or redacted
    fields:
      # choices are keep, drop
      defaultMode: keep
      names: {}
        # ClientUsername: drop
      headers:
        # choices are keep, drop, redact
        defaultMode: keep
        names: {}
          # Authorization: redact
1 Like
7

@nodesocket Did you find the solution for this? i'm also having similar issue

8

@ns-ajith yes I did and happy to help. Buried way down deep in the AWS documentation is a guide on enabling proxy protocol on classic network load balancers. Essentially it involves the following:

  1. Create a load balancer policy via the AWS CLI (replace <load-balancer-name>)
aws elb create-load-balancer-policy --load-balancer-name <load-balancer-name> --policy-name ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
  1. Enable the load balancer policy on instance ports. VERY IMPORTANT: Look in the load balancer listeners tab in the AWS web console and find the two http/https instance ports (they are dynamic per provision), those are the ports needed below. Replace <load-balancer-name> and <instance-port-1> and <instance-port-2>.
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name <load-balancer-name> --instance-port <instance-port-1> --policy-names ProxyProtocol-policy

aws elb set-load-balancer-policies-for-backend-server --load-balancer-name <load-balancer-name> --instance-port <instance-port-2> --policy-names ProxyProtocol-policy
  1. Upgrade Traefik deployment.
# or similar
helm upgrade traefik --values values.yaml ./

IMPORTANT: We are still running Traefik v1 and also required the following Traefik configuration to be set. I'm not sure what the corresponding v2 configuration would be. If you could report back after you get it working much appreciated. We are looking to upgrade to v2 shortly.

proxyProtocol:
    enabled: true
    trustedIPs:
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
  forwardedHeaders:
    enabled: true
    trustedIPs:
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
9

Thanks, @nodesocket for the quick reply. I can able to achieve this with V2 with the same configuration, But I'm not sure how to achieve this with ALB. With CLB policy it works like charm.

Also, i'm looking to implement this with ECS but I cannot use CLB with ecs-fargte. :Traefik ECS provider is not forwarding client IP with AWS ALB

10

@nodesocket just FYI I was able to achieve this with the V2 and ALB. you don't have to make any patches to the load balancer ii just works with the trustedIPs.