Issues with installing Traefik Pilot on Kubernetes

Hey, I had some issues when trying to add the pilot token to our Traefik. It mostly crashed or didn't do anything, and upon checking the traefik cli from the pod itself by running:

traefik --experimental.pilot.token=, I got the following error:

INFO[0000] Configuration loaded from flags.
2020/08/27 08:36:42 traefik.go:76: command traefik error: error while building entryPoint http: error preparing server: error opening listener: listen tcp :80: bind: permission denied

We are using the following rbac:

Can anyone help please?

1 Like

Traefik cannot bind directly to port 80/443 because of security limitations on the cluster.

Please see this post for additional configuration details. Traefik v2 unable to bind port 80 in k8s

1 Like

I have those already setup, and I still get the same error. I'm going to link the values file of our deployment:

# Default values for Traefik
image:
  name: traefik
  tag: v2.3.0-rc4
#
# Configure the deployment
#
deployment:
  enabled: true
  # Number of pods of the deployment
  replicas: 1
  # Additional deployment annotations (e.g. for jaeger-operator sidecar injection)
  annotations: {}
  # Additional pod annotations (e.g. for mesh injection or prometheus scraping)
  podAnnotations: {}
# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    enabled: false
    # Additional ingressRoute annotations (e.g. for kubernetes.io/ingress.class)
    annotations:
      kubernetes.io/ingress.class: traefik-private
    # Additional ingressRoute labels (e.g. for filtering IngressRoute by custom labels)
    labels: {}
rollingUpdate:
  maxUnavailable: 1
  maxSurge: 1
#
# Add volumes to the traefik pod.
# This can be used to mount a cert pair or a configmap that holds a config.toml file.
# After the volume has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
# - "--providers.file.filename=/config/dynamic.toml"
volumes: []
# - name: public-cert
#   mountPath: "/certs"
#   type: secret
# - name: configs
#   mountPath: "/config"
#   type: configMap
globalArguments:
  - "--global.checknewversion"
  - "--providers.kubernetescrd.ingressclass=traefik-private"
  - "--providers.kubernetesingress.ingressclass=traefik-private"
#
# Configure Traefik static configuration
# Additional arguments to be passed at Traefik's binary
# All available options available on https://docs.traefik.io/reference/static-configuration/cli/
## Use curly braces to pass values: `helm install --set="additionalArguments={--providers.kubernetesingress,--logs.level=DEBUG}"`
additionalArguments:
  - "--metrics.prometheus=true"
  - "--metrics.prometheus.entryPoint=metrics"
  - "--entrypoints.web.address=:8000"
  - "--entrypoints.websecure.address=:8443"
# Environment variables to be passed to Traefik's binary
env: []
# - name: SOME_VAR
#   value: some-var-value
# - name: SOME_VAR_FROM_CONFIG_MAP
#   valueFrom:
#     configMapRef:
#       name: configmap-name
#       key: config-key
# - name: SOME_SECRET
#   valueFrom:
#     secretKeyRef:
#       name: secret-name
#       key: secret-key
envFrom: []
# - configMapRef:
#     name: config-map-name
# - secretRef:
#     name: secret-name
# Configure ports
ports:
  # The name of this one can't be changed as it is used for the readiness and
  # liveness probes, but you can adjust its config to your liking
  traefik:
    port: 9000
    # Use hostPort if set.
    # hostPort: 9000
    # Defines whether the port is exposed if service.type is LoadBalancer or
    # NodePort.
    #
    # You SHOULD NOT expose the traefik port on production deployments.
    # If you want to access it from outside of your cluster,
    # use `kubectl proxy` or create a secure ingress
    expose: false
    # The exposed port for this service
    exposedPort: 9000
  web:
    port: 8000
    # hostPort: 8000
    expose: true
    exposedPort: 80
    # Use nodeport if set. This is useful if you have configured Traefik in a
    # LoadBalancer
    # nodePort: 32080
  websecure:
    port: 8443
    # hostPort: 8443
    expose: true
    exposedPort: 443
    # nodePort: 32443
  metrics:
    port: 8082
    # hostPort: 8082
    expose: true
    exposedPort: 8082
    # nodePort: 8082
# Options for the main traefik service, where the entrypoints traffic comes
# from.
service:
  enabled: true
  type: LoadBalancer
  # Additional annotations (e.g. for cloud provider specific config)
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-internal: 'true'
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-subnets: '<SUBNETS>'
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
  # Additional entries here will be added to the service spec. Cannot contains
  # type, selector or ports entries.
  spec:
    externalTrafficPolicy: Local
    # loadBalancerIP: "1.2.3.4"
    # clusterIP: "2.3.4.5"
  loadBalancerSourceRanges: []
    # - 192.168.0.1/32
    # - 172.16.0.0/16
  externalIPs: []
    # - 1.2.3.4
## Create HorizontalPodAutoscaler object.
##
autoscaling:
  enabled: false
#   minReplicas: 1
#   maxReplicas: 10
#   metrics:
#   - type: Resource
#     resource:
#       name: cpu
#       targetAverageUtilization: 60
#   - type: Resource
#     resource:
#       name: memory
#       targetAverageUtilization: 60
# Enable persistence using Persistent Volume Claims
# ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
# After the pvc has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
# - "--certificatesresolvers.le.acme.storage=/data/acme.json"
# It will persist TLS certificates.
persistence:
  enabled: false
  accessMode: ReadWriteOnce
  size: 128Mi
  # storageClass: ""
  path: /data
  annotations: {}
  # subPath: "" # only mount a subpath of the Volume into the pod
# If hostNetwork is true, runs traefik in the host network namespace
# To prevent unschedulabel pods due to port collisions, if hostNetwork=true
# and replicas>1, a pod anti-affinity is recommended and will be set if the
# affinity is left as default.
hostNetwork: false
# Additional serviceAccount annotations (e.g. for oidc authentication)
serviceAccountAnnotations: {}
resources:
  requests:
    cpu: "300m"
    memory: "150Mi"
  limits:
    cpu: "300m"
    memory: "150Mi"
affinity: {}
# # This example pod anti-affinity forces the scheduler to put traefik pods
# # on nodes where no other traefik pods are scheduled.
# # It should be used when hostNetwork: true to prevent port conflicts
#   podAntiAffinity:
#     requiredDuringSchedulingIgnoredDuringExecution:
#     - labelSelector:
#         matchExpressions:
#         - key: app
#           operator: In
#           values:
#           - {{ template "traefik.name" . }}
#       topologyKey: failure-domain.beta.kubernetes.io/zone
nodeSelector: {}
tolerations: []
# Pods can have priority.
# Priority indicates the importance of a Pod relative to other Pods.
priorityClassName: ""
# Set the container security context
# To run the container with ports below 1024 this will need to be adjust to run as root
securityContext:
  capabilities:
    drop: [ALL]
  readOnlyRootFilesystem: true
  runAsGroup: 65532
  runAsNonRoot: true
  runAsUser: 65532
podSecurityContext:
  fsGroup: 65532
1 Like

If you get the same error, is it possible you set it up incorrectly? The error indicates that it still tries to bind to port 80.

Look at the kubernetes manifests and check if the values for entrypoints are correct.

I dont think there should be any problem with that, as I can see that it tries to run on the above ports.
(I tried to test pilot on traefik-private)

1 Like

Hey, can you look at the kubernetes manifests and check if the values for entrypoints are correct? Note entrypoint as per notsureifkevin's link, not endpoints.

Sorry, my bad. However, the generated manifest still has the correct entrypoints and ports.

Actually, I just re-read your top post more carefully, and realised that you are getting this error from CLI. I missed that on the first reading, apologies.

So this then makes sense, you are trying to start second instance of traefik inside the pod and it cannot bind.

We run two traefik deployments yes, however, they are in different pods, and even on different helm deployments, so if I understood you correctly, then there shouldnt be a case where these 2 traefik installations are in the same pod.

You start second instance of traefik from CLI from the pod where you already have traefik running.

1 Like

It was true, upon checking again, I managed to make it work, here's the yaml in case anyone is interested:

# Default values for Traefik
image:
  name: traefik
  tag: v2.3.0-rc4

#
# Configure the deployment
#
deployment:
  enabled: true
  # Number of pods of the deployment
  replicas: 1
  # Additional deployment annotations (e.g. for jaeger-operator sidecar injection)
  annotations: {}
  # Additional pod annotations (e.g. for mesh injection or prometheus scraping)
  podAnnotations: {}

# Create an IngressRoute for the dashboard
ingressRoute:
  dashboard:
    enabled: yes
    # Additional ingressRoute annotations (e.g. for kubernetes.io/ingress.class)
    annotations:
      kubernetes.io/ingress.class: traefik-public
    # traefik.ingress.kubernetes.io/router.middlewares: kube-system-localipipwhitelist@kubernetescrd
    # Additional ingressRoute labels (e.g. for filtering IngressRoute by custom labels)
    labels: {}

rollingUpdate:
  maxUnavailable: 1
  maxSurge: 1

#
# Add volumes to the traefik pod.
# This can be used to mount a cert pair or a configmap that holds a config.toml file.
# After the volume has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
# - "--providers.file.filename=/config/dynamic.toml"
volumes: []
# - name: public-cert
#   mountPath: "/certs"
#   type: secret
# - name: configs
#   mountPath: "/config"
#   type: configMap

globalArguments:
  - "--global.checknewversion"
  - "--providers.kubernetescrd.ingressclass=traefik-public"
  - "--providers.kubernetesingress.ingressclass=traefik-public"
#
# Configure Traefik static configuration
# Additional arguments to be passed at Traefik's binary
# All available options available on https://docs.traefik.io/reference/static-configuration/cli/
## Use curly braces to pass values: `helm install --set="additionalArguments={--providers.kubernetesingress,--logs.level=DEBUG}"`
additionalArguments:
  - "--metrics.prometheus=true"
  - "--metrics.prometheus.entryPoint=metrics"
  - "--entrypoints.web.address=:8000"
  - "--entrypoints.websecure.address=:8443"
  - "--experimental.pilot.token="

# Environment variables to be passed to Traefik's binary
env: []
# - name: SOME_VAR
#   value: some-var-value
# - name: SOME_VAR_FROM_CONFIG_MAP
#   valueFrom:
#     configMapRef:
#       name: configmap-name
#       key: config-key
# - name: SOME_SECRET
#   valueFrom:
#     secretKeyRef:
#       name: secret-name
#       key: secret-key

envFrom: []
# - configMapRef:
#     name: config-map-name
# - secretRef:
#     name: secret-name

# Configure ports
ports:
  # The name of this one can't be changed as it is used for the readiness and
  # liveness probes, but you can adjust its config to your liking
  traefik:
    port: 9000
    # Use hostPort if set.
    # hostPort: 9000

    # Defines whether the port is exposed if service.type is LoadBalancer or
    # NodePort.
    #
    # You SHOULD NOT expose the traefik port on production deployments.
    # If you want to access it from outside of your cluster,
    # use `kubectl proxy` or create a secure ingress
    expose: false
    # The exposed port for this service
    exposedPort: 9000
  web:
    port: 8000
    # hostPort: 8000
    expose: true
    exposedPort: 80
    # Use nodeport if set. This is useful if you have configured Traefik in a
    # LoadBalancer
    # nodePort: 32080
  websecure:
    port: 8443
    # hostPort: 8443
    expose: true
    exposedPort: 443
    # nodePort: 32443
  metrics:
    port: 8082
    # hostPort: 8082
    expose: false
    exposedPort: 8082
    # nodePort: 8082

# Options for the main traefik service, where the entrypoints traffic comes
# from.
service:
  enabled: true
  type: LoadBalancer
  # Additional annotations (e.g. for cloud provider specific config)
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-internal: 'false'
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: 
    service.beta.kubernetes.io/aws-load-balancer-subnets: 
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
  # Additional entries here will be added to the service spec. Cannot contains
  # type, selector or ports entries.
  spec:
    externalTrafficPolicy: Local
    # loadBalancerIP: "1.2.3.4"
    # clusterIP: "2.3.4.5"
  loadBalancerSourceRanges: []
    # - 192.168.0.1/32
    # - 172.16.0.0/16
  externalIPs: []
    # - 1.2.3.4

## Create HorizontalPodAutoscaler object.
##
autoscaling:
  enabled: false
#   minReplicas: 1
#   maxReplicas: 10
#   metrics:
#   - type: Resource
#     resource:
#       name: cpu
#       targetAverageUtilization: 60
#   - type: Resource
#     resource:
#       name: memory
#       targetAverageUtilization: 60

# Enable persistence using Persistent Volume Claims
# ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
# After the pvc has been mounted, add the configs into traefik by using the `additionalArguments` list below, eg:
# additionalArguments:
# - "--certificatesresolvers.le.acme.storage=/data/acme.json"
# It will persist TLS certificates.
persistence:
  enabled: true
  accessMode: ReadWriteOnce
  size: 128Mi
  # storageClass: ""
  path: /data
  annotations: {}
  # subPath: "" # only mount a subpath of the Volume into the pod

# If hostNetwork is true, runs traefik in the host network namespace
# To prevent unschedulabel pods due to port collisions, if hostNetwork=true
# and replicas>1, a pod anti-affinity is recommended and will be set if the
# affinity is left as default.
hostNetwork: false

# Additional serviceAccount annotations (e.g. for oidc authentication)
serviceAccountAnnotations: {}

resources:
  requests:
    cpu: "300m"
    memory: "150Mi"
  limits:
    cpu: "300m"
    memory: "150Mi"
affinity: {}
# # This example pod anti-affinity forces the scheduler to put traefik pods
# # on nodes where no other traefik pods are scheduled.
# # It should be used when hostNetwork: true to prevent port conflicts
#   podAntiAffinity:
#     requiredDuringSchedulingIgnoredDuringExecution:
#     - labelSelector:
#         matchExpressions:
#         - key: app
#           operator: In
#           values:
#           - {{ template "traefik.name" . }}
#       topologyKey: failure-domain.beta.kubernetes.io/zone
nodeSelector: {}
tolerations: []

# Pods can have priority.
# Priority indicates the importance of a Pod relative to other Pods.
priorityClassName: ""

# Set the container security context
# To run the container with ports below 1024 this will need to be adjust to run as root
securityContext:
  capabilities:
    drop: [ALL]
  readOnlyRootFilesystem: true
  runAsGroup: 65532
  runAsNonRoot: true
  runAsUser: 65532

podSecurityContext:
  fsGroup: 65532

However, I could only put the token in with the cli command in the additionalArguments block, and not with the yaml format.
Where would the experimental block need to be located for it to work that way?

And also, thank you very much for the lots of help!

There are three different, mutually exclusive (e.g. you can use only one at the same time), ways to define static configuration options in Traefik:

  1. In a configuration file
  2. In the command-line arguments
  3. As environment variables

If you are using the command line already (additionalArgumetns) you have to continue using this for the rest of static configuration.