Connection refused while trying to connect to my ingress

I have installed a Kubernetes cluster on 4 Raspberry Pis 4 for educational purposes. Now I am trying to setup an ingress using Traefik 1.7. As a starting point I want to put my ingress to an Nginx pod. Trying to connect to the ingress on my cluster results in a connection refused response.

> curl 192.168.178.31 -H "HOST: nginx"
curl: (7) Failed to connect to 192.168.178.31 port 80: Connection refused
> kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
kube01   Ready    master   14h   v1.17.3   192.168.178.31   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     docker://19.3.6
kube02   Ready    <none>   14h   v1.17.3   192.168.178.46   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     docker://19.3.6
kube03   Ready    <none>   14h   v1.17.3   192.168.178.47   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     docker://19.3.6
kube04   Ready    <none>   14h   v1.17.3   192.168.178.48   <none>        Raspbian GNU/Linux 10 (buster)   4.19.97-v7l+     docker://19.3.6

Regarding troubleshooting I followed the information provided on the Kubernetes site

What works:

  • Exposing nginx deployment as nodeport and connecting to it
  • Kubernetes Dashboard
  • Curl into service from Master node (curl 10.104.113.207:80)
  • From ingress controller (sh in Traefik; installed alpine image): curl -H "HOST: nginx" localhost
  • From nginx pod: 'nslookup nginx-deploy-main'
  • From node kube01: "nslookup nginx-deploy-main.default.svc.cluster.local 10.38.0.0"
  • From node kube01: curl 10.104.113.207

I have no glue what else I can check. Given all the things that works, it seem to be really related to the ingress not processing the request. I am really stuck and appreciate any help.

> kubectl get pods --all-namespaces
NAMESPACE              NAME                                          READY   STATUS    RESTARTS   AGE
default                nginx-deploy-main-7cc547b6f7-tls5s            1/1     Running   0          13h
kube-system            coredns-6955765f44-tddgb                      1/1     Running   0          14h
kube-system            coredns-6955765f44-wq986                      1/1     Running   0          14h
kube-system            etcd-kube01                                   1/1     Running   0          14h
kube-system            kube-apiserver-kube01                         1/1     Running   0          14h
kube-system            kube-controller-manager-kube01                1/1     Running   0          14h
kube-system            kube-proxy-58d2r                              1/1     Running   0          14h
kube-system            kube-proxy-kxcsg                              1/1     Running   0          14h
kube-system            kube-proxy-n5vjd                              1/1     Running   0          14h
kube-system            kube-proxy-sqnrm                              1/1     Running   0          14h
kube-system            kube-scheduler-kube01                         1/1     Running   0          14h
kube-system            traefik-ingress-controller-57b54878b6-d7q4z   1/1     Running   0          12h
kube-system            weave-net-brd4s                               2/2     Running   0          14h
kube-system            weave-net-jkrsh                               2/2     Running   0          14h
kube-system            weave-net-nk6nr                               2/2     Running   0          14h
kube-system            weave-net-tbbh4                               2/2     Running   0          14h
kubernetes-dashboard   dashboard-metrics-scraper-7b8b58dc8b-w8dzl    1/1     Running   0          14h
kubernetes-dashboard   kubernetes-dashboard-866f987876-bzwnv         1/1     Running   0          14h

My Service:

> kubectl describe svc nginx-deploy-main
Name:              nginx-deploy-main
Namespace:     default
Labels:            run=nginx
Annotations:       <none>
Selector:          run=nginx-main
Type:              ClusterIP
IP:                10.104.113.207
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.38.0.2:80
Session Affinity:  None
Events:            <none>

The Nginx pod:

> kubectl describe pod nginx-deploy-main-7cc547b6f7-tls5s
Name:         nginx-deploy-main-7cc547b6f7-tls5s
Namespace:    default
Priority:     0
Node:         kube04/192.168.178.48
Start Time:   Thu, 05 Mar 2020 20:27:20 +0100
Labels:       pod-template-hash=7cc547b6f7
              run=nginx-main
Annotations:  <none>
Status:       Running
IP:           10.38.0.2
IPs:
  IP:           10.38.0.2
Controlled By:  ReplicaSet/nginx-deploy-main-7cc547b6f7
Containers:
  nginx:
    Container ID:   docker://e05485ef423dfa370eb1e20933cdd44a792ce78b3c1d9246f7e32e0d958e891a
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:2539d4344dd18e1df02be842ffc435f8e1f699cfc55516e2cf2cb16b7a9aea0b
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 05 Mar 2020 20:27:36 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-56nt9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-56nt9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-56nt9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

The ingress:

> kubectl describe ingress
Name:             ingress-resource-1
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host   Path  Backends
  ----   ----  --------
  nginx
         /   nginx-deploy-main:80 (10.38.0.2:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"networking.k8s.io/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"ingress-resource-1","namespace":"default"},"spec":{"rules":[{"host":"nginx","http":{"paths":[{"backend":{"serviceName":"nginx-deploy-main","servicePort":80},"path":"/"}]}}]}}

Events:  <none>

Log file of ingress controller:

> kubectl logs traefik-ingress-controller-57b54878b6-d7q4z -n kube-system

time="2020-03-05T19:58:38Z" level=info msg="Traefik version v1.7.14 built on 2019-08-14_09:46:58AM"
time="2020-03-05T19:58:38Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2020-03-05T19:58:38Z" level=info msg="Preparing server http &{Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0x621da80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2020-03-05T19:58:38Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0x621daa0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2020-03-05T19:58:38Z" level=info msg="Starting provider configuration.ProviderAggregator {}"
time="2020-03-05T19:58:38Z" level=info msg="Starting server on :8080"
time="2020-03-05T19:58:38Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}"
time="2020-03-05T19:58:38Z" level=info msg="ingress label selector is: \"\""
time="2020-03-05T19:58:38Z" level=info msg="Creating in-cluster Provider client"
time="2020-03-05T19:58:38Z" level=info msg="Starting server on :80"
time="2020-03-05T19:58:38Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T19:58:38Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T20:08:40Z" level=warning msg="A new release has been found: 2.1.6. Please consider updating."
time="2020-03-05T20:23:45Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T20:23:45Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T21:10:00Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T21:10:00Z" level=info msg="Server configuration reloaded on :80"
time="2020-03-05T21:10:49Z" level=info msg="Server configuration reloaded on :8080"
time="2020-03-05T21:10:49Z" level=info msg="Server configuration reloaded on :80"

Endpoints:

> kubectl get endpoints nginx-deploy-main
NAME                ENDPOINTS      AGE
nginx-deploy-main   10.38.0.2:80   14h

Kubeproxy Log on kube01:

> journalctl -u kubeproxy
-- Logs begin at Tue 2020-03-03 17:24:11 GMT, end at Fri 2020-03-06 10:15:36 GMT. --
-- No entries --