Swarm: Traefik returning 404 on compose services

Hi,

I have two devices running Docker; an Intel NUC and a Raspberry Pi. My NUC is used as a mediaplayer/mediaserver. This also is the manager node. The Pi is being used as Home Assistant and MQTT machine and is set as worker node. I wanted to add them to a swarm so I could use Traefik for reverse proxy and HTTPS on both machines.

NUC:
1 docker-compose file for Traefik, Consul and Portainer.
1 docker-compose file for my media apps, NOT IN SWARM (Sabnzbd, Transmission-vpn, Sonarr, Radarr etc).

Pi:
1 docker-compose file for Home Assistant, MQTT etc.

Traefik and Portainer are up and running. I got them setup with `docker stack deploy`. Next I tried to setup my media apps, but they don't need to be connected with the Pi so I tried `docker compose`. Portainer shows the apps are running, but when I go to their subdomain Traefik returns 404 page not found. This makes me conclude that apps running outside the swarm, but connected to Traefik don't work. They also don't show up in the Traefik dashboard.

docker-compose.traefik.yml - docker stack deploy

version: '3.7'

networks:
  traefik_proxy:
    external: true
  agent-network:
    attachable: true

volumes:
  consul-data-leader:
  consul-data-replica:
  portainer-data:

services:
  consul-leader:
    image: consul
    command: agent -server -client=0.0.0.0 -bootstrap -ui
    volumes:
      - consul-data-leader:/consul/data
    environment:
      - CONSUL_BIND_INTERFACE=eth0
      - 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true}'
    networks:
      - traefik_proxy
    deploy:
      labels:
        - traefik.frontend.rule=Host:consul.${DOMAINNAME?Variable DOMAINNAME not set}
        - traefik.enable=true
        - traefik.port=8500
        - traefik.tags=${TRAEFIK_PUBLIC_TAG:-traefik-public}
        - traefik.docker.network=traefik_proxy
        - traefik.frontend.entryPoints=http,https
        - traefik.frontend.redirect.entryPoint=https
        - traefik.frontend.auth.forward.address=http://oauth:4181
        - traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
        - traefik.frontend.auth.forward.trustForwardHeader=true
  consul-replica:
    image: consul
    command: agent -server -client=0.0.0.0 -retry-join="consul-leader"
    volumes:
      - consul-data-replica:/consul/data
    environment:
      - CONSUL_BIND_INTERFACE=eth0
      - 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true}'
    networks:
      - traefik_proxy
    deploy:
      replicas: ${CONSUL_REPLICAS:-3}
      placement:
        preferences:
          - spread: node.id

  traefik:
    image: traefik:v1.7
    hostname: traefik
    restart: always
    networks:
      - traefik_proxy
    ports:
      - target: 80
        published: 80
      - target: 443
        published: 443
      - target: 8080
        published: 8145
    deploy:
      replicas: ${TRAEFIK_REPLICAS:-3}
      placement:
        constraints:
          - node.role == manager
        preferences:
          - spread: node.id
      labels:
        traefik.enable: 'true'
        traefik.backend: traefik
        traefik.protocol: http
        traefik.port: 8080
        traefik.tags: traefik-public
        traefik.frontend.rule: Host:traefik.${DOMAINNAME}
        traefik.frontend.headers.SSLHost: traefik.${DOMAINNAME}
        traefik.docker.network: traefik_proxy
        traefik.frontend.passHostHeader: 'true'
        traefik.frontend.headers.SSLForceHost: 'true'
        traefik.frontend.headers.SSLRedirect: 'true'
        traefik.frontend.headers.browserXSSFilter: 'true'
        traefik.frontend.headers.contentTypeNosniff: 'true'
        traefik.frontend.headers.forceSTSHeader: 'true'
        traefik.frontend.headers.STSSeconds: 315360000
        traefik.frontend.headers.STSIncludeSubdomains: 'true'
        traefik.frontend.headers.STSPreload: 'true'
        traefik.frontend.headers.customResponseHeaders: X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
        traefik.frontend.headers.customFrameOptionsValue: 'allow-from https:${DOMAINNAME}'
        traefik.frontend.auth.forward.address: 'http://oauth:4181'
        traefik.frontend.auth.forward.authResponseHeaders: X-Forwarded-User
        traefik.frontend.auth.forward.trustForwardHeader: 'true'
    domainname: ${DOMAINNAME}
    dns:
      - 1.1.1.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${USERDIR}/docker/traefik:/etc/traefik
      - ${USERDIR}/docker/shared:/shared
    environment:
      CF_API_EMAIL: ${CLOUDFLARE_EMAIL}
      CF_API_KEY: ${CLOUDFLARE_API_KEY}
    command:
      #- "storeconfig" #This is the push to consul, secondary traefik must be created and interfaced to this traefik. Remove this traefik's open ports, it shuts down once consul is messaged.
      - '--logLevel=INFO'
      - '--InsecureSkipVerify=true' #for unifi controller to not throw internal server error message
      - '--api'
      - '--api.entrypoint=apiport'
      - '--defaultentrypoints=http,https'
      - '--entrypoints=Name:http Address::80 Redirect.EntryPoint:https'
      - '--entrypoints=Name:https Address::443 TLS TLS.SniStrict:true TLS.MinVersion:VersionTLS12 CipherSuites:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256'
      - '--entrypoints=Name:apiport Address::8080'
      - '--file'
      - '--file.directory=/etc/traefik/rules/'
      - '--file.watch=true'
      - '--acme'
      - '--acme.storage=/etc/traefik/acme/acme.json'
      - '--acme.entryPoint=https'
      # not yet ready?
      #      - "--acme.TLS-ALPN-01=true"
      - '--acme.dnsChallenge=true'
      - '--acme.dnsChallenge.provider=cloudflare'
      - '--acme.dnsChallenge.delayBeforeCheck=60'
      - '--acme.dnsChallenge.resolvers=1.1.1.1,1.0.0.1'
      - '--acme.onHostRule=true'
      - '--acme.email=admin@${DOMAINNAME}'
      - '--acme.acmeLogging=true'
      - '--acme.domains=${DOMAINNAME},*.${DOMAINNAME},'
      - '--acme.KeyType=RSA4096'
      #Let's Encrypt's staging server,
      #caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
      - '--docker'
      - '--docker.swarmMode'
      - '--docker.domain=${DOMAINNAME}'
      - '--docker.watch'
      - '--docker.exposedbydefault=false'
      #- "--consul"
      #- "--consul.endpoint=consul:8500"
      #- "--consul.prefix=traefik"
      - '--retry'
      - 'resolvers=[192,168.1.1:53,1.1.1.1:53,]'
    depends_on:
      - consul-leader

docker-compose.media.yml - docker compose (NOT IN SWARM)

  sabnzbd:
    image: linuxserver/sabnzbd
    container_name: sabnzbd
    restart: always
    network_mode: service:transmission-vpn
    # depends_on:
    #  - transmission-vpn
    #   ports:
    #      - '${SABNZBD_PORT}:8080'
    volumes:
      - ${USERDIR}/docker/sabnzbd:/config
      - /media/Data/Downloads:/Downloads
    #      - ${USERDIR}/Downloads/incomplete:/incomplete-downloads
    environment:
      PUID: ${PUID}
      PGID: ${PGID}
      TZ: ${TZ}
      UMASK_SET: 002
    deploy:
      replicas: 1
      labels:
        traefik.enable: 'true'
        traefik.backend: sabnzbd
        traefik.protocol: http
        traefik.port: 8080
        traefik.tags: traefik_proxy
        traefik.frontend.rule: Host:sabnzbd.${DOMAINNAME}
        #     traefik.frontend.rule: Host:${DOMAINNAME}; PathPrefix: /sabnzbd
        traefik.frontend.headers.SSLHost: sabnzbd.${DOMAINNAME}
        traefik.docker.network: traefik_proxy
        traefik.frontend.passHostHeader: 'true'
        traefik.frontend.headers.SSLForceHost: 'true'
        traefik.frontend.headers.SSLRedirect: 'true'
        traefik.frontend.headers.browserXSSFilter: 'true'
        traefik.frontend.headers.contentTypeNosniff: 'true'
        traefik.frontend.headers.forceSTSHeader: 'true'
        traefik.frontend.headers.STSSeconds: 315360000
        traefik.frontend.headers.STSIncludeSubdomains: 'true'
        traefik.frontend.headers.STSPreload: 'true'
        traefik.frontend.headers.customResponseHeaders: X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
        #      traefik.frontend.headers.frameDeny: "true" #customFrameOptionsValue overrides this
        traefik.frontend.headers.customFrameOptionsValue: 'allow-from https:${DOMAINNAME}'
        traefik.frontend.auth.forward.address: 'http://oauth:4181'
        traefik.frontend.auth.forward.authResponseHeaders: X-Forwarded-User
        traefik.frontend.auth.forward.trustForwardHeader: 'true'


Sabnzbd runs through a VPN container. This container is connected to traefik_proxy like Traefik itself.

I already tried multiple things like removing the deploy command and just using labels etc but that didn't help at all. My Traefik logs also don't show anything that might be saying what's going wrong.

One Traefik instance can only listen to either Swarmmode instances or compose. It can't listen for both