networking – GCP load balancer can’t connect to instance group: unhealthy upstream error

My Goal

I want to connect my domain (dashboards.example.com) to my Google VM instance on port 3000 to showcase Metabase. I do so through a Google DNS and load balancer.

My problem

While trying to do so and accessing Metabase by going to https://dashboards.example.com I encounter a black screen showing me no unhealthy upstream. Furthermore, the health checks of the load balancers (http & https) say the backend services are unhealthy.

Is there anyone who might know what I am missing? I think I am overlooking something, however, while scanning the whole internet I can’t seem to find the solution.

I tried things like re-starting my machine or clearing the domain and browser cache.

The GCP setup

DNS
My A class DNS points towards the two IP addresses from the load balancers (global external). When I ping the domain dashboards.example.com I correctly see the IP from one of the load balancers.

Load Balancers

• HTTPS I have a global external HTTPS load balancer with a Google managed SSL certificate. The frontend has an Ephemeral IP address and uses port 443. The network service tier is premium. The backend backend-https points to an instance group, uses the HTTPS protocol with named port https. The instance group is in europe-west4 and I added port number 3000. The health check also points to this port. The routing route is the basic one with the host and path empty while pointing at the same backend.

• HTTP I also have a global external HTTP load balancer. The frontend has an Ephemeral IP address and uses port 80. The network service tier is premium. The backend backend-http (different backend pointing to the same instance group) uses the HTTP protocol with named port http. The instance group is in europe-west4 and I added port number 3000. The health check also points to this port. The routing route is the basic one with the host and path empty while pointing at the same backend.

Then in the overview, the health checks already fail, I can’t figure out why.

Instance Group
The instance group is in zone europe-west4-a and contains one VM instance. Furthermore it has two named ports called https & http both leading to port 3000.

Networking
Everything is added in the same subnet or VPC. I added firewall rules to allow traffic from the health checks, SSH & the load balancers towards all instances in the network on TCP port 3000.

Extra
Another weird thing I notice is that the health checks appear to be healthy the moment I shut the VM down.

Is anyone able to spot my mistake?

I tried things like re-starting my machine or clearing the domain and browser cache. I also tried to reconfigure the load balancers multiple times. Without success unfortunately.