Target Group health checks failing to private instances

Security Group Configuration:

Ensure that the security group associated with your instances allows inbound traffic on port 80 from the security group of the ALB. Ensure that the security group associated with your ALB allows inbound traffic on port 80 from anywhere (0.0.0.0/0).

Network Access Control List (NACL):

Verify that the NACLs associated with your subnets allow inbound and outbound traffic on port 80.

Health Check Configuration:

Confirm that the health check settings for the target group are correctly configured. The path should be / and the port should be 80.

Instance Web Server:

Ensure that your web server is running correctly and can respond to HTTP requests on port 80. You can SSH into the instance and use curl to test this.

curl http://localhost/

Route Tables:

Verify that the route tables for your private subnets are correctly configured. Even though your instances don't need access to the internet, the ALB must be able to reach them.

Target Group Registration:

Ensure that the instances are correctly registered to the target group and that they are in the running state. Here are some specific steps you can take:

Security Group Settings Instances Security Group:

Inbound rules: Type: HTTP Protocol: TCP Port Range: 80 Source: Security group of the ALB ALB Security Group:

Inbound rules: Type: HTTP Protocol: TCP Port Range: 80 Source: Anywhere (0.0.0.0/0) Testing Network Access SSH into one of your instances and check if the web server is responding correctly:

Testing Network Access

SSH into one of your instances and check if the web server is responding correctly:

curl http://localhost/

Check Health Check Path Ensure that the health check path is correct in the target group configuration. It should be / unless your web server is serving health checks at a different path.

Verify Network ACLs Check the network ACLs associated with your subnets to ensure that they are allowing traffic on port 80:

Inbound rules should allow:

HTTP (80) from the ALB security group Outbound rules should allow:

HTTP (80) to the ALB security group Example Terraform Configuration for Security Groups Here is an example of how you might configure your security groups using Terraform:

resource "aws_security_group" "alb_sg" {
  name        = "alb-sg"
  description = "Security group for ALB"
  vpc_id      = var.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group" "instance_sg" {
  name        = "instance-sg"
  description = "Security group for instances"
  vpc_id      = var.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    security_groups = [aws_security_group.alb_sg.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Debugging Steps Check the ALB Access Logs: Enable access logs for your ALB and check if the health check requests are reaching your instances.

Instance Logs: Check the logs on your instances to see if they are receiving and responding to health check requests.

Reachability Analyzer: Use the AWS VPC Reachability Analyzer to diagnose network connectivity issues.

Hi Fon,

Please go through the below steps I hope it will help to resolve your issue.

The issue with the health checks failing for the internal ALB targeting instances in private subnets could be due to several reasons. Let's troubleshoot step by step:

Security Groups:

• Ensure the security group attached to your instances allows inbound traffic on port 80 from the security group of the ALB. It should be something like:

Type: HTTP
Protocol: TCP
Port Range: 80
Source: <ALB Security Group>

The security group attached to the ALB should allow outbound traffic on port 80 to the instances. This is usually open by default.

Health Check Configuration:

• Verify the health check path is correct. By default, it is set to /, but if your application serves content on a different path, update the health check path accordingly.
• Ensure the health check port matches the port on which your application is listening (in this case, port 80).

Network ACLs:

• Check the Network ACLs associated with the subnets. Ensure that they allow inbound and outbound traffic on port 80.

Route Tables:

• Even though your instances are in private subnets, ensure there is proper routing within the VPC. The instances should be able to communicate with the ALB. Check the route tables for the subnets and ensure there are no restrictive routes.

Instance Health:

• Verify that your instances are running and the web server is correctly set up and running.
• Manually SSH into the instances (if possible) and use curl or a similar tool to test the HTTP server locally.

curl http://localhost:80

Logs:

• Check the ALB access logs and instance logs for any clues or errors.
• Enable and review the ALB’s access logs to see if the health check requests are reaching the instances.

ALB DNS Resolution:

• Ensure that the ALB DNS name resolves correctly within the VPC and the instances can access it.

Target Group Registration:

• Verify the instances are correctly registered to the target group.

Hi Fon,

Please try this solution,

Step 1 Security Group Check

Verify the security group associated with your internal ALB. To an inbound rule allows HTTP traffic on port 80 from the security group of your instances, not "anywhere." You can find the instance security group ID in the EC2 console.

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html

Step 2 Network Reachability

https://docs.aws.amazon.com/vpc/latest/userguide/route-table-options.html

Route Table Update

Navigate to the route table associated with your private subnets. Edit the route table and add a route with a destination of "0.0.0.0/0" and a target of your NAT Gateway (assuming you have one). This grants instances access to the internal ALB for health checks.

https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

health check configuration in your target group. Ensure it's set for HTTP on port 80 with a path of "/"

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html

Are you able to connect directly to one of the servers from inside your VPC and test if the application is returning a healthy response to requests for the / path that the health check is using? Without a route through a NAT gateway to the internet, an external dependency might easily cause the application to fail, while it would work when residing in a public subnet and having internet connectivity.