How do I properly configure my cloudwatch.config file after upgrading from Amazon Linux 2 v4.5.3 to Amazon Linux 2023 v5.1.8?

0

Before upgrading the OS from Linux 2 I had a working cloudwatch.config file. It looked like this:

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/tomcat/application.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat/application.log"]]}`
      log_stream_name = {instance_id}
      datetime_format = %Y-%m-%d %H:%M:%S.%f
      multi_line_start_pattern = {datetime_format}
      file = /var/log/tomcat/application.log

commands:
  "01":
    command: systemctl enable awslogsd.service
  "02":
    command: systemctl restart awslogsd

After seeing that CloudWatch logs Agent is deprecated I ended up with this cloudwatch.config file:

packages:
  yum:
    amazon-cloudwatch-agent: []

files:
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json":
    mode: "000644"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/tomcat/application.log",
                  "log_group_name": {
                    "Fn::Join": [
                      "/",
                      [
                        "/aws/elasticbeanstalk",
                        {"Ref": "AWSEBEnvironmentName"},
                        "var/log/tomcat/application.log"
                      ]
                    ]
                  },
                  "log_stream_name": "{instance_id}",
                  "timestamp_format": "%Y-%m-%d %H:%M:%S.%f",
                  "multi_line_start_pattern": "{timestamp_format}"
                }
              ]
            }
          }
        }
      }

commands:
  "01":
    command: |
      echo "Configuring and starting the CloudWatch agent" >> /var/log/eb-command.log
      /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s >> /var/log/eb-command.log 2>&1 || echo "Failed to start CloudWatch agent" >> /var/log/eb-command.log
  "02":
    command: |
      echo "Enabling CloudWatch agent service" >> /var/log/eb-command.log
      systemctl enable amazon-cloudwatch-agent.service >> /var/log/eb-command.log 2>&1 || echo "Failed to enable CloudWatch agent service" >> /var/log/eb-command.log

If I go to my Elastic Beanstalk environment and request logs, I can see that the logs are populated and the filename and location are what I expect. I can also see that a log group is created in Cloudwatch, but there are no logs in the Log Stream, which I expect there to be.

2 Answers
1

Hello.

Can you confirm that the CloudWatch Agent is running on your EC2 instance?
Also, check the CloudWatch Agent logs to see if any errors are output.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/troubleshooting-CloudWatch-Agent.html#CloudWatch-Agent-files-and-locations

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log or /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
profile picture
EXPERT
answered a month ago
profile picture
EXPERT
reviewed a month ago
  • I appreciate the quick response. From messages log:

    systemd[1]: Started amazon-cloudwatch-agent.service - Amazon CloudWatch Agent.
    audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=amazon-cloudwatch-agent comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    ...
    start-amazon-cloudwatch-agent[1946]: 2024/07/03 07:37:20 Configuration validation first phase succeeded

    I'm not sure where to locate the cloudwatch agent logs, but if I'm understanding correctly from the logs above, the cloudwatch agent is running on the EC2 instance.

  • CloudWatch Agent logs should be output to one of the following locations. Can you check the logs below to see if there are any errors?

    /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
    /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log
    
0
files:
  # ####################################################################################################################
  # First we create a custom config file that lists additional files that should be included when we request
  # "Last 100 lines" of the logs (not just when we select "Full").
  # Note: Several other files are included by default.
  # This ".conf" file is picked up automatically since it is added to the taillogs.d folder.
  # See: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html
  # ####################################################################################################################
  "/opt/elasticbeanstalk/tasks/taillogs.d/my-company-tail-log.conf" :
    mode: "000755"
    owner: root
    group: root
    content: |
      /var/log/cfn-init-cmd.log

  # ####################################################################################################################
  # We also create a custom config file to specify that the application.log file should be streamed to Cloudwatch,
  # (in addition to other files that are streamed to Cloudwatch by default, like eb-engine.log, eb-hooks.log etc)
  # This ".json" file is not picked up automatically, but we point to it in the container_commands below.
  # ####################################################################################################################
  "/opt/aws/amazon-cloudwatch-agent/etc/my-company-amazon-cloudwatch-agent.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/tomcat9/application.log",
                  "log_group_name": "`{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat9/application.log"]]}`",
                  "log_stream_name": "{instance_id}",
                  "timestamp_format": "%Y-%m-%d %H:%M:%S.%f"
                }
              ]
            }
          }
        }
      }

# ####################################################################################################################
# Note: The execution of the commands below is logged in file cfn-init-cmd.log
# ####################################################################################################################
container_commands:
  "01":
    command: |
      sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/my-company-amazon-cloudwatch-agent.json

This is what my coworker ended up using which worked for populating the cloudwatch logs. There are some comments that explain what's going on in this config file, but I don't understand it enough to explain it, so I will leave this question unanswered. If anyone can explain why this is working in a comment I will accept this answer.

answered a month ago