Multiline Parsing Configuration in Built-In Fluent Bit Log Router on EKS Fargate

0

I am currently utilizing the log router of EKS Fargate and encountering issues with Multiline Parsing. Despite following the documentation provided for Fluent Bit's multiline parsing at Fluent Bit Docs https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#built-in-multiline-parsers , I have not been successful in resolving the issue.

Could you please guide how to configure this correctly? Below is my current configuration for handling logs:

kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  flb_log_cw: "true"  # Set to true to ship Fluent Bit process logs to CloudWatch.
  filters.conf: |
    [FILTER]
      Name parser
      Match *
      Key_name log
      Parser cri

    [FILTER]
      Name                kubernetes
      Match               kube.*
      Merge_Log           On
      K8S-Logging.Parser  On
      K8S-Logging.Exclude On      

    [FILTER]
      Name                rewrite_tag
      Match               kube.*
      Rule                $kubernetes['namespace_name'] ^pe-stg-nginx-ingress$ nginx-ingress false
      Rule                $kubernetes['namespace_name'] ^pe-stg-fg$ app false

  output.conf: |
    [OUTPUT]
      Name  es
      Match nginx-ingress
      Index ***
      Host  ***
      HTTP_User ***
      HTTP_Passwd ***
      Port  ***
      tls   On
      tls.verify Off
      Suppress_Type_Name On

  output.conf: |
    [OUTPUT]
      Name  es
      Match app
      Index ***
      Host  ***
      HTTP_User ***
      HTTP_Passwd ***
      Port  ***
      tls   On
      tls.verify Off
      Suppress_Type_Name On

  parsers.conf: |
    [PARSER]
      Name java
      Format cri
      Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
      Time_Key    time
      Time_Format %Y-%m-%dT%H:%M:%S.%L%z

However, the logs are still not being parsed as expected. Enter image description here

Here is an example of the Java application log that I wish to parse in multiline format:

2024-06-19T03:43:17.747-04:00  INFO 1 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-06-19T03:43:17.848-04:00 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'pendingRolloverScheduler': Unsatisfied dependency expressed through field 'pendingRolloverService': Error creating bean with name 'pendingRolloverService': Unsatisfied dependency expressed through field 'rolloverService': Error creating bean with name 'rolloverService': Invocation of init method failed
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:767) ~[spring-beans-6.0.13.jar!/:6.0.13]
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:747) ~[spring-beans-6.0.13.jar!/:6.0.13]
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:145) ~[spring-beans-6.0.13.jar!/:6.0.13]

I would greatly appreciate your assistance in resolving this matter. Thank you very much for your support.

1 Answer
0

Hello,

Please note that there is a feature request that is already on container roadmap - "[EKS][Fargate] [request]: Multi-line logging support with Fluent Bit" [1]. Please refer [1] more details. The GitHub issue also mentions that "EKS Fargate has added support for builtin multiline parsers and it is enabled by default [2]. Configurable multi-line parsers are not supported yet."

You can also refer the comment [3] as mentioned in the above GitHub issue to gain some insights. Additionally you can check the Multi-line filter as well, which helps to concatenate messages that originally belong to one context but were split across multiple records or log line. You can refer the document [3] to know more about the same. However, feel free to ignore this if its not related to your use-case.

The GitHub issue on container-roadmap is in "Coming Soon", however, unfortunately, there is no ETA on the open request. You can do a "+1" on the issue to gain more traction on the same.

[1] https://github.com/aws/containers-roadmap/issues/1326

[2] https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#built-in-multiline-parsers

[3] https://docs.fluentbit.io/manual/pipeline/filters/multiline-stacktrace

You can also stay updated on the latest AWS services and feature updates by regularly visiting (i) https://forums.aws.amazon.com/, (ii) https://aws.amazon.com/new/ and (iii) https://aws.amazon.com/blogs/

AWS
answered 2 months ago