how to handle file processing with Lambda
Hello I've files which are constantly created on S3 with a standard format, fixed lenght; say it's AAA-YYY-ZZZ. Depending on the values assumed by YYY there are different lambda to be invoked to process the file (these lambdas are already implemented in Java, not easy to refactor or change).
From what we've seen, we have two options:
- set up a filter based on the prefix or suffix of the file name. However, S3 does not support filters based on substrings other than prefixes or suffixes. I could use AAA or ZZZ value, but not YYY. So this doesn't seem to be an option.
- Use an intermediary lambda function, that will select and call the second lambda.
Is there any other way we can do this filtering/triggering? Or which is the best approach to architect this ?
- Newest
- Most votes
- Most comments
Hello.
I thought option 2 was effective if it was not possible to refactor the currently existing Lambda code.
I think the problem can be solved by calling Lambda with an event trigger when the file is created in S3 and calling the existing Lambda if it matches "YYY".
You can send the S3 events to EventBridge and there you can create different filters using wildcards, each filter will trigger a different function.
Look at this blog post that describes wildcards, which includes a very similar example to yours.
You can also leverage the S3 Object Tags. If it's possible for the client app to add tag each time an object is created in S3 bucket, e.g., { "YYY": "<value_of_yyy>" }, then you can get your (intermediary) Lambda function (trigger) to evaluate the value of this tag and call the corresponding (Java) Lambda function to process the object based on the value of your YYY tag.
Relevant content
- Accepted Answerasked a year ago
AWS OFFICIALUpdated 4 years ago- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
