Questions tagged [airflow]
Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.
airflow
10,748
questions
1
vote
0
answers
6
views
some dags getting disappeared from airflow randomly and then coming back randomly
We recently deployed Airflow 2.9.1 using helm, some dags getting disappeared from airflow randomly and coming back after refreshing the airflow UI. each time I refresh the browser I see different ...
- 125
0
votes
0
answers
23
views
How do I trigger Airflow Job via Microsoft teams?
In Apache airflow, can we trigger the dag externally.
for instance I have to trigger a job via Microsoft teams.
Is this possible via Microsoft teams or any other chat tools?
0
votes
0
answers
5
views
How to dynamically generate Airflow DAGs with groups, task and conditionals?
I am trying to create a bunch of python scripts to automate the creation of DAGs for Airflow 2.9.x. The idea is to list all tasks in Excel as these DAGs will have to orchestrate tasks for many ...
- 107
0
votes
0
answers
12
views
Do I need Apache Spark to execute my Airflow DAG tasks?
I have a workflow with multiple DAGs. Every DAG has multiple tasks.
These tasks are simple ETL tasks. It involves geo data in the form of kmls, csvs.
An example task:
We have meta data of road ...
0
votes
0
answers
14
views
Non-critical errors in logs cluttering output
I am working with an Airflow environment managed through AWS Managed Workflows for Apache Airflow (MWAA). In my log outputs, I have been noticing non-critical errors that are cluttering the logs, ...
0
votes
0
answers
10
views
Datadog exclude the logs
I am trying to exclude Info-related logs on Datadog from airflow but this is not helping. Any thoughts if I am doing something wrong?
ad.datadoghq.com/worker.logs: '[{"source": "airflow&...
- 29
0
votes
0
answers
28
views
My Airflow can't detect my Jinja Template
i want to connect my query to my json file using jinja template
and i got an error in my airflow
it said "jinja2.exceptions.TemplateNotFound: /configuration/query.sql"
this my code
...
0
votes
1
answer
22
views
How to make DAG running past midnight look at last 3 days of data?
I have a DAG running overnight where I need to run a python function on the past 3 days of data.
My schedule interval is 0 22 * * * and my start_date is pendulum.today(tz='America/Los_Angeles')....
- 1
0
votes
0
answers
42
views
trying to install a package in an "airflow" context, but module can't be found (ModuleNotFoundError)
I have used docker desktop to install airflow. And I have copied some sample dags available on localhost to my vscode environment and that code works fine. But when I tried to apply my python code ...
0
votes
0
answers
18
views
Connecting Airflow to Snowflake account
I am having issue connecting the airflow to Snowflake. I think that the airflow admin GUI has changed a bit. But the main issue, is I am not sure how what is what when it comes account in the ...
0
votes
1
answer
21
views
Uploading dags on Airflow-EKS
Today I have a question
In resume, I have an EKS cluster and I installed Apache Airflow in it.
I decided to create the PV and PVC to maintain the Dags instead of locally in web pod.
The definition is
...
- 45
0
votes
0
answers
11
views
How do I attach the contents of a failed Airflow task log to the failure notification email message?
I have an Airflow DAG that runs a BashOperator task. When it fails I get an email with not much detail:
Try 1 out of 1
Exception:
Bash command failed. The command returned a non-zero exit code 1.
Log:...
- 22.6k
0
votes
0
answers
36
views
Can only create a functional MWAA environment through the UI console, not CDK
I'm trying to create a new MWAA environment through CDK. I'm able to deploy the infrastructure, but the dags won't appear in the Airflow frontend. When I look at the dag processor manager logs I see ...
- 2,057
0
votes
1
answer
25
views
Python doesn't see added module to the same dir
I added to my utils dir a new file that contains dict variable which I want to import into another file.
mapping.py
config = [
{
"filter_column": "SHOW_NAME",
&...
- 1,355
0
votes
1
answer
19
views
Capture airflow run duration
I have a requirement to gather run duration (time) for the last 3 months, for a particular airflow job.
In our CDE environment we use airflow to call spark DBT jobs, of late the run duration of job ...
- 1