Questions tagged with Amazon EMR

Everyday a new emr cluster span up and terminated after completing the step job. Checking the cloudtrail, seems a Data Pipeline created it. I am not sure how to get more details like who created, what...

I want to save my pyspark dataframe in RecordIO protobuf format. I am using Amazon EMR to run my pyspark scripts, and I want to use AWS SageMaker to train a machine learning model. SageMaker pipe mode...

Hello, I have a driver and executor pod template files exist in s3. But when I run the job, it failed wirh FileNotFoundException: s3:/<s3bucket>/podtemplate/driver.yaml (no such file or directory)...

Hello, Please share the difference between AWS Glue and AWS EMR and which one we should use and when? Thanks,

In continuation to the ticket - https://repost.aws/questions/QUfxjbaGrXRTSKGy4-rnQ8Uw/how-to-upgrade-python-version-in-emr-since-python-3-7-support-discontinued , the latest version of EMR 6.14.0...

I am facing issues using inline clustering and compaction in EMR, with the following error.. EMR : 6.13.0 Hudi: 0.13.1 com.esotericsoftware.kryo.KryoException: Unable to find class:...

We have some parquet format files on AWS s3 and we want to create iceberg table with these files. Can Glue Crawler do this?

I am using EMR 6.13.0(EMR on EC2) and it is present in eu-central-1 region. i have my s3 buckets in eu-central-1 and eu-west-1 region, i can be able to access the s3 buckets present in eu-central-1...

I am trying to create a EMR cluster with version 6.13.0 and spark installed on it with version Spark 3.4.1 But the service role attached gives error that it does not have EC2 permissions. I tried...

I have a emr 5.29 cluster running spark and hive application and received notification that AL1 will be unsupported after Dec 23. I am working on applications migration to emr 5.36. Questions, 1. What...

I am running catalog queries like spark.catalog.tables, or show databases or show tables. All these commands failing emr_instance_profile role is not authorized to perform: glue:CreateDatabase on...

Would like to have information about Graviton instance support EMR studio schedule/plan/roadmap?