How to configure Spark to maximize resource usage while using AWS EMR
Apache Spark AWS EMR

How to configure Spark to maximize resource usage while using AWS EMR

How to configure EMR to use all available resources when running a Spark cluster
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When we run a Spark cluster on EMR, we often create a new cluster for every Spark job. In that case, we want to use all available resources, but changing the configuration is annoying and error-prone. How many times have you forgotten to change the Spark settings after changing the EMR instances to less powerful ones? Of course, the Spark job failed because it could not allocate the resources you wanted.

What is even worse, when you forget to change the settings after changing the instance to a bigger one, you pay for a better cluster, but you are not using it entirely.

Fortunately, Spark’s EMR version has a special configuration parameter that replaces all of the cumbersome parameters, such as the executor memory, the executor cores, or parallelism.

Instead of them, we should enable the maximizeResourceAllocation feature:

--conf maximizeResourceAllocation=true

when we call the spark-submit script.

Older post

How to use AWSAthenaOperator in Airflow to verify that a DAG finished successfully

How to check that an AWS Athena table contains data after running an Airflow DAG.

Newer post

How to find and terminate an idle Redshift session

How to find the idle session that is blocking the connection pool in Redshift

Related Posts

Data Engineering Apache Spark

What is shuffling in Apache Spark, and when does it happen?

When does an Apache Spark cluster perform the shuffle operation?
Published on: 20 May 2022

Apache Spark

How to measure Spark performance and gather metrics about written data

How to track Spark metrics in AWS CloudWatch
Published on: 05 Mar 2021

Apache Spark

How to combine two DataFrames with no common columns in Apache Spark

Use full outer join to combine two Apache Spark DataFrames with no common columns
Published on: 06 Dec 2020

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>