How to read multiple Parquet files with different schemas in Apache Spark
Apache Spark

How to read multiple Parquet files with different schemas in Apache Spark

What to do when Apache Spark skips Parquet files with incompatible schemas
Bartosz Mikulski
Bartosz Mikulski – 1 min read

When we read multiple Parquet files using Apache Spark, we may end up with a problem caused by schema differences. When Spark gets a list of files to read, it picks the schema from either the Parquet summary file or a randomly chosen input file:

Table of Contents

  1. How to merge Parquet schemas in Apache Spark?
spark.read.parquet(
List(
    "file_a",
    "file_b",
    "file_c"): _*
)

Most likely, you don’t have the Parquet summary file because it is not a popular solution. In this case, Spark will try to apply the schema of a randomly chosen file to every file in the list.

It is an annoying problem because if we have additional columns in some files, we may end up with a dataset that does not contain those extra columns because Spark read the schema from a file without those columns.

How to merge Parquet schemas in Apache Spark?

To solve the issue, we must instruct Apache Spark to merge the schemas from all given files into one common schema. We can do that using the mergeSchema configuration parameter:

spark.read.option("mergeSchema", "true").parquet(...)
Older post

How to determine the partition size in Apache Spark

How to choose the proper partition size and the number of partitions to run an Apache Spark job

Newer post

Use regexp_replace to replace a matched string with a value of another column in PySpark

Use regex to replace the matched string with the content of another column in PySpark

Related Posts

Data Engineering Apache Spark

What is shuffling in Apache Spark, and when does it happen?

When does an Apache Spark cluster perform the shuffle operation?
Published on: 20 May 2022

Apache Spark

How to measure Spark performance and gather metrics about written data

How to track Spark metrics in AWS CloudWatch
Published on: 05 Mar 2021

Apache Spark

How to combine two DataFrames with no common columns in Apache Spark

Use full outer join to combine two Apache Spark DataFrames with no common columns
Published on: 06 Dec 2020

Are you looking for an experienced AI consultant? Do you need assistance with your RAG or Agentic Workflow?
Schedule a call, send me a message on LinkedIn. Schedule a call or send me a message on LinkedIn

>