---
title: "How to read multiple Parquet files with different schemas in Apache Spark"
description: "What to do when Apache Spark skips Parquet files with incompatible schemas"
author: "Bartosz Mikulski"
author_bio: "Principal AI Engineer & MLOps Architect. I bridge the gap between \"it works in a notebook\" and \"it works for 200 million users.\""
author_url: https://mikulskibartosz.name
author_linkedin: https://www.linkedin.com/in/mikulskibartosz/
author_github: https://github.com/mikulskibartosz
canonical_url: https://mikulskibartosz.name/read-multiple-parquet-files-with-different-schemas-in-apache-spark
---

When we read multiple Parquet files using Apache Spark, we may end up with a problem caused by schema differences. When Spark gets a list of files to read, it picks the schema from either the Parquet summary file or a randomly chosen input file:

```scala
spark.read.parquet(
List(
    "file_a",
    "file_b",
    "file_c"): _*
)
```

Most likely, you don't have the Parquet summary file because it is not a popular solution. In this case, Spark will try to apply the schema of a randomly chosen file to every file in the list.

It is an annoying problem because if we have additional columns in some files, we may end up with a dataset that does not contain those extra columns because Spark read the schema from a file without those columns.

## How to merge Parquet schemas in Apache Spark?

To solve the issue, we must instruct Apache Spark to merge the schemas from all given files into one common schema. We can do that using the `mergeSchema` configuration parameter:

```scala
spark.read.option("mergeSchema", "true").parquet(...)
```