The “fillna” function in Pandas not only can replace missing values with a given constant value, like in this example:

Table of Contents

  1. Get Weekly AI Implementation Insights

import pandas as pd
import numpy as np
df = pd.DataFrame([[np.nan], [2], [np.nan], [0]])
df
A dataframe with missing values
A dataframe with missing values
df.fillna(47)
Missing values replaced with a constant
Missing values replaced with a constant

You can also replace a missing value with the next (or previous) value in the data frame!

df.fillna(method = "ffill")
Missing values filled with the previous existing value.
Missing values filled with the previous existing value.

Note that the first value cannot be replaced because nothing is preceding it.

Get Weekly AI Implementation Insights

Join engineering leaders who receive my analysis of common AI production failures and how to prevent them. No fluff, just actionable techniques.

You can also use the value of the next row to fill a missing value.

df.fillna(method = "bfill")
Missing values filled with the next existing value.
Missing values filled with the next existing value.

Get Weekly AI Implementation Insights

Join engineering leaders who receive my analysis of common AI production failures and how to prevent them. No fluff, just actionable techniques.

Older post

Forward feature selection in Scikit-Learn

Two workarounds to get an equivalent of forward feature selection in Scikit-Learn

Newer post

Import Jupyter Notebook from GitHub

The easiest way to access someone else’s code in your own notebook

Engineering leaders: Is your AI failing in production? Take the 10-minute assessment
>