Missing values

We found 11 training penguins with something missing: two with no measurements at all, eight with no sex, and one whose sex is “_”. Scikit-learn can fill all of them with one line of code, and that line never asks how the holes got there. We will ask first, because the answer decides what we are allowed to put in them.

Why is the value missing?

Our file can’t tell us. The raw data the researchers published has 17 columns, and somebody removed 10 of them before the table got to OpenML. One of the 10 is called Comments. I found our 11 penguins in the raw file, and here is what the researchers wrote about them:

So we have three different stories. The researchers marked the nests and tried to catch both adults at each one. Twice, they caught only one, and the other bird got a row anyway, with a species, an island, and nothing else.

Rows 8 to 11 are the two adults from one nest and the two adults from the next one, both visited on November 9, 2007. All four birds were measured, and none of them gave a usable blood sample. I don’t know what went wrong that day, but it went wrong for two nests in a row during the first weeks of a three-year study.

The last five gave blood, and the test failed. The paper describes the test: a female bird carries two different sex chromosomes and a male carries two copies of the same one, so the lab copies a gene that exists in a different version on each of them and checks whether one version shows up or two. The primers are the short pieces of DNA that start the copying. When they “did not amplify,” nothing got copied, and there was nothing to read.

Look at row 336, the penguin with the “_”. In the raw file, its sex is an ordinary NA with the same comment as the other three Gentoos. The underscore was put there later, by somebody between the researchers and OpenML.

Now, count the species. Five Adelies and four Gentoos have no sex, and not a single Chinstrap. Four of the five failed tests are Gentoos. In our table, a missing sex says something about the species, which is the thing we’re trying to predict, and it says it for reasons that probably have nothing to do with penguins: a bad day on Torgersen and a lab test that failed five times, four of them on Gentoos. Nine holes from two or three incidents are far too few to call a pattern, and a model will call it one anyway.

Your statistics course had names for this. Missing completely at random: the reason for the hole has nothing to do with the data. Missing at random: the reason is connected to columns you can see. Missing not at random: the reason is connected to the value that should be in the hole. A model adds a question the names don’t ask: is the hole connected to the target? Nothing in our seven columns could have answered any of it. I needed the researchers’ comments for that.

Rule 23: Find out why a value is missing before you fill it. The cause tells you whether the other columns can stand in for it, and whether the hole itself says something about the target.

The fix we can’t use

The best replacement for a missing value is the real value. Somebody measures again, or you ask the person who filled in the form, or you find the system that had the number before an export lost it. It’s often an hour of work, and people skip it anyway.

We won’t do it. I’m not going to Antarctica to look for the Adelie nobody caught in 2007. The paper mentions marked nests and says nothing about rings on the birds, but even if every one of them wore a ring with a number, think about the dates. The penguins in our table were adults with eggs between 2007 and 2009. These three species live 10 to 20 years in the wild, and I’m writing this in 2026, so the youngest of them would be about 20 now. Most likely, our penguin died of old age or got eaten by a bear. No. Wait. Bears are on the other end of the planet. Our penguins get eaten by leopard seals. Or it’s alive and nests somewhere else.

And if we found it, we would learn its sex, which doesn’t change, and nothing else. A penguin’s weight today says little about what it weighed 17 years ago, so the two empty rows would stay empty.

The options, from best to worst

Here is the order I go through, starting with the option we have just lost:

1. Get the real value.
2. Predict the missing value from the other columns, with k-nearest neighbors or any other model.
3. Fill in the mean or the median of the column (the most frequent value for a category), and add a column that says the value was missing.
4. If the rows are ordered in time, carry the previous value forward.
5. Drop the row.
6. Drop the column.
7. Fill in the mean or the median without recording that you did it.
8. Fill in a constant.

With the cars, I chose number 5. I dropped the six cars without horsepower in Examining the data and admitted we could do better. Number 2 was available: horsepower, displacement, cylinders, and weight move together (every pair correlates between 0.84 and 0.95 in the correlation matrix), so the other three could have predicted the missing one.

A few of the items move up or down the list, depending on the model and on when you fill the holes.

Dropping a row works only during training. When the model is in use, somebody hands it one row and waits for an answer, and “your row was dropped” isn’t one. Whatever you do with the holes in the training data, the pipeline needs a plan for a hole in a new row.

Dropping the column gives up on the feature for every row. It’s the right call when most of the column is empty, or when nobody will have the value at prediction time, and our sex column may end up there.

Number 4 says “previous” on purpose. Filling a hole with the next value in a time series copies information from the future into the past, and we spent the whole of The danger of overfitting to the test set keeping the future out of the training folds.

The mean without an indicator sits near the bottom because of what it does to the column. Every filled row lands exactly in the middle, so the column gets narrower than it really is and its correlations with the other columns get weaker. Without the indicator, nobody can tell afterward which of the values were measured.

A constant is the worst choice for a model that multiplies the column by a coefficient or measures distances with it. Write −1 into a column of body masses, and a linear model will treat that penguin as lighter than air. For decision trees, I would move it up the list. A tree only asks whether a value is above or below a threshold, so a constant outside the real range gives it one split that separates the missing rows from all the others, which is an indicator column for free. Scikit-learn’s decision trees, random forests, and histogram gradient boosting also accept NaN as it is and decide during training which side of each split the missing rows go to. Two papers say the same thing about models in general: Josse and coauthors show that filling with a constant can be enough for prediction when the holes carry no information, and Le Morvan and coauthors show that a flexible enough model with enough data can make up for almost any imputation.

One method is missing from the list. Multiple imputation draws several plausible values for every hole, builds a separate copy of the dataset from each draw, runs the same analysis on every copy, and combines the results. It was made for statistics, where you need confidence intervals that admit some of the values were guessed. For prediction, a single imputation usually does as well and costs a fraction of the work. If you need it, IterativeImputer with sample_posterior=True, run several times with different seeds, is where to start in scikit-learn.

The indicator column has a trap of its own, and our penguins walk into it. A sex_missing column would be true for five Adelies, four Gentoos, and no Chinstraps. A tree would learn that a penguin without a sex is never a Chinstrap. We know where that pattern comes from (one day in 2007 and one lab test), and the model would carry it to every penguin it sees later. In other datasets, the hole is honest information: a loan applicant who leaves the income field empty is telling you something, and you find that out the same way we did, by asking how the field got empty.

Two penguins we drop

Rows 3 and 339 have a species and an island. Any imputation would give them the body of an average penguin, and we would be training on two birds we made up. They go:

numeric = ['culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g']

train_df = train_df.dropna(subset=numeric).copy()
# The "_" is a missing value, so it becomes one
train_df['sex'] = train_df['sex'].astype(object).replace('_', np.nan)
train_df['sex'].value_counts(dropna=False)

We have 273 penguins left. Nine of them have four good measurements and no sex, and I don’t want to throw away nine measured birds because of one empty cell.

What scikit-learn offers

Besides the experimental IterativeImputer, scikit-learn has three classes for this in sklearn.impute.

SimpleImputer puts one value into every hole of a column: the mean, the median, the most frequent value, or a constant you choose. The last two work on text. On our column, the most frequent value is MALE, because we have 134 males and 130 females. All nine penguins would become males on the strength of four birds. MissingIndicator builds the true/false columns from the list above. (SimpleImputer(add_indicator=True) does both jobs at once.)

KNNImputer finds the rows most similar to the one with the hole and averages their values. It works on numbers only. On our column, it fails with could not convert string to float: 'FEMALE', so the text has to become numbers first: 0 for FEMALE and 1 for MALE.¹

The second problem is what comes out. If three of the five nearest penguins are males, the imputer returns 0.6, and no penguin has a sex of 0.6. For a column with two values, we can live with it, because the 0.6 is the share of males among the neighbors, and rounding it gives the sex most of them have.

It stops working at three values. Say we wanted to fill in a missing species, with Adelie as 1, Chinstrap as 2, and Gentoo as 3. A penguin whose three nearest neighbors are two Gentoos and one Adelie gets (3 + 3 + 1) / 3 = 2.33. What do we do with 2.33? If we round it, the penguin becomes a Chinstrap, a species none of its neighbors belongs to. The numbers were labels, and the imputer did arithmetic on them. We have seen arithmetic on labels before, in One-hot encoding the origin, where US + Europe = Japan.

Nearest neighbors that vote

What we want for a category is a vote. In The nearest cars, the model found the k most similar training cars and averaged their MPG. The classification version finds the k most similar rows and answers with the label most of them have. Scikit-learn calls it KNeighborsClassifier.

Here are the five nearest neighbors of the Adelie in row 11 (37.8 mm by 17.3 mm bill, 180 mm flipper, 3700 g), among the 264 penguins whose sex we know:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

known = train_df.dropna(subset=['sex'])
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
knn.fit(known[numeric], known['sex'])

distances, positions = knn[-1].kneighbors(knn[0].transform(train_df.loc[[11], numeric]))
known.iloc[positions[0]].drop(columns='island').assign(distance=distances[0].round(2))

Three females and two males, so the classifier says FEMALE. (The classifier never saw the species, but all five neighbors are Adelies anyway.)

The scaler is in the pipeline for the reason I gave in Before the first model: without it, the body mass in grams decides the distance almost alone.

Is the imputer any good?

Usually, you can’t check an imputation, because the true values are the ones you don’t have. Here, we can get close. We know the sex of 264 penguins, so we can hide it from some of them, predict it, and compare. Hiding a part of the data and predicting it is what cross-validation does:

from sklearn.dummy import DummyClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

pd.DataFrame({
    'most frequent sex': cross_val_score(DummyClassifier(strategy='most_frequent'), known[numeric], known['sex'], cv=folds),
    'kNN, unscaled': cross_val_score(KNeighborsClassifier(n_neighbors=5), known[numeric], known['sex'], cv=folds),
    'kNN, scaled': cross_val_score(knn, known[numeric], known['sex'], cv=folds),
}).rename_axis('fold').round(3)

The penguins have no time order to protect, so the folds can shuffle. StratifiedKFold does to each fold what the stratified split did to the test set, this time for the sex: every fold gets males and females in equal shares.

The number in the table is the accuracy: the share of penguins the model got right. For a classifier, cross_val_score uses it unless you ask for something else. It’s the first classification metric in this book, and a later chapter is about the ways it can fool you.

Filling every hole with the most frequent sex (what SimpleImputer offered) is right 51% of the time. The scaled neighbors are right 92% of the time, 242 penguins out of 264. Without the scaler, the same model drops to 73% (194 out of 264). I didn’t tune k. Five is the default of KNNImputer, and we have other things to do in this chapter.

Is the 92% the same for every species?

from sklearn.model_selection import cross_val_predict

predicted = cross_val_predict(knn, known[numeric], known['sex'], cv=folds)
by_species = (predicted == known['sex']).groupby(known['species'], observed=True).agg(['sum', 'count', 'mean'])
by_species.set_axis(['right', 'penguins', 'accuracy'], axis=1).round(2)

The Chinstraps come out lowest, with eight mistakes on 54 birds where the rate of the other two species predicts about four. Four penguins are too few to explain, so I won’t try. None of our nine penguins is a Chinstrap, which is lucky.

You may ask why the classifier doesn’t get the species. A male Adelie weighs less than a female Gentoo, so the species should help. It’s the target, though. The model we are going to build won’t know the species of a new penguin, so nothing that prepares its input can use it. I checked what the rule costs us here: when I force the neighbors to come from the same species, the accuracy goes from 91.7% to 92.0%. The neighbors of a penguin are almost always its own species already, as the table for row 11 showed.

The imputer’s nine answers

Now, the imputer. It gets the scaled measurements and the sex as 0 and 1:

from sklearn.impute import KNNImputer

table = pd.DataFrame(StandardScaler().fit_transform(train_df[numeric]), columns=numeric, index=train_df.index)
table['sex'] = train_df['sex'].map({'FEMALE': 0, 'MALE': 1})

imputed = pd.DataFrame(KNNImputer(n_neighbors=5).fit_transform(table), columns=table.columns, index=table.index)
train_df.loc[train_df['sex'].isna(), ['species'] + numeric].assign(imputed_sex=imputed['sex']).sort_index()

For seven penguins, all five neighbors agree. Row 11 gets 0.4, the two males out of five we saw above. Row 336, the penguin with the underscore, gets 0.6. It weighs 4875 g, and the average Gentoo female weighs 4684 g while the average male weighs 5508 g, so I wouldn’t bet much on that one.

Rounded, the nine numbers give the same nine answers as knn.predict(). KNNImputer measures the distance on the columns both rows have, which for these nine are the four measurements, and it takes the neighbors from the rows where the sex is known. For a column of zeros and ones, it’s the voting classifier with an extra step. (One small difference: the imputer’s scaler saw all 273 penguins and the classifier’s saw 264. I checked, and both pick the same five neighbors for every one of the nine.)

Without the scaler, two of the nine answers flip (rows 246 and 336), and nothing would have told us. KNNImputer takes unscaled columns without a warning, and a filled 0 looks the same in the table as a measured one.

If we rounded these numbers, seven of the nine would be females and two would be males. With 92% accuracy, about one of the nine should be wrong. I was going to write that we will never know which one. Then I looked at rows 10 and 11 again.

Two females in one nest

Rows 10 and 11 are the two adults of one nest. I said so myself in Why is the value missing?, and the imputer has just made both of them females. The researchers found an egg in that nest, so one of the two answers has to be wrong.

Rule 12 told us to check a model’s feature importance against what we know about the business. An imputer’s output deserves the same check, and this one fails it. It also shows a bigger mistake, and that one is mine. In The fix we can’t use, I looked for the real values in Antarctica. Every penguin without a sex has a mate, the mate is in the raw file, and I had the raw file open.

raw = pd.read_csv('https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/inst/extdata/penguins_raw.csv')
raw['species'] = raw['Species'].str.split().str[0]
# Both files keep the penguins of a species in the same order, so a stable sort lines them up
raw = raw.sort_values('species', kind='stable').set_index(df.index)
(raw['Body Mass (g)'].fillna(0) == df['body_mass_g'].fillna(0)).all()

It returns True, so row 8 of the raw table is our row 8 (Rule 1 applies to my own joins too). Here are the four penguins from November 9:

raw.loc[8:11, ['studyName', 'Individual ID', 'Date Egg', 'Body Mass (g)', 'Sex']]

I read N5A1 as nest 5, adult 1. The nest numbers start again every season and for every species, so a nest is the study name, the species, and the number together:

raw['nest'] = raw['studyName'] + ' ' + raw['species'] + ' ' + raw['Individual ID'].str.extract(r'(N\d+)')[0]
raw.groupby('nest').size().value_counts()

We get 172 nests with two penguins each, and nothing left over. Now, every penguin gets the sex, the body mass, and the row number of its mate:

by_nest = raw.groupby('nest')
mates = pd.DataFrame({
    'mate_sex': by_nest['Sex'].transform(lambda sex: sex.iloc[::-1].values),
    'mate_mass': by_nest['Body Mass (g)'].transform(lambda mass: mass.iloc[::-1].values),
    'mate_row': by_nest['Sex'].transform(lambda sex: sex.index[::-1].values),
})

Before we use a rule, we check it. I take the nests where both adults are in the training data and both have a sex from the lab:

pairs = train_df.join(mates)
pairs = pairs[pairs['mate_row'].isin(train_df.index)].dropna(subset=['sex', 'mate_sex'])
len(pairs) // 2, (pairs['sex'] != pairs['mate_sex']).sum() // 2

It returns (104, 104). In 104 nests out of 104, one adult is a male and the other is a female. The nearest neighbors needed four measurements and a scaler to get 92%. “The opposite of the mate” needs one lookup and hasn’t been wrong yet.

What does it say about our nine penguins?

holes = train_df[train_df['sex'].isna()].join(mates).sort_index()
holes[['species', 'body_mass_g', 'mate_row', 'mate_sex', 'mate_mass']].assign(kNN=imputed['sex'])

Five of the nine have a mate with a lab result, and all five mates are males. So rows 47, 246, 286, 324, and 336 are females, and we know it from a DNA test, only not from their own.

The imputer agreed on four of them, each time with all five neighbors. It was wrong about row 336, the penguin with the underscore and the 0.6. She weighs 4875 g, which is on the heavy side for a Gentoo female, and her mate weighs 6000 g.

(Two of the five mates, rows 287 and 337, are test penguins. I read two cells of each: the sex, and the body mass printed in the table above. Both are features and no model has seen them, but Rule 14 says looks get counted, so I’m counting four cells.)

What the failed tests have in common

Those five penguins are exactly the five whose “sexing primers did not amplify.” All five are females. I had no reason to expect females before I looked, so the fair question is how often five holes would all share a sex, whichever one, if the failed test had nothing to do with it. The answer is once in 16 tries, about 6%.

In Why is the value missing?, I wrote that the holes probably have nothing to do with penguins, and that nine holes are too few to call a pattern. Five are fewer. So I won’t claim the test fails on females, and I don’t know enough about DNA tests to say why it would. But it looks connected to the sex, and I can’t rule it out. If it is, we have the third name from the statistics course, missing not at random: the reason for the hole is connected to the value that should be in it.

Think about what that would do to the simple methods. SimpleImputer with the most frequent value would have written MALE into all five cells and been wrong five times out of five, because it assumes the penguins with a hole look like the penguins without one. And the sex_missing column I warned about wouldn’t only have pointed at the species. For these five, it would have been the sex itself, written in another column.

The two nests without any answer

Rows 8 to 11 are left. Their mates have no sex either, so there is nothing to look up. We still know more than the imputer did: each nest holds one male and one female, so we only have to decide which is which.

In Males and females, the males were heavier in every species. Is the male also the heavier bird of his own nest?

different = pairs[pairs['body_mass_g'] != pairs['mate_mass']]
len(different) // 2, ((different['body_mass_g'] > different['mate_mass']) == (different['sex'] == 'MALE')).sum() // 2

It returns (103, 97). One nest has two birds of the same weight. In 97 of the other 103, the heavier bird is the male, which is 94%. That’s about as good as the nearest neighbors (the two numbers come from different birds, so I won’t rank them), and it takes one column, because it compares a penguin with its own mate and no longer with five strangers of roughly the same size.

In nest 5, row 9 (4250 g) is heavier than row 8 (3475 g). In nest 6, row 11 (3700 g) is heavier than row 10 (3300 g). The imputer had row 11 as a female on a vote of three to two, and it was unanimous about row 10, so of the two females in one nest, it was the close vote that was wrong.

By the new labels, the imputer got seven of the nine right. Its two misses, rows 11 and 336, are the only two penguins where the neighbors disagreed. The 0.4 and the 0.6 were telling us which answers to doubt, and rounding would have thrown that away.

opposite = {'MALE': 'FEMALE', 'FEMALE': 'MALE'}
from_mate = holes['mate_sex'].map(opposite)
from_mass = pd.Series(np.where(holes['body_mass_g'] > holes['mate_mass'], 'MALE', 'FEMALE'), index=holes.index)

train_df['sex_source'] = 'lab'
train_df.loc[holes.index, 'sex_source'] = np.where(from_mate.notna(), 'mate', 'heavier of the pair')
train_df.loc[holes.index, 'sex'] = from_mate.fillna(from_mass)
train_df.loc[holes.index, ['species', 'body_mass_g', 'sex', 'sex_source']]

The training data now has 137 females and 136 males, and no holes.

The sex_source column stays in the table and out of the features. As a feature, it would be the indicator trap again. As a record, it’s what separates this from number 7 on my list: anybody who opens the table later can see which 264 values came from the lab, which five came from a mate, and which four are my guess with a 94% rule behind it.

So number 1 on the list was available the whole time, for five penguins out of nine, in a file I had already read once for its comments. I left The fix we can’t use the way I first wrote it because that’s how it happens: I decided where the real value had to be, saw that I couldn’t go there, and stopped looking.

Rule 24: Before you give up on the real value, look for it in the data you already have.

The two empty penguins have mates too, by the way. Both mates are females, so we now know the birds nobody caught were males, which doesn’t help a row without measurements.

What a new penguin gets

None of it works on a new penguin. Whoever hands the model one bird won’t bring its mate along. A model that uses the sex needs something it can do alone, to one row, and there the nearest neighbors still have a job. They go into the pipeline, so that during cross-validation the imputer learns only from the training part of each fold, like the encoder in Fitting the encoder inside the folds:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OrdinalEncoder

sex_encoder = OrdinalEncoder(categories=[['FEMALE', 'MALE']], handle_unknown='use_encoded_value', unknown_value=np.nan)

fill_sex = make_pipeline(
    ColumnTransformer([
        ('numeric', StandardScaler(), numeric),
        ('sex', sex_encoder, ['sex']),
    ]),
    KNNImputer(n_neighbors=5),
)

The encoder knows two values. Anything else, the “_” included, becomes a NaN inside the pipeline, and the imputer fills it. We cleaned the underscore by hand in the training table, but a model in use can’t count on its caller to remember that step.

The pipeline doesn’t round. It passes the 0.6 on as the share of males among the neighbors, and a model can use a number like that.

The pipeline also sets a trap for the next chapter. The species model will train on real sexes, from the lab and from the mates. A new penguin arrives without one and gets a guess made from its own four measurements. If the validation folds keep the real sex, they will measure a column the model won’t have in use, and the score will overstate what the sex is worth. So whenever a model uses the sex, we will blank it in the validation part of every fold and let fill_sex impute it, the way it will happen to a new penguin.

The nests bring back Rule 8 as well. In What is missing, I said that mates sit on both sides of the split and that I didn’t think it leaks. We now know that 172 nests out of 172 hold one pair, and the folds will split pairs the same way the test set does. For the species model, I still think it’s harmless: a mate has the same species and the opposite sex, and its measurements are its own. It would stop being harmless the moment a feature came from the nest, and “the opposite of the mate” is such a feature, which is one more reason it fills the training table and stays out of the pipeline. When we have a species model, splitting the folds by nest will show whether I’m right.

One thing we can’t do is check any of this on the test set. The shuffle in Splitting the penguins put all 11 incomplete penguins into the training data, so the 69 test penguins have no holes. I could reshuffle until some land there, but then I would be picking the split by what is in it, and Rule 14 counts that as a look at the test set.

We have already classified something

On the way to those nine cells, we trained a model that predicts a category from four measurements, and we compared it with a dummy on rows it hadn’t seen. I never called it a classifier, but it is one. It predicts the sex instead of the species, and the species model in the next chapter gets built the same way.

The same is true of every imputation that predicts the missing value. It is a model, so it needs a baseline and a validation of its own, and it can leak the way the encoder did in Fitting the encoder inside the folds.

There is one more reason to prefer the mates. Whatever the nearest neighbors say about a penguin’s sex, they made it out of the four measurements, and the species model gets the same four measurements directly. The values we kept come from somewhere else: five from the lab result of another bird, and four from comparing two birds that share a nest.

And we still haven’t decided whether the species model gets the sex at all. So in the next chapters, we will train the models in several versions, with and without the sex, with and without the island, and see what each column is worth.