Models that don't draw lines

Every model since Improving the model adds up columns multiplied by coefficients, and whatever I did to the columns or the coefficients, the folds kept landing at 3.3. In Standardization, I promised you algorithms that measure the distance between rows. This chapter has one of them, and one more that doesn’t measure anything and doesn’t care about the scale of the columns at all.

The nearest cars

We have already measured distances between cars twice. In The Skodas in PCA space, the nearest training car to the Skoda Citigo turned out to be a rotary Mazda, and in Bending the line the cars far from everything the model was trained on got the worst predictions. k-nearest neighbors (kNN in the tables) turns that measurement into a model. To predict the MPG of a car, it finds the k training cars closest to it and returns the average of their MPG. The distance is the same one we computed by hand for the Skodas:

There’s nothing to learn. fit() stores the training rows, and all the work happens in predict(). If you wanted to port this model to the washing machine from Why look inside the model?, you would ship all 307 training cars with it.

from sklearn.neighbors import KNeighborsRegressor

neighbors = {}
for k in [1, 3, 5, 10, 20, 40]:
    scaled = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=k))
    neighbors[k] = {
        'training RMSE': root_mean_squared_error(y_train, scaled.fit(X_train[numeric], y_train).predict(X_train[numeric])),
        'fold RMSE, scaled': fold_rmse(scaled, X_train[numeric]).mean(),
        'fold RMSE, unscaled': fold_rmse(KNeighborsRegressor(n_neighbors=k), X_train[numeric]).mean(),
    }

pd.DataFrame(neighbors).T.rename_axis('k').to_markdown()

k is the knob. With k=1, the model copies the MPG of the single closest training car. Asked about the training cars themselves, it finds each car’s own row, so its training RMSE is almost 0. (The 0.17 comes from one pair: an AMC Ambassador SST and an AMC Matador share every spec, and one did 17 mpg while the other did 14. Same maker, same engine, same weight to the pound, 3 mpg apart.) On the folds, k=1 scores 3.67, as bad as the year-blind straight line we started Improving the model with. With k=40, every prediction is an average of 40 cars (an eighth of the training data), and a light car gets averaged with heavier ones.

The polynomial degree in Bending the line had the same trade-off between bias and variance, with the knob turned the other way: here, a small k is the flexible model. The scaled version does best at k=10, with 3.20, below the pipeline’s 3.33.

What the scaler decides

Now, look at the last column. Without the scaler, k=3 and k=5 beat every scaled version. In Standardization, I promised that every algorithm measuring distances would get its scaler inside the pipeline, and here the unscaled model wins. Why?

X_train[numeric].std()

cylinders         1.751972
displacement    108.665214
horsepower       40.089363
weight          881.504450
acceleration      2.756004
dtype: float64

The distance adds up squared differences in the original units. Two cars that differ by one standard deviation in every column differ by 881 lbs and by 2.8 s, and the formula squares 881 and 2.8 as they are. Between two random training cars, weight makes up 98% of the squared distance. So without the scaler, the nearest neighbors are the cars with the closest weight, and the other columns only break the ties among cars of similar weight. That happens to be a decent model, because weight is the best single column we have.

The scaler gives every column the same vote, with each difference measured in standard deviations. Now a car’s acceleration counts as much as its weight, and acceleration says little about fuel economy. Let’s take its vote away:

without_acceleration = ['cylinders', 'displacement', 'horsepower', 'weight']
fold_rmse(make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10)), X_train[without_acceleration]).mean()

np.float64(3.147081414623216)

3.15, better than any scaled version with all five columns, though still behind the unscaled k=3 (3.12). I tried dropping each of the five columns, and acceleration was the one that helped most, so this number is the best of five tries. I’m keeping all five columns for the rest of the chapter. I only wanted to show how the scaler changes who gets a say in the distance. With it, acceleration counts as much as weight, and nothing in the pipeline checks whether it should. If you use nearest neighbors on your own data, pick the columns before you tune k.

Decision trees

The second model doesn’t measure distances at all. A decision tree asks yes-or-no questions about one column at a time. To pick the first question, it tries every column and every threshold halfway between two neighboring values in the training data. It keeps the split that leaves the smallest sum of squared errors when each side predicts the average MPG of its own cars. Then it does the same on each side, and again, until something stops it. Here is a tree stopped after two questions:

from sklearn.tree import DecisionTreeRegressor, export_text

small_tree = DecisionTreeRegressor(max_depth=2, random_state=42).fit(X_train[numeric], y_train)
print(export_text(small_tree, feature_names=numeric))

|--- weight <= 2764.50
|   |--- weight <= 2219.50
|   |   |--- value: [30.31]
|   |--- weight >  2219.50
|   |   |--- value: [24.26]
|--- weight >  2764.50
|   |--- horsepower <= 127.00
|   |   |--- value: [19.36]
|   |--- horsepower >  127.00
|   |   |--- value: [14.54]

Cars under 2220 lbs get 30.31 mpg, the average of the training cars in that box. Heavy cars with more than 127 hp get 14.54. Every car in a box gets the same prediction, so a 2,300-pound car and a 2,700-pound car both get 24.26. The tree turns weight into bands and picks the edges of the bands itself. (Scikit-learn calls the boxes leaves.)

If you wanted this one in the washing machine, it would take three comparisons and four numbers:

double predict_mpg(double horsepower, double weight) {
    if (weight <= 2764.5) {
        return weight <= 2219.5 ? 30.31 : 24.26;
    }
    return horsepower <= 127 ? 19.36 : 14.54;
}

The thresholds are values of the original columns, which is why a tree doesn’t care about scaling. Weight in pounds, in tons, or standardized puts the same cars on each side of every question, so the tree asks the same questions with different numbers in them. Here is the tree I’ll use for the rest of the chapter (min_samples_leaf comes in a moment), without and with a StandardScaler:

tree = DecisionTreeRegressor(min_samples_leaf=10, random_state=42)
fold_rmse(tree, X_train[numeric]).mean(), fold_rmse(make_pipeline(StandardScaler(), tree), X_train[numeric]).mean()

(np.float64(3.344056346025409), np.float64(3.344056346025409))

Both score 3.344056346025409. The scaler moves every threshold along with the values, so the same cars end up in the same boxes.

A tree that nothing stops keeps splitting until every box holds a single car, or cars with identical specs. The setting most tutorials reach for is max_depth, the number of questions on the way to a box. I prefer min_samples_leaf, the smallest number of training cars a box may hold, because it tells you what stands behind every prediction. With 10, each prediction is the average of at least 10 real cars. A depth limit doesn’t promise that. One branch can end in a box of two cars while the box next to it holds a hundred.

trees = {}
for leaf_size in [1, 2, 5, 10, 20, 40, 80]:
    tree = DecisionTreeRegressor(min_samples_leaf=leaf_size, random_state=42)
    fitted = clone(tree).fit(X_train[numeric], y_train)
    trees[leaf_size] = {
        'leaves': fitted.get_n_leaves(),
        'training RMSE': root_mean_squared_error(y_train, fitted.predict(X_train[numeric])),
        'fold RMSE': fold_rmse(tree, X_train[numeric]).mean(),
    }

pd.DataFrame(trees).T.rename_axis('min_samples_leaf').to_markdown()

With min_samples_leaf=1, the tree grows 254 leaves for 307 cars and memorizes the training data (training RMSE 0.12). On the folds, it scores 3.51. The gap between the training and the fold error is the overfitting from Bending the line, but look at the size of the damage. Degree 4 scored 70 on the folds and sent a Buick to 477 mpg. The memorizing tree loses 0.16 mpg to the best leaf size. Every box predicts the average of real training cars, so whatever a new car looks like, its prediction stays between the worst and the best MPG the tree has seen.

The column doesn’t make a clean U, either. Leaf sizes 2 and 5 do worse than 1, and 10 beats both of its neighbors. Five folds of 28 to 36 cars can’t measure a 0.1-mpg difference reliably, so I read the table as “somewhere between 10 and 20” and take 10. At 80 cars per box, the tree has three leaves and underfits (4.97).

The ceiling

The best the nearest-neighbor model can ever predict is the average of the 10 best cars in the training data, whatever specs you give it. The tree can’t go above the average of its best box. A line has no such limit, for better (it can follow a trend past the end of the data) and for worse (477 mpg).

Does the ceiling cost us anything? The best cars of 1978 beat everything from the years before them, so here is what each model predicts for them on the folds:

knn = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10))
tree = DecisionTreeRegressor(min_samples_leaf=10, random_state=42)

best_1978 = y_train[X_train['model year'] == 78].nlargest(3).index
pd.DataFrame({
    'car name': X_train.loc[best_1978, 'car name'],
    'actual': y_train[best_1978],
    'kNN': y_train[best_1978] - fold_residuals(knn, X_train[numeric])[best_1978],
    'tree': y_train[best_1978] - fold_residuals(tree, X_train[numeric])[best_1978],
    'final pipeline': y_train[best_1978] - fold_residuals(final, X_train)[best_1978],
}).to_markdown()

All three models miss these cars by 5 to 17 mpg, and the line doesn’t use its freedom to go higher. It predicts the Rabbit lower than the other two. All three cars weigh under 2100 lbs, and light cars from the earlier years did around 30 mpg, so every model predicts around 30. What sets the Rabbit apart is its diesel engine, and no column records it (Rule 16).

So on our folds, the ceiling costs nothing we can measure. It starts to cost when the cars you predict keep getting better than anything in the training data, and of these three models, only the line can follow them there.

Here are both models next to the final pipeline:

fold_comparison = pd.DataFrame({
    'kNN': fold_rmse(knn, X_train[numeric]),
    'tree': fold_rmse(tree, X_train[numeric]),
    'final pipeline': fold_rmse(final, X_train),
})
fold_comparison.to_markdown()

The nearest neighbors win the first four years by 0.06 to 0.46 mpg and lose 1979 by 0.57. That’s the shape the tuned Ridge from Regularization and tuning had: here, a lead of 0.13 on average, a swing between the years bigger than the lead, and the loss in the year closest to the cars we have to predict. Rule 19 calls it a tie, and k=10 is the best of six tries on top of that. The tree’s average (3.34) is 0.02 from the pipeline’s, so there isn’t even a lead to check fold by fold. I’m keeping the pipeline.

Here is the average error per year, the lean from What the residuals tell you, for all three:

pd.DataFrame({
    'kNN': fold_residuals(knn, X_train[numeric]),
    'tree': fold_residuals(tree, X_train[numeric]),
    'final pipeline': fold_residuals(final, X_train),
}).groupby(X_train['model year']).mean().to_markdown()

All three lean the same way and by about the same amount, from 0.4-1.5 mpg in 1975 to 3.7-4.0 in 1979. Neither new model knows the year, so, as Rule 16 says, neither can tell a 1979 car from a 1975 car with the same specs.

One more look at the tree before we move on. Here is the first question it asks when fitted on the training years of each fold:

first_questions = {}
for year, (train_rows, _) in zip(range(75, 80), folds):
    fold_tree = clone(tree).fit(X_train[numeric].iloc[train_rows], y_train.iloc[train_rows])
    first_questions[year] = {
        'column': numeric[fold_tree.tree_.feature[0]],
        'threshold': fold_tree.tree_.threshold[0],
    }

pd.DataFrame(first_questions).T.rename_axis('validation year').to_markdown()

Trained on 1970-1974 or on 1970-1975, the tree asks about displacement first. Add the 1976 cars, and it switches to weight. A different first question means different boxes under it, so a car near the edge of a box can get a different prediction because of one extra year of data. That’s the variance from Bending the line. Every box still predicts an average of real cars, so it can’t send anything to 477 mpg, but a car can move from one box to the next.

Rule 20: A model that predicts an average of training targets, like nearest neighbors or a tree, can’t predict outside the range of the targets it was trained on. Before you pick one, check whether the values you will have to predict can lie outside that range.

Both models tie with the pipeline, and the tree has a second problem: one more year of training data changes its first question. The next chapter fits hundreds of trees on slightly different data and averages their answers.