All the rules of this book in one place, in the order in which they appeared. Each number links to the paragraph where the rule is stated, with the reasons around it.
The regression problem we are going to (ab)use
- Rule 1: Never trust the data. All values are considered wrong until proven correct.
- Rule 2: If you see one impossible value, it’s never the only wrong value in the dataset. It’s just the only one you found. Assume the entire process that produces the data may be wrong (from data collection, through preprocessing, to storage).
- Rule 3: A check that passes rules out only the errors you thought to look for.
- Rule 4: Data that looks plausible still needs checking. When the data looks the way you expected, go find a source somebody else produced and compare against it.
- Rule 5: Turn what you know about the business into a prediction specific enough to fail, then test it against the column. A prediction that survives is weak evidence. One that fails hands you something you didn’t know.
- Rule 6: Normalize before you read a trend. If two things changed at once, the raw line cannot tell you which one you are looking at.
- Rule 7: Check n before you trust a group statistic. A number computed from two rows is not a smaller version of the same number computed from two hundred.
How to tell if you are right or wrong
- Rule 8: Two rows that differ as text can still be the same thing. Deciding which rows are duplicates takes domain knowledge, which a string comparison doesn’t have.
- Rule 9: The split is a modeling decision. Choose it on purpose, and derive it from the question you are answering.
- Rule 10: You can’t judge a metric without a baseline. Put a dummy model under it and the noise floor over it before you call it good.
- Rule 11: An average error is an average over cases you would treat differently. It tells you the model is wrong, never where and how much the mistake costs you.
- Rule 12: Always verify whether the model’s feature importance doesn’t contradict the business knowledge. Sometimes it will detect a data preparation problem. Often, you will learn you didn’t know something about the problem you were trying to solve.
- Rule 13: Watching the inputs can only catch the inputs moving. If the relationship itself moved, nothing in X will tell you.
- Rule 14: Every look at the test set spends a little of it. Decide in advance how many looks you get.
- Rule 15: A coefficient belongs to the whole set of columns it was fitted with. If the set or the rows change, correlated columns can trade their effects until the signs flip.
- Rule 16: Feature engineering can only rearrange the information you already have. When your worst misses share something your data doesn’t record, or records too rarely to learn from, stop transforming and go collect it.
- Rule 17: A least-squares fit with an intercept makes its training errors average out to 0, in the units it was fitted in, however wrong the model is. Measure the bias only on rows the model didn’t learn from.
- Rule 18: When the training error keeps falling and the validation error starts rising, the model has started fitting the noise in the training rows. Stop adding flexibility where the validation error stops improving.
- Rule 19: The best score of a search is the best of many tries, so it flatters the winner. Compare the winner with the model you already have, fold by fold, and if its lead ever changes between two neighboring folds by more than its average size, call it a tie. The tie goes to the model you can explain.
- Rule 20: A model that predicts an average of training targets, like nearest neighbors or a tree, can’t predict outside the range of the targets it was trained on. Before you pick one, check whether the values you will have to predict can lie outside that range.
- Rule 21: A decision you made because of one model’s weakness belongs to that model. When you change the model, check whether the decision still holds.
- Rule 22: A correlation computed across mixed groups can point the opposite way from the correlation inside every one of them. Before you trust a correlation, color the scatter plot by each group you know about.
- Rule 23: Find out why a value is missing before you fill it. The cause tells you whether the other columns can stand in for it, and whether the hole itself says something about the target.
- Rule 24: Before you give up on the real value, look for it in the data you already have.
Is it right? Classification metrics
- Rule 25: A percentage is a count divided by a count. Know both before you compare two percentages.
The first classifiers and their mistakes
- Rule 26: When different models get the same rows wrong, stop changing the model and open those rows.
- Rule 27: Resample only the rows the model learns from. The rows you score on stay real and keep their real class shares.
- Rule 28: The cheapest answer to an imbalanced problem is to keep the model and choose the threshold on validation data, by what each kind of mistake costs.
- Rule 29: Choosing columns is part of training. A choice made with the validation rows’ answers can’t be checked on those rows.
- Rule 30: Before you search, measure how far the score moves when nothing changes. A search can’t show you a gain smaller than that.
Explainability: Why did the model say that?
- Rule 31: An explanation of one answer compares that row with other rows. Before you read the numbers, find out which rows.
Clustering: groups nobody named
- Rule 32: A clustering algorithm finds groups in any table, random numbers included. Before you name the groups, check that they come back when the rows change, and that they come back more often than the groups of a random table with the same correlations.