Machine learning for energy data · Lesson 9 of 13

Overfitting and regularisation

An example lesson built by Chalkwork Labs from public sources, to show the shapes. Your course is drafted from your material and checked by you.

A model can look brilliant on the data it learned from and fail on the next day's data. This lesson shows why, on two years of real British electricity demand. You will bend a curve too far, watch it break, and then use one knob to fix it.

1. What overfitting is, in plain words

Imagine a student who prepares for an exam by memorising last year's answer sheet. On last year's questions they score 100%. On this year's questions they fail. They did not learn the subject. They learned the sheet.

A model can do the same. Overfitting means a model has fitted the noise in its training data as if it were a pattern. It then does worse on new data. The opposite is underfitting: a model too simple to follow even the real pattern.

Here is the data for this lesson. Each dot is one working day in 2023 or 2024. Across is the day's mean temperature in central England. Up is Great Britain's mean electricity demand that day.

working days from to . Demand: NESO national demand, daily mean in GW. Temperature: Met Office HadCET daily mean. Weekends and English bank holidays are left out, so that two big non-weather effects stay out of the picture.

The pattern is clear. Cold days need heating, so demand is high on the left. Demand falls as it gets warmer. Above about °C it flattens and may turn up a little, as fans and cooling come on. Only % of days are warmer than that, so the right-hand end is thin.

The question for a model is: given the temperature, what will demand be? Temperature is the input. Demand is the output we want to predict. The model is a curve through the cloud.

No curve passes through every dot. Some scatter is real: school holidays, the economy, a cloudy afternoon. A good model follows the pattern and ignores the scatter. An overfitted model chases the scatter.

2. Two errors, not one

To catch overfitting you need two numbers, not one. First, split the days into two groups. The train/test split puts most days into a training set, which the model learns from, and keeps the rest as a test set, which the model never sees. Here the split is random: training days and test days.

The training error is the model's average miss on the days it learned from.
The test error is its average miss on the days it has never seen.
The difference is the generalisation gap. A big gap is the signature of overfitting.

The error itself is the mean absolute error: take each day's gap between prediction and truth, drop the sign, and average. It is in GW, the same unit as demand. Lower is better.

A straight line through the cloud gives a training error of GW and a test error of GW. A curve with one bend gives GW and GW. Both numbers moved. The test number is the one that matters, because tomorrow is always unseen.

The training error alone tells you nothing. Any model with enough freedom can drive it to zero by memorising. Only the test error measures learning.

3. Be the model: how many bends?

The model here is a polynomial: a curve built from temperature, temperature squared, temperature cubed, and so on. Its degree is the highest power it uses. Degree 1 is a straight line. Degree 2 has one bend. Degree 12 may have up to eleven bends.

The degree is a hyperparameter: a setting you choose before the model learns. The model then learns the best numbers in front of each power. You choose the freedom. The data chooses the shape.

Move the controls. The curve is refitted live, and both errors update.

Degree:

Regularisation strength:

Training days used:

Split

–

training error (GW)

–

test error (GW)

–

gap (test minus train)

Blue dots: training days. Orange dots: test days. The black curve is the fitted model. "Random" holds out 30% of days at random. "2023 to 2024" learns from 2023 and is tested on 2024, a year it has never seen. Fewer training days are taken from the front of a shuffled list, so they are a random sample.

Try these in order:

Degree 1, then 2, then 3, with all training days. The test error falls as the curve gets the bend it needs.
Degree 12, still with all days. The curve wiggles at the ends but the test error barely moves: GW. With days and 12 numbers to learn, there is little room to memorise.
Degree 12 with 40 training days. The curve leaves the chart. The training error is still GW. The test error is GW. That is overfitting: the curve chases 40 dots and has no idea what to do between or beyond them.
Degree 2 with 40 days. Test error GW. A simple curve learns the pattern from a few days. A flexible one needs many.

Overfitting is about freedom relative to data. Twelve bends on 353 days is fine. Twelve bends on 40 days is memorising. The same model is honest or dishonest depending on how much data it has.

4. The validation curve: error against degree

Instead of one degree at a time, plot both errors for every degree from 1 to 12. This is a validation curve: error against a model setting, here the degree. It shows the whole trade in one picture. (A learning curve is a different picture: error against the number of training days. The training-days slider lets you trace that one too.)

Regularisation strength:

Training days used:

Split

–

degree with the lowest test error

–

that test error (GW)

–

training error at that degree (GW)

Blue: training error. Orange: test error. Points above the top of the chart are drawn at the top edge and listed below the chart. The shaded band is the gap.

Read the shape. The training error only ever falls, or stays flat, as the degree rises. More freedom always fits the training days at least as well. The test error falls, bottoms out, and then rises. The rise is overfitting. With all random training days the best degree is and the curve is nearly flat. With 40 days the best degree is low and the right-hand side explodes.

Switch the split to 2023 to 2024. The model learns one year and is tested on the next. The best test error is now GW at degree , against GW under the random split. An unseen year is a harder test than unseen days from the same years. Why? Open question. Candidates: 2024 demand was a little lower overall, and the two years had different cold spells.

5. Regularisation: pay for every bend

Regularisation is any rule that limits a model's freedom so that it cannot fit noise. For a polynomial, the usual rule is a penalty on big coefficients. The model pays a price for every large number in front of a power. The strength slider sets that price. At 0 there is no penalty. At 1,000 the penalty is so heavy that the curve is pulled almost flat.

This is the fix for the 40-day disaster. Degree 12 on 40 training days, at each strength:

Strength	training error (GW)	test error (GW)

At strength the test error is GW, down from GW. The training error went up, from to GW. That is the trade. Regularisation costs training error and buys test error. Push it too far and the curve is too stiff to follow the real pattern: strength 1,000 underfits.

Go back to the chart in section 3. Set degree 12 and 40 days, then raise the strength step by step. Watch the curve calm down.

Regularisation only matters when the model has room to overfit. Try strength 10 with all days at degree 12. The test error hardly moves. There was little noise being fitted, so there was little to fix.

6. A worked example: one cold day

Take the coldest test day under the random split: , mean temperature °C. Real demand that day was GW. Three models predict it:

Model	prediction (GW)	miss (GW)	test error over all test days
Degree 2, all days, no regularisation
Degree 12, 40 days, no regularisation
Degree 12, 40 days, strength 10

The degree-2 model misses by GW, about twice its average miss, but its answer is at least a demand-like number. The overfitted model predicts a number that is not demand at all. The regularised model is sane on average but still far off on this day. Why? The 40 training days run from °C to °C. This day, at °C, is colder than anything the model has seen. No amount of regularisation can teach a model about temperatures that are not in its data. Only more data can.

Quick check

1. A model scores 0.3 GW on its training days and 4.0 GW on test days. What is the most likely explanation?

2. Degree 12 gives a fine test error on 353 training days and a terrible one on 40. What changed?

3. You raise the regularisation strength and the training error goes up. Is that a problem?

7. Read the code

Each block holds real Python that runs in this page. You will not write or edit any of it. Read the steps, press Run, then answer the question under the block. Lines starting with # are notes. Python ignores them.

Block 1: The table and the split

Read the three steps, then press Run. Check that the two counts match section 2.

Look at the table. One row per working day: date, temperature and demand. Each row is already marked as a training or test day.
Split the table into the two groups. The training rows come first, in a shuffled order, so "the first 40" is a random sample.
Print the size of each group.

# The table df: one row per working day in 2023 and 2024 (no weekends, no bank holidays)
df = pd.DataFrame(_days)
print(df.head())

# The honest split: the model learns from "train" and is scored on "test"
train = df[df["set"] == "train"]
test = df[df["set"] == "test"]
print(len(train), "training days,", len(test), "test days")

Block 2: The fit recipe

This is the model in sections 3 and 4, written out. Read the four steps, then press Run. The two errors should match the degree-2 numbers in section 2.

Turn a list of temperatures into columns: temperature, temperature squared, and so on up to the degree. The temperature is first put into standard units, so that the powers stay a sensible size.
Fit: scale each column, add the penalty alpha to the diagonal, and solve for the coefficients. With alpha = 0 this is plain least squares. A bigger alpha pulls every coefficient towards zero.
Predict: build the same columns for new temperatures and combine them with the learned coefficients.
Fit a degree-2 curve on the training days with no penalty, then score it on both groups.

def powers(temp, degree, t_mean, t_sd):
    z = (np.asarray(temp) - t_mean) / t_sd
    return np.column_stack([z ** k for k in range(1, degree + 1)])

def fit(days, degree, alpha):
    x, y = days["temp_c"].to_numpy(), days["demand_gw"].to_numpy()
    X = powers(x, degree, x.mean(), x.std())
    mu, sd = X.mean(axis=0), X.std(axis=0)
    Xs = (X - mu) / sd
    coef = np.linalg.solve(Xs.T @ Xs + alpha * np.eye(degree), Xs.T @ (y - y.mean()))
    return dict(degree=degree, t=(x.mean(), x.std()), mu=mu, sd=sd, coef=coef, b=y.mean())

def predict(m, temp):
    X = (powers(temp, m["degree"], *m["t"]) - m["mu"]) / m["sd"]
    return X @ m["coef"] + m["b"]

model = fit(train, 2, 0)
print("train error:", round(mae(predict(model, train["temp_c"]), train["demand_gw"]), 2), "GW")
print("test error:", round(mae(predict(model, test["temp_c"]), test["demand_gw"]), 2), "GW")

The test error (1.52 GW) is a little higher than the training error (1.45 GW). What does that tell you?

Block 3: Try values. Degree, strength and data

The controls set the three numbers at the top of the code. Start as it is: degree 12 on 40 days with no penalty. Then raise the strength to 1, 10 and 100. Then set the days to 353 and the strength back to 0.

Degree: Strength (alpha): Training days:

degree = {{degree}}    # how many powers of temperature the curve may use
alpha = {{alpha}}      # the penalty on big coefficients; 0 means none
days = {{days}}        # how many training days the model sees

few = train.iloc[:days]   # the first rows of the shuffled training days
model = fit(few, degree, alpha)
print("degree", degree, "alpha", alpha, "days", days)
print("train error:", round(mae(predict(model, few["temp_c"]), few["demand_gw"]), 2), "GW")
print("test error:", round(mae(predict(model, test["temp_c"]), test["demand_gw"]), 2), "GW")

At strength 10 the training error rose from 1.03 to 1.71 GW and the test error fell from 669 to 3.21 GW. Which statement is right?

Spot the mistake

Someone picks the degree like this. It runs, and the number looks good. Click the line whose idea is wrong.

best = None
for degree in range(1, 13):
    model = fit(train, degree, 0)
    err = mae(predict(model, test["temp_c"]), test["demand_gw"])
    if best is None or err < best[1]:
        best = (degree, err)
print("best degree", best[0], "with test error", round(best[1], 2), "GW")

Starting with nothing chosen is fine.
Trying every degree from 1 to 12 is a sensible search. The problem is how each one is judged.
Fitting on the training days is right.
Each degree is judged on the test days. So the degree was chosen by peeking at the test set twelve times, and the final 1.36 GW is the best of twelve peeks, not an honest estimate. The honest way: hold back a slice of the training days, pick the degree on that slice, then score the test days once. Done that way here, the slice picks degree 8 and the one honest test score is 1.38 GW.
Keeping the lowest error is the right way to pick a winner. The data it is measured on is the problem.
Storing the winner is fine.
Printing the result is fine. The number itself is flattered.

What to remember

Overfitting is fitting the noise. The model looks good on the days it learned from and bad on new days.
Always measure two errors. The training error alone can be driven to zero. Only the test error measures learning.
The gap between them is the warning sign.
Freedom is only a problem relative to data. Twelve bends on 353 days is fine. Twelve bends on 40 days is memorising.
Regularisation charges the model for every bend. It raises the training error and, when there was noise being fitted, lowers the test error.
Never choose a setting by looking at the test set. Use a slice of the training data, then score the test set once.
No model can know about conditions that are not in its data. A cold day colder than any training day will be missed.

Sources

NESO, Historic Demand Data, 2023 and 2024 files, column ND (national demand, MW, half-hourly). NESO Open Data Licence, which is the Open Government Licence v3.0 with NESO amendments.
Met Office Hadley Centre, Central England Temperature (HadCET), daily mean series. Open Government Licence. Please acknowledge the source if you reuse the data.
Processing: weekdays only, with the 17 English bank holidays of 2023 and 2024 removed (GOV.UK list); demand averaged over each day's 48 half-hours and shown in GW; days joined to temperature by date. The script that builds every number on this page is export_01.py in the course folder.

← Course: Machine learning for energy data More example lessons →