Forecast · Methodology · Skill Audit

Forecast engine skill audit.

The weather forecast this site serves, scored every night against the station on the beach: 26 settled days at leads of 0 to 6 days, each set beside persistence and climatology. As of 2026-10-18. Surf calls have too few rated days to score yet.

The served forecast, scored every night.

As of 2026-10-18, with station data through 2026-10-17. What is scored is the 7-day weather outlook this site showed readers, saved each hour the page was opened (from 2026-09-20). For each day and lead the earliest outlook issued that day is used; for lead 0 (the same day) it must be issued by 6 am, before the day's high. Each outlook is compared with what the on-site station recorded on 26 settled days, 2026-09-21 to 2026-10-17. The site's outlook runs 7 days, leads 0 to 6; there is no lead-7 forecast to score.

Two baselines that use no weather model are scored on the same days. Persistence forecasts the value the station recorded on the last full day before the outlook was issued. Climatology forecasts the station's average for that time of year (a 15-day window from other years of its record since November 2023; for rain, the median). Skill is 1 minus the forecast's error divided by the baseline's error: above zero the forecast beats the baseline, at zero it ties, below zero the baseline is better. The small range under each skill value is a 95% bootstrap interval over days; green where the whole range is above zero, red where it is below.

Where it beats the baselines, and where it does not

  • High temperature: beats persistence at every lead, with the whole 95% interval above zero at 1, 3 and 5 days; does not beat climatology at 2 to 6 days.
  • Low temperature: beats persistence at every lead, with the whole 95% interval above zero at 2 and 4 days; does not beat climatology at 3 days.
  • Peak gust: does not beat persistence at 0 to 1, 3 and 6 days; does not beat climatology at any lead.
  • Rain amount: beats persistence at every lead, with the whole 95% interval above zero at 3 to 5 days; does not beat climatology at 2 days; with the whole 95% interval above zero at 4 days.
  • Rain day (yes/no at 1 mm): the forecast's hit rate is above all three baselines at 0 to 5 days, and ties or trails the best baseline at 6 days.

High temperature (°F, mean absolute error)

Lead (days)DaysForecastBiasPersistenceSkill vs persistenceClimatologySkill vs climatology
0262.31-0.92.56+0.09
-0.27 to +0.29
2.80+0.17
-0.40 to +0.49
1262.63-1.13.25+0.20
+0.02 to +0.30
2.80+0.06
-0.60 to +0.40
2253.60-1.64.14+0.12
-0.16 to +0.27
2.86-0.26
-1.09 to +0.17
3243.350.54.18+0.26
+0.11 to +0.39
2.89-0.16
-0.67 to +0.17
4233.570.54.69+0.21
-0.07 to +0.36
2.85-0.25
-0.88 to +0.09
5223.420.54.72+0.25
+0.05 to +0.39
2.86-0.19
-0.86 to +0.16
6213.520.74.49+0.18
-0.06 to +0.34
2.87-0.23
-0.80 to +0.11

Low temperature (°F, mean absolute error)

Lead (days)DaysForecastBiasPersistenceSkill vs persistenceClimatologySkill vs climatology
0260.770.10.89+0.11
-0.11 to +0.25
0.89+0.14
-0.25 to +0.40
1260.730.00.92+0.18
-0.01 to +0.34
0.89+0.18
-0.15 to +0.40
2250.80-0.01.10+0.25
+0.09 to +0.39
0.88+0.09
-0.26 to +0.34
3240.93-0.11.07+0.11
-0.13 to +0.30
0.88-0.06
-0.52 to +0.20
4230.81-0.11.14+0.27
+0.11 to +0.41
0.88+0.08
-0.28 to +0.31
5220.850.11.00+0.17
-0.06 to +0.32
0.90+0.06
-0.41 to +0.35
6210.760.00.88+0.10
-0.29 to +0.35
0.93+0.19
-0.15 to +0.40

Peak gust (kt, mean absolute error)

Lead (days)DaysForecastBiasPersistenceSkill vs persistenceClimatologySkill vs climatology
0263.282.52.84-0.20
-1.08 to +0.30
2.05-0.60
-1.43 to -0.02
1263.662.23.08-0.15
-0.66 to +0.22
2.05-0.79
-1.45 to -0.29
2252.511.13.40+0.27
-0.01 to +0.48
2.02-0.24
-0.63 to +0.01
3243.482.03.53+0.00
-0.76 to +0.52
2.09-0.67
-1.82 to +0.07
4232.240.72.96+0.25
-0.13 to +0.48
1.90-0.18
-0.75 to +0.19
5222.491.03.03+0.15
-0.27 to +0.50
1.64-0.52
-1.45 to -0.02
6213.001.72.64-0.18
-1.35 to +0.43
1.68-0.78
-1.86 to -0.02

Rain amount (mm, mean absolute error)

Lead (days)DaysForecastBiasPersistenceSkill vs persistenceClimatologySkill vs climatology
0246.431.98.25+0.20
-0.75 to +0.57
8.11+0.21
-0.84 to +0.56
1237.572.412.81+0.42
-0.17 to +0.71
8.45+0.10
-0.98 to +0.51
22211.216.111.70+0.07
-1.25 to +0.64
8.69-0.29
-2.15 to +0.36
3216.10-1.313.87+0.62
+0.25 to +0.82
9.04+0.33
-0.26 to +0.60
4206.44-1.615.46+0.60
+0.35 to +0.79
9.43+0.32
+0.01 to +0.50
5195.491.116.42+0.66
+0.40 to +0.84
9.84+0.44
-0.26 to +0.73
6188.600.716.35+0.45
-0.02 to +0.68
10.36+0.17
-0.65 to +0.48

Rain day (yes/no at 1 mm, days called right)

Lead (days)Wet daysForecastAlways dryPersistenceClimatology
01517 of 249 of 2416 of 2315 of 24
11416 of 239 of 2311 of 2214 of 23
21314 of 229 of 2210 of 2113 of 22
31315 of 218 of 2111 of 2013 of 21
41317 of 207 of 2010 of 1913 of 20
51315 of 196 of 198 of 1813 of 19
61212 of 186 of 189 of 1712 of 18

The raw model, for comparison.

Before the site picked its sources variable by variable, it served the Open-Meteo default blend. That raw forecast, the archived same-day run for the village grid point, is still scored every night as a comparator: 98 settled days, 2026-07-07 to 2026-10-16, against the same two baselines. Persistence here uses the previous day in this ledger.

VariableDaysRaw model MAEPersistenceSkill vs persistenceClimatologySkill vs climatology
High temperature (°F)902.461.95-0.25
-0.62 to +0.01
2.34-0.05
-0.31 to +0.15
Low temperature (°F)901.571.45-0.09
-0.35 to +0.13
1.43-0.10
-0.29 to +0.06
Rain amount (mm)984.885.54+0.13
-0.21 to +0.35
4.28-0.14
-0.63 to +0.15

Rain day (yes/no at 1 mm): the raw model called 66 of 98 days right; always-dry 62 of 98, persistence 61 of 94, climatology 58 of 98.

Days with thin station coverage are left out (4 of 102 ledger rows: 2026-07-14, 2026-07-15, 2026-07-17, 2026-10-06); temperatures start 2026-07-18. The raw model's grid point moved on 2026-09-14 (from 17.5897°, -101.4317° to 17.8375°, -101.7606°).

Five worst raw-model rain misses

DateStation (mm)Forecast (mm)Error (mm)
2026-07-2939.12.536.6
2026-09-293.731.928.2
2026-10-0950.924.626.3
2026-08-1602222
2026-09-3024.841.917.1

Surf calls, owner-rated.

The surf engine publishes a 1-to-10 quality call. The table sets every day the owner rated or described against the engine's call for that day, taken only from the words logged at the time. Days with no logged engine score show a dash; nothing is filled in after the fact. 4 days have both a numeric owner rating and an engine score. That is too few to say whether the surf call beats guessing, so no surf accuracy score is given.

DateOwner's wordsOwner ratingEngine callOut of sample
2026-05-31
morning
Massive but fat.3/102/10no
2026-06-01
day
Mon was unsurfable.unrideable4/10no
2026-06-02
day
Tue was unsurfable.unrideable9/10no
2026-06-03
day
30-min glassy window between 8-9am.5/105/10no
2026-06-03
forecast-only
finally looks surfable today.surfable5/10no
2026-06-07
day
“pretty good and held throughout the day.”whole-wave—no
2026-06-08
day
“today is wild, unsurfable and messy stormy conditions.”unrideable—no
2026-09-15
day
“last tuesday and wednesday were great. tuesday the wind held all day and the shape was perfect.”whole-wave, perfect—no
2026-09-16
day
“great”whole-wave—no
2026-09-19
day
“saturday was very good too.”whole-wave, very good—no
2026-09-21
morning
“looking pretty wonky out there right now. glassy and wonky.”wonky4/10no
2026-09-22
late morning
“still looking pretty glassy right now and is looking more lined up.”5/101/10no
2026-09-26
morning
“the waves this morning were really good. clean, good shape”7.5/103/10no
2026-10-12
afternoon
“its looking cleaner now. not great shape but cleaner”cleaner, not great shape—yes
2026-10-12
midday
“right now it looks big and messy”3/10—yes
2026-10-12
morning
“this morning looked glassy but a lot of water”8/10—yes
2026-10-14
late morning
“the wind picked up at 10am today and made the waves choppy and lots of current”choppy, lots of current—yes
2026-10-15
morning
“today was really good but degraded a bit at 11:15am when the wind came up”really good—yes
2026-10-17
early morning (~07:00)
“right now its sunny out. the waves are big and powerful but nothing crazy”big and powerful but nothing crazy—yes
What these numbers can and cannot say

One station, one season, few days. The served-forecast tables cover 26 days of one rainy season; a single storm day moves the rain numbers a lot, and most intervals are wide enough to include zero. The sources the site serves were chosen on station data from before 2026-09-20, so these days are a fresh test of that choice. The surf table is rated by one person, and the engine was re-tuned on 2026-09-22 and 2026-09-26 after seeing those ratings, so rows up to 2026-09-26 are in-sample. Rebuilt every night by scripts/score_served_forecasts.py and scripts/build_skill_scoreboard.py.

The June surf audit: too few days to judge.

In June 2026 the surf engine's 1-to-10 call could be checked against a logged surf verdict on only 3 days. Three days cannot say whether the surf call beats guessing, so no accuracy score is shown for it. Since 2026-09-27 the site stores every surf call it serves, by lead, so each day the owner rates from now on can be scored against the call that was on the page. A surf score will be published once at least 20 such days exist.

Breaking down each miss.

Three misses traceable through the engine:

DateEngine QActual conditionGrade errorAttributed ruleStatus
2026-05-31 2 shoulder-only (surfable) +5 vs ideal Engine under-scored: mixed-trains penalty (std=1.17s) fired correctly for the chaotic day-level water, but the shoulder of the wave was still rideable. The engine has no "shoulder-rideable" sub-call — it scores the whole day, not the shoulder of the break.
No snapshot — heuristic attribution
2026-06-02 9 unrideable −6 vs ideal Wave-power penalty set too leniently. Wave power was around 30 kW/m (roughly equivalent to an 8–9 ft swell at 14+ seconds — beyond what Saladita can handle cleanly). The engine's penalty only triggered at ≥25 kW/m (−2 points), which was not enough to cancel the bonuses awarded for a favorable swell direction (+3 for 200°) and long period (+1.5 for 14s+). The threshold needed to be lower (≤20 kW/m to trigger a −3 penalty) to flag that the size-plus-period combination produces walling closeouts at Saladita, not peeling rides.
Not fixed — the wave-power penalty was retired on 2026-09-27 under the size-neutral rule; this day would score high again. Open.
2026-06-08 — unrideable (blown, stormy) — No storm proximity feature existed. Tropical Storm Boris (~270 km southeast of Saladita) created onshore chaotic conditions. Standard weather apps flagged the storm; the engine had no National Hurricane Center advisory feed at all. This is a feature gap — not a miscalibrated threshold, but a completely missing input. No engine snapshot exists for this date; the miss was reported by the user and confirmed by the storm track record.
Fixed 2026-06-08

The two large misses had different causes. On June 2 a rule existed with the wrong threshold; on June 8 no storm input existed at all. The storm input was added on 2026-06-08. The wave-power penalty behind the June 2 miss was retired on 2026-09-27, so that day would still score high today.