Step 4 Model Options Guide

Application: Local Regression Studio
Version: 1.0.11
Purpose: Explain the model choices and hyperparameters shown in Step 4: Select a model and tune hyperparameters.

Contents

1. How Step 4 fits into the workflow
2. Choosing a model
3. Tuning methods
4. Linear regression
5. Ridge regression
6. Decision-tree regression
7. Random-forest regression
8. Gaussian-process regression
9. Artificial neural-network regression
10. Common tuning advice
11. Performance and safety limits
12. Glossary

1. How Step 4 fits into the workflow

Step 4 selects the model and its hyperparameters. The model is trained later in Step 5.

The app uses the three datasets as follows:

The test dataset is not used to select hyperparameters.

Preprocessing is fitted using the training data only. The same fitted preprocessing is then applied to validation, test, and prediction data.

Target transformation is selected in Step 2. Model settings in Step 4 operate on the target scale used during model fitting. Results and prediction intervals are converted back to the original target scale where supported.

2. Choosing a model

Quick comparison

Suggested starting point

For a first analysis:

1. Train linear regression as a baseline.
2. Try ridge regression if features are correlated or numerous.
3. Try random forest for nonlinear behaviour.
4. Use decision trees when interpretability of rules is important.
5. Use Gaussian processes when uncertainty is important and the dataset is small enough.
6. Use an ANN when the relationship is complex and there is enough training data.

No single model is always best. Compare validation performance, test performance, residual diagnostics, and uncertainty calibration.

3. Tuning methods

Manual configuration

The app trains one model using the values entered by the user.

Use manual configuration when:

• You already know reasonable values.
• You want a quick fit.
• The model has few hyperparameters.
• You are investigating the effect of one setting.

Grid search

For each tunable numeric parameter, specify:

• Minimum
• Maximum
• Number of grid points
• Linear or logarithmic spacing

The app evaluates combinations formed from the generated values.

Example:

Minimum: 0.001
Maximum: 100
Grid points: 6
Spacing: Logarithmic

This creates six values spread across several orders of magnitude.

Grid search is useful for small searches, but the number of combinations grows rapidly:

total candidates =
points on axis 1 × points on axis 2 × ...

The app limits excessively large searches.

Ordinary random search

The app independently samples values from each declared range.

Advantages:

• Usually more efficient than a large grid when many parameters are present.
• Can explore values that do not lie on fixed grid lines.

Limitation:

• Independent random samples can cluster in some areas and leave other regions poorly covered.

Latin hypercube sampling

Latin hypercube sampling, or LHS, divides each continuous parameter range into intervals and samples once from each interval before combining the dimensions.

Advantages:

• More even coverage of each parameter range than ordinary random sampling.
• Often useful when the number of trials is limited.
• Particularly suitable for ANN and GP tuning.

For integer parameters, sampled values are rounded. Duplicate configurations may be removed.

Number of search samples

This is the number of candidate configurations for random search or LHS.

More samples provide broader exploration but require more model fits.

Recommended initial values:

For ANN tuning, remember that every candidate may run for many epochs. Early stopping can reduce the actual workload.

Linear versus logarithmic spacing

Use linear spacing when equal absolute changes are meaningful.

Example:

Tree depth: 3, 6, 9, 12, 15

Use logarithmic spacing when the useful scale spans several powers of ten.

Examples:

Ridge strength: 0.001 to 100
Learning rate: 0.0001 to 0.01
GP noise: 0.001 to 1

All values must be positive for logarithmic spacing.

4. Linear regression

What it does

Linear regression predicts the target as a weighted sum of the processed input features:

prediction =
intercept
+ coefficient 1 × feature 1
+ coefficient 2 × feature 2
+ ...

One-hot encoded categorical columns can also be included.

Step 4 options

Linear regression has no model-specific hyperparameters in this version.

When to use it

Use linear regression when:

• A roughly linear relationship is plausible.
• Interpretability matters.
• You want a fast baseline.
• The dataset is not extremely high-dimensional.
• Analytical prediction intervals are useful.

Strengths

• Very fast.
• Coefficients can be interpreted.
• Provides analytical confidence and prediction intervals.
• Useful for detecting whether more complex models are necessary.

Limitations

• Does not naturally capture curves, thresholds, or complex interactions.
• Sensitive to strong multicollinearity.
• Outliers can have a large effect.
• Analytical uncertainty assumes the linear model and error assumptions are reasonably appropriate.

Related preprocessing advice

• Standardisation is optional for prediction quality, but can improve numerical conditioning.
• Removing one reference level from each one-hot encoded categorical feature is recommended.
• Inspect residuals for curvature and changing variance.

5. Ridge regression

What it does

Ridge regression is linear regression with a penalty that shrinks coefficients toward zero.

It is useful when:

• Features are correlated.
• There are many processed input columns.
• Ordinary linear regression coefficients are unstable.
• One-hot encoding creates a wide feature matrix.

Ridge strength — lambda

Interface label: Ridge strength (lambda)

Lambda controls the amount of coefficient shrinkage.

If lambda is too small

• The model may behave like unstable ordinary linear regression.
• Correlated features can receive large compensating coefficients.
• Validation error may be higher.

If lambda is too large

• The model can underfit.
• Predictions may be pulled too close to the mean.
• Important relationships can be suppressed.

Suggested starting values

When numeric features are standardised:

Manual starting value: 1
Typical tuning range: 0.001 to 100
Recommended spacing: logarithmic

The best scale depends strongly on preprocessing and the number of features.

Uncertainty

The current app uses approximate bootstrap intervals for ridge regression.

These are not identical to exact analytical intervals and should be evaluated using:

• Test-set coverage
• Mean interval width
• Coverage error
• Interval score

6. Decision-tree regression

What it does

A decision tree repeatedly divides observations using rules such as:

feature A ≤ threshold

Each final region, called a leaf, returns a prediction based on training targets in that leaf.

Trees can represent nonlinear relationships and feature interactions.

Maximum depth

Interface label: Maximum depth

Maximum depth limits how many split levels the tree can create.

Suggested starting value:

8

Typical tuning range:

3 to 15

Watch the training and validation loss versus depth plot. If training loss falls but validation loss rises, the tree is becoming too complex.

Minimum rows per leaf

Interface label: Minimum rows per leaf

This is the minimum number of training rows allowed in a terminal leaf.

Suggested starting value:

5

Typical tuning range:

2 to 30

For noisy or small datasets, larger leaves are often safer.

Candidate thresholds per feature

Interface label: Candidate thresholds per feature

For each considered feature, the app does not necessarily test every possible split point. This setting limits the approximate number of candidate thresholds examined.

Suggested starting value:

24

Typical practical range:

12 to 48

Values above the default may help when a feature has a complex threshold structure, but they increase fitting time.

Features considered per split

Interface label: Features considered per split

Options:

All

Every processed feature can be evaluated at each split.

Recommended for:

• A single decision tree
• Small and medium feature counts
• Maximum deterministic split search

Square root

Approximately the square root of the processed features is randomly considered at each split.

Example:

100 processed features → about 10 considered per split

This adds randomness and is more commonly useful in forests.

Log2

Approximately log2(number of features) + 1 features are considered per split.

This is usually smaller than square root when the feature count is large.

Uncertainty

The app can calculate an approximate leaf-based interval using target variation among training observations in the same terminal leaf.

Limitations:

• Leaves with few rows provide unreliable uncertainty estimates.
• A single tree does not naturally provide a full predictive distribution.
• The interval should be treated as approximate.

7. Random-forest regression

What it does

A random forest averages predictions from many decision trees. Each tree receives a sampled set of rows and may consider a random subset of features at each split.

Averaging reduces the instability of a single tree.

Number of trees

Interface label: Number of trees

Suggested starting value:

40

Typical tuning range:

10 to 100

Increase the number of trees until validation error and interval behaviour stabilise. The training-history plot shows error versus the number of trees.

Maximum depth

Same basic meaning as for a decision tree.

Suggested starting value:

8

Deeper trees increase flexibility and computation. Random forests tolerate deeper trees better than single trees, but very deep trees can still overfit or produce unstable interval estimates.

Minimum rows per leaf

Same basic meaning as for a decision tree.

Suggested starting value:

5

Larger leaves often improve smoothness and interval stability.

Row sampling fraction

Interface label: Row sampling fraction

This controls the fraction of training rows sampled for each tree.

Examples:

Suggested starting value:

0.8

Lower values:

• Increase diversity between trees.
• Reduce rows available to each tree.
• May help reduce correlation but can increase individual-tree error.

Higher values:

• Give each tree more information.
• Can reduce diversity.
• Increase work per tree.

Candidate thresholds per feature

Same meaning as for a decision tree.

Suggested starting value:

20

Features considered per split

All

Every processed feature is considered.

Square root

Approximately the square root of the feature count is considered.

This is the default for the forest in v0.2.1 and generally encourages useful diversity.

Log2

A smaller random subset is considered.

This can be useful for very wide datasets or when stronger tree diversity is desired.

Uncertainty

The app uses variation among individual tree predictions to form an approximate ensemble interval.

Important limitations:

• Between-tree variation is not automatically a calibrated prediction interval.
• Trees are correlated, so their spread may underestimate uncertainty.
• Test-set coverage should always be checked.
• Increasing the number of trees stabilises the estimate but does not guarantee correct coverage.

8. Gaussian-process regression

What it does

A Gaussian process, or GP, defines a distribution over possible functions. Predictions include:

• A predictive mean
• A predictive standard deviation
• A native model-based prediction interval

GPs are especially useful when:

• The dataset is relatively small.
• The target varies smoothly with the input features.
• Predictive uncertainty is important.
• Extrapolation risk needs to be visible.

Exact GP computation grows rapidly with the number of training rows. Use representative-subset modes for larger datasets.

GP training mode

Interface label: GP training mode

Exact GP

Uses every available training row after preprocessing.

Advantages:

• Uses all training observations.
• Provides the direct exact-GP calculation for the selected kernel and parameters.

Limitations:

• Memory grows approximately with the square of the number of training rows.
• Matrix factorisation time grows approximately with the cube of the number of rows.
• Model files can be large because prediction requires stored training-related state.

Recommended use:

Small datasets, preferably around 1,000 rows or fewer

The app has a default hard GP-state limit of 2,000 rows.

Representative-subset GP

Uses a selected subset of training rows to construct the GP.

Advantages:

• Reduces memory and computation.
• Allows GP-style modelling on larger datasets.

Limitations:

• It is an approximation.
• Information outside the selected subset may be lost.
• It is not a full sparse variational GP.

Automatically optimise subset size

The app evaluates several subset sizes using validation RMSE.

It then selects:

The smallest subset whose validation RMSE is within the specified tolerance of the best candidate.

This balances predictive performance and computational cost.

Kernel

Interface label: Kernel

The kernel defines how similarity between input rows translates into similarity of predicted target values.

Because the app uses one shared length scale, numeric input scaling is strongly recommended.

RBF / squared exponential

Produces very smooth functions.

Use when:

• The target is expected to change smoothly.
• Sharp corners or abrupt transitions are unlikely.

Characteristics:

• Highly smooth.
• Can over-smooth abrupt behaviour.
• A common default.

Matérn 3/2

Allows rougher functions than RBF.

Use when:

• The relationship is smooth but can change more abruptly.
• RBF appears too smooth.

Matérn 5/2

Intermediate smoothness between RBF and Matérn 3/2.

Use when:

• You want a smooth function with somewhat more local flexibility than RBF.

Rational quadratic

Acts like a mixture of RBF behaviours with different length scales.

Use when:

• The target has variation occurring at several characteristic scales.

It has an additional Rational-quadratic alpha parameter.

Linear kernel

Represents primarily linear covariance behaviour.

Use when:

• A linear relationship is expected.
• You want GP uncertainty around a broadly linear model.

It does not provide the same nonlinear flexibility as RBF or Matérn kernels.

Length scale

Interface label: Length scale

The length scale controls how far input points can move before their predictions become substantially different.

If too small

• The GP may follow local noise.
• Predictions may change sharply between nearby observations.
• Uncertainty can vary rapidly.
• The covariance matrix may become difficult to condition.

If too large

• The model may underfit.
• Important local structure can disappear.
• Predictions may become nearly constant or overly smooth.

Suggested initial value when processed numeric features are standardised:

1

Typical tuning range:

0.1 to 10

Recommended spacing:

Logarithmic

Shared length scale

Version 0.2.1 uses one shared length scale for all processed features. It does not yet fit a separate automatic-relevance-determination length scale for every feature.

Signal variance

Interface label: Signal variance

Signal variance controls the typical magnitude of function variation attributed to the underlying GP signal.

In the kernel calculation, it acts as the covariance amplitude.

If too small

• Predictions can be pulled too strongly toward the training-target mean.
• The model may underfit genuine variation.

If too large

• The model may attribute too much variation to the underlying function.
• It may interact with the noise setting and length scale in unstable ways.

Suggested starting value:

1

Typical tuning range:

0.1 to 10

Recommended spacing:

Logarithmic

Its appropriate scale depends on the transformed target values used during fitting.

Observation-noise standard deviation

Interface label: Observation-noise standard deviation

This describes the expected random variation in an observed target around the smooth underlying GP function.

Conceptually:

observed target = underlying function + random observation noise

Examples of possible noise sources:

• Measurement error
• Natural variation not represented by the selected features
• Unrecorded operating conditions
• Repeated measurements that differ under nominally identical inputs

Units

The value is expressed in the units of the target scale used for model fitting.

Examples:

• If the target is fitted in degrees Celsius, the value is in degrees Celsius.
• If a natural-log target transformation is used, it is on the log-target scale.
• It is not automatically a percentage.

Effect of the value

If too small

• The model may overfit noisy observations.
• Matrix factorisation may be less stable.
• Prediction intervals may be unrealistically narrow.

If too large

• The model may underfit.
• Real structure may be treated as noise.
• Prediction intervals may be unnecessarily wide.

Default:

0.1

Typical tuning range in the current interface:

0.001 to 1

Recommended spacing:

Logarithmic

Important scaling note

The fixed default is not equally meaningful for every target scale. Compare it with the standard deviation and typical measurement error of the training target.

For example:

Target standard deviation = 100
Noise standard deviation = 0.1

would imply almost noise-free observations.

A reasonable initial estimate may come from:

• Known instrument precision
• Repeat-measurement variability
• Domain expertise
• A small fraction of the training-target standard deviation

Let validation performance and uncertainty coverage guide tuning.

Observation noise versus numerical jitter

These are not the same.

Do not use jitter as a substitute for real observation noise.

Numerical jitter

Interface label: Numerical jitter

Jitter is a very small positive number added to the diagonal of the GP covariance matrix.

Purpose:

• Improve numerical stability.
• Help Cholesky factorisation when rows are duplicated or nearly duplicated.
• Address rounding and conditioning problems.

Default:

0.00000001

The app automatically increases jitter by factors of ten if factorisation fails, up to a limited number of attempts.

When to increase it manually

Increase jitter if you see an error similar to:

Covariance matrix is not positive definite

Possible causes include:

• Duplicate or nearly duplicate processed input rows
• Extremely small noise
• Extreme kernel parameters
• Poor feature scaling

Increase jitter gradually, for example:

1e-8 → 1e-7 → 1e-6

Large jitter can alter predictions, so it should remain much smaller than the main covariance and noise scales.

Stochastic kernel-search iterations

Interface label: Stochastic kernel-search iterations

The app starts from the entered length scale, signal variance, and noise standard deviation, then proposes multiplicative changes and keeps proposals that reduce the negative log marginal likelihood.

Default:

8

Maximum in the interface:

30

This is a lightweight stochastic local search, not a guarantee of finding the global optimum.

How the internal GP optimiser works

The GP does not use Adam, stochastic gradient descent, L-BFGS or another gradient-based optimiser. It uses a lightweight stochastic local search in log-parameter space.

• Lightweight means that it uses a small browser-friendly search procedure without a large numerical-optimisation dependency.
• Stochastic means that candidate parameter changes are generated using seeded random numbers.
• Local search means that it explores values near the current best settings instead of systematically searching the entire allowed range.
• Log-parameter space means that positive parameters are changed multiplicatively. If a parameter is theta, the search changes log(theta) and converts it back with exp. This keeps length scale, signal variance, observation noise and rational-quadratic alpha positive.

For one iteration, the app approximately performs these steps:

1. Start from the current kernel parameters.
2. Propose seeded random multiplicative changes to the parameters relevant to the selected kernel.
3. Fit the GP covariance system using the proposal.
4. Calculate negative log marginal likelihood.
5. Keep the proposal only if it lowers that objective.
6. Reduce the proposal size gradually during later iterations.

Because the procedure does not calculate gradients, it does not use a learning rate. Its closest analogue is the internal proposal scale, which controls the size of random multiplicative changes and is reduced automatically.

This local procedure can improve sensible starting values but can miss a better solution farther away. Grid search, ordinary random search or Latin hypercube sampling can first explore a wider region using validation RMSE; the selected configuration can then receive the final local kernel search.

When external grid, random, or LHS tuning is used, the candidate comparison uses validation RMSE. The final selected GP can then run the configured kernel-optimisation iterations.

Rational-quadratic alpha

Interface label: Rational-quadratic alpha

Used only with the rational-quadratic kernel.

It controls how strongly the kernel behaves like a mixture of different length scales.

Broad interpretation:

Default:

1

This setting has no effect for RBF, Matérn, linear, or other kernels.

Representative subset size

Interface label: Representative subset size

Used in representative-subset mode.

This is the number of training rows retained for GP covariance calculations.

Default:

500

Allowed range:

20 to 2,000

A suitable value depends on:

• Number of training rows
• Feature-space complexity
• Number of sources or regimes
• Browser memory
• Required accuracy

Subset selection

Interface label: Subset selection

Farthest-point sampling

Starts with one row and repeatedly chooses the row farthest from the selected set in processed feature space.

Advantages:

• Encourages broad feature-space coverage.
• Deterministic apart from the initial seeded point.
• Often a good default for smooth GP models.

Limitations:

• Can emphasise extreme points.
• More computationally expensive than random sampling.

K-means++ representative sampling

Selects points probabilistically using distance from already selected points.

Advantages:

• Encourages broad coverage.
• Less focused exclusively on the most extreme point than pure farthest-point sampling.

Limitations:

• The current implementation selects representative rows using a k-means++-style seeding process; it does not run a full iterative k-means optimisation.

Random sampling

Randomly selects training rows using the configured seed.

Advantages:

• Fast.
• Simple.
• Useful as a baseline.

Limitations:

• May miss rare regimes or sparse feature-space regions.
• Different seeds can produce different results.

Automatic subset minimum

Interface label: Automatic subset minimum

The smallest subset size considered during automatic subset optimisation.

Default:

100

Choose a smaller value only when:

• The dataset is small.
• The relationship is simple.
• Fast fitting is more important than accuracy.

Automatic subset maximum

Interface label: Automatic subset maximum

The largest subset size considered.

Default:

1,000

The value cannot exceed the app’s GP hard limit.

Larger maxima improve the chance of matching exact-GP performance but increase computation and model-file size.

Subset sizes evaluated

Interface label: Subset sizes evaluated

The number of subset-size candidates between the automatic minimum and maximum.

Default:

5

For example:

Minimum: 100
Maximum: 1,000
Count: 5

generates approximately evenly spaced candidate sizes.

More candidates provide a finer comparison but require more GP fits.

Validation tolerance for smaller subset

Interface label: Validation tolerance for smaller subset

After evaluating subset candidates, the app identifies the best validation RMSE. It then selects the smallest subset whose RMSE is within this relative tolerance of the best.

Default:

0.01

This means approximately:

within 1% of the best validation RMSE

Examples:

Increasing tolerance prioritises speed and smaller model files over the very best validation score.

GP tuning ranges

When automatic tuning is enabled, v0.2.1 exposes:

• Length scale
• Signal variance
• Observation-noise standard deviation

Suggested default ranges:

Adjust these ranges when the transformed target scale or processed feature scale differs greatly from the assumptions behind the defaults.

9. Artificial neural-network regression

What it does

The ANN is a feed-forward dense neural network. It sends processed input values through one to three hidden layers and produces one numeric output.

ANNs can represent complex nonlinear relationships and feature interactions.

They usually require:

• More training data than simpler models
• Standardised numeric features
• Careful validation
• Regularisation
• Early stopping

Hidden layer 1 neurons

Interface label: Hidden layer 1 neurons

Number of units in the first hidden layer.

Default:

64

Larger values increase model capacity and trainable parameters.

Hidden layer 2 neurons

Interface label: Hidden layer 2 neurons

Default:

32

Set to 0 to remove the second hidden layer.

Hidden layer 3 neurons

Interface label: Hidden layer 3 neurons

Default:

0

Set above zero to create a third hidden layer.

Architecture guidance

Suggested presets:

Small

32 → 16

Suitable for:

• Small datasets
• Mobile devices
• Quick experiments

Medium

64 → 32

Suitable for:

• General desktop use
• Moderate nonlinear relationships

Larger

128 → 64 → 32

Suitable for:

• Larger datasets
• Complex relationships
• Desktop devices with adequate memory

A larger model is not automatically better. If training loss falls while validation loss rises, the model is overfitting.

Activation function

Interface label: Activation function

ReLU

max(0, input)

Advantages:

• Fast
• Common default
• Usually effective for dense regression networks

Limitations:

• Units can become inactive if their inputs remain negative.

Leaky ReLU

Like ReLU, but retains a small negative slope.

Advantages:

• Reduces the risk of permanently inactive units.
• Often a safe alternative to ReLU.

Tanh

Maps values to approximately -1 to 1.

Advantages:

• Smooth
• Can work well for smaller networks and standardised inputs

Limitations:

• Can saturate and produce small gradients.

Sigmoid

Maps values to approximately 0 to 1.

Limitations:

• Often saturates in hidden layers.
• Usually not the first choice for modern regression networks.

Suggested default:

ReLU

Optimiser

Interface label: Optimiser

Adam

Adaptive optimiser using running estimates of gradient moments.

Advantages:

• Usually a strong default.
• Often converges faster with less manual tuning.

Suggested default learning rate:

0.001

Stochastic gradient descent

Uses direct gradient steps.

Advantages:

• Simpler behaviour.
• Can generalise well with careful tuning.

Limitations:

• More sensitive to the learning rate.
• May require more epochs.
• Can converge slowly.

A smaller learning rate is often needed when training becomes unstable.

Learning rate

Interface label: Learning rate

Controls the size of each weight update.

Default:

0.001

Typical tuning range:

0.0001 to 0.01

Recommended spacing:

Logarithmic

If training loss becomes NaN, increases sharply, or fluctuates severely, reduce the learning rate.

Batch size

Interface label: Batch size

Number of training rows used for one gradient update.

Default:

32

Suggested values:

16, 32, 64, 128

Rules:

• Do not set batch size larger than the available training rows unless you intentionally want full-batch behaviour.
• Smaller devices may require smaller batches.
• Large batches do not necessarily improve validation performance.

Maximum epochs

Interface label: Maximum epochs

An epoch is one full pass through the training dataset.

Default:

150

This is an upper limit. Early stopping can end training sooner.

The Step 6 training-history plot shows:

• Training loss versus epoch
• Validation loss versus epoch
• Best validation epoch

Do not select the final epoch solely because training loss is lowest. The best validation epoch is usually more relevant.

Dropout rate

Interface label: Dropout rate

During training, dropout randomly disables a fraction of hidden activations.

Default:

0.1

Examples:

Benefits:

• Reduces overfitting.
• Supports Monte Carlo dropout uncertainty.

If too high:

• The network may underfit.
• Training can become slow or unstable.
• Predictions may be overly smooth.

Typical tuning range:

0 to 0.4

L2 regularisation

Interface label: L2 regularisation

Adds a penalty for large weights.

Default:

0.0001

Typical tuning range:

0.000001 to 0.01

Recommended spacing:

Logarithmic

L2 and dropout can both be used, but strong values for both may over-regularise the network.

Early-stopping patience

Interface label: Early-stopping patience

Number of consecutive epochs without sufficient validation improvement before training stops.

Default:

20

Smaller values:

• Stop sooner.
• Reduce computation.
• Risk stopping during a temporary plateau.

Larger values:

• Allow more time for improvement.
• Increase computation.
• May continue training after overfitting has begun.

The app restores the weights from the best validation epoch.

Minimum validation improvement

Interface label: Minimum validation improvement

The validation loss must improve by at least this amount to reset the patience counter.

Default:

0.00001

If too small:

• Tiny numerical fluctuations may count as improvement.
• Training may continue longer.

If too large:

• Meaningful small improvements may be ignored.
• Training may stop too early.

The appropriate value depends on the scale of the internally fitted target loss.

ANN uncertainty method

Interface label: ANN uncertainty method

Monte Carlo dropout

The app keeps dropout active during repeated prediction passes.

The distribution of stochastic predictions is combined with residual variation to form an approximate interval.

Advantages:

• Requires one trained network.
• Cheaper than training multiple independent models.

Limitations:

• Approximate.
• Depends on the selected dropout rate.
• May be poorly calibrated.
• A dropout rate of zero provides little model-spread information.

Small deep ensemble

The app trains several independently initialised networks and combines their predictions.

Advantages:

• Often gives more robust variation estimates than one network.
• Captures some sensitivity to initialisation.

Limitations:

• Multiplies training time and model-file size.
• Still approximate.
• Ensemble members can remain highly correlated.

Monte Carlo dropout passes

Interface label: Monte Carlo dropout passes

Number of stochastic predictions used to estimate the interval.

Default:

50

Allowed range:

10 to 300

More passes:

• Stabilise estimated quantiles.
• Increase prediction time.
• Do not fix a poorly calibrated model.

Suggested values:

30–50 for routine use
100 or more for more stable final reporting

Ensemble members

Interface label: Ensemble members

Number of independently initialised networks.

Default:

3

Allowed range:

2 to 7

More members:

• Improve stability of the ensemble distribution.
• Increase training time and model size almost proportionally.

Suggested starting value:

3

ANN parameter count

The app estimates the total number of trainable parameters.

For a dense layer:

parameters =
input units × output units
+ output bias units

Total parameters are summed across all layers.

Current safety guidance:

Recommendations are lower on mobile or memory-constrained devices.

A model with more parameters than training rows is not automatically invalid, but it has a higher overfitting risk and generally needs strong validation and regularisation.

ANN tuning ranges

Version 0.2.1 exposes the following parameters for automatic tuning:

• Hidden layer 1 neurons
• Hidden layer 2 neurons
• Learning rate
• Dropout rate
• L2 regularisation

Default ranges:

Random search or LHS is usually more practical than a full ANN grid.

During candidate tuning:

• The validation set selects the candidate.
• Ensemble size is reduced to one for candidate comparison.
• The final chosen configuration can then use the selected uncertainty method.

Models added in v1.0.11

Elastic-net regression

• Overall penalty (lambda): total coefficient shrinkage. Zero disables the penalty.
• L1 mixing ratio: zero is ridge-like, one is lasso-like, and intermediate values combine the penalties.
• Maximum coordinate-descent iterations: fitting limit.
• Convergence tolerance: maximum coefficient change required to stop.

Tune lambda logarithmically and the L1 ratio linearly. Standardised inputs and a dropped one-hot reference level are recommended. Intervals use an approximate constant residual scale.

Huber robust regression

• Huber threshold: residual size at which observations begin to be downweighted.
• Maximum IRLS iterations: reweighting limit.
• Convergence tolerance: coefficient-stability requirement.
• Numerical ridge stabiliser: small diagonal adjustment; zero is allowed.

A smaller Huber threshold is more aggressive. Inspect the downweighted-row count and residual diagnostics.

Gradient-boosted trees

• Boosting stages: number of sequential trees.
• Boosting learning rate: contribution from each stage. Smaller values generally require more stages.
• Tree depth per stage: interaction complexity.
• Minimum rows per leaf: smoothness and regularisation.
• Row subsampling fraction: stochastic row sampling per stage.
• Candidate thresholds per feature: split-search detail and runtime.

Use the training/validation learning curve to identify overfitting. The interval is an approximate constant residual-scale interval.

k-nearest-neighbour regression

• Number of neighbours: local sample size.
• Weighting: uniform or inverse-distance.
• Distance metric: Manhattan (1) or Euclidean (2) power.
• Maximum stored rows: memory, package-size and prediction-time guardrail.

Standardisation is essential. kNN has no native feature-importance chart in this release. Its empirical neighbour interval is approximate.

Linear quantile regression

• Central quantile: point prediction quantile; 0.5 is the conditional median.
• Iterations per fitted quantile: limit for lower, central and upper fits.
• Optimisation learning rate: initial subgradient step size.
• L2 regularisation: coefficient shrinkage; zero disables it.

The requested interval level determines the final lower and upper quantiles. These are conditional quantile estimates, not a finite-sample coverage guarantee.

For complete guidance, see EXPANDED_MODELLING_GUIDE.md.

10. Common tuning advice

Change one thing at a time when learning

Before starting a large search:

1. Fit the default.
2. Examine training and validation performance.
3. Identify signs of underfitting or overfitting.
4. Narrow the ranges to plausible values.
5. Run random search or LHS.

Signs of underfitting

• Training and validation errors are both high.
• Residual plots show strong structure.
• Predictions vary too little.
• Training loss stops improving at a poor value.

Possible responses:

• Increase tree depth.
• Reduce minimum leaf size.
• Reduce ridge strength.
• Reduce GP length scale.
• Increase ANN size.
• Reduce ANN regularisation.
• Train ANN longer or adjust the learning rate.

Signs of overfitting

• Training error is much lower than validation error.
• Validation loss rises while training loss continues falling.
• Uncertainty intervals have poor test coverage.
• Predictions are unstable across sources or regimes.

Possible responses:

• Increase ridge strength.
• Reduce tree depth.
• Increase minimum leaf size.
• Increase forest diversity.
• Increase GP noise or length scale.
• Reduce ANN size.
• Increase dropout or L2.
• Use earlier stopping.
• Add more representative training data.

Use the validation set, not the test set

Do not repeatedly tune settings after looking at test performance. Doing so makes the test set part of model development and weakens its role as an independent assessment.

Compare uncertainty quality as well as point error

For models with intervals, examine:

• Coverage
• Mean interval width
• Coverage error
• Interval score
• Coverage by source or regime

A very wide interval can achieve high coverage without being useful.

11. Performance and safety limits

Decision tree

Cost increases with:

• Training rows
• Processed feature count
• Maximum depth
• Candidate thresholds per feature

Random forest

Cost increases approximately with:

• Number of trees
• Rows sampled per tree
• Tree depth
• Processed feature count

Gaussian process

Exact GP is the most restrictive model.

Memory is dominated by an approximately n × n covariance matrix.

The app uses a default maximum GP state of:

2,000 rows

Use representative-subset mode for larger training sets.

ANN

Cost increases with:

• Trainable parameter count
• Training rows
• Epochs
• Number of tuning candidates
• Monte Carlo passes
• Ensemble members

Early stopping is enabled to reduce unnecessary epochs.

Browser considerations

A configuration that works well on a desktop may be slow or fail on a mobile device.

Before a large job:

• Close unnecessary browser tabs.
• Use the offline package or a stable hosted version.
• Start with fewer candidates.
• Use a smaller GP subset.
• Use a smaller ANN.
• Confirm the workflow on a small sample first.
• Save useful models and results externally.

12. Glossary

Candidate configuration

One complete combination of hyperparameter values evaluated during tuning.

Epoch

One full pass through the ANN training data.

Hyperparameter

A model setting chosen before or around training, such as tree depth, ridge strength, GP length scale, or ANN learning rate.

Kernel

A GP function that measures similarity between processed input rows.

Latent function

The underlying smooth relationship represented by the GP before observation noise is added.

Negative log marginal likelihood

A GP fitting objective balancing data fit and model complexity. Lower values are better for the same data and model formulation.

Observation noise

Random target variation that is not attributed to the underlying GP function.

Overfitting

Learning training-specific details or noise that do not generalise to validation, test, or new data.

Prediction interval

An interval intended to contain an individual future observation at a requested probability level, under the model and method assumptions.

Regularisation

A constraint or penalty intended to reduce overfitting, such as ridge shrinkage, ANN dropout, or ANN L2.

Representative subset

A selected group of training rows used to approximate a GP that would otherwise use the full training dataset.

Trainable parameter

A numeric weight or bias learned by an ANN.

Validation RMSE

Root mean squared error calculated on the validation dataset and used to compare model settings.

Practical default configurations

Linear baseline

Model: Linear regression
Numeric scaling: Standardise
Drop one categorical reference level: Yes

Ridge baseline

Lambda: 1
Tuning range: 0.001–100, logarithmic

Decision-tree baseline

Maximum depth: 8
Minimum rows per leaf: 5
Candidate thresholds per feature: 24
Features per split: All

Random-forest baseline

Number of trees: 40
Maximum depth: 8
Minimum rows per leaf: 5
Row sampling fraction: 0.8
Candidate thresholds per feature: 20
Features per split: Square root

Exact GP baseline

Mode: Exact GP
Kernel: RBF
Length scale: 1
Signal variance: 1
Observation-noise standard deviation: 0.1
Jitter: 1e-8
Stochastic kernel-search iterations: 8

Review whether 0.1 is sensible relative to the fitted target scale.

Representative GP baseline

Mode: Representative-subset GP
Subset size: 500
Subset method: Farthest-point sampling
Other GP settings: Same as exact GP baseline

ANN baseline

Hidden layers: 64 → 32
Activation: ReLU
Optimiser: Adam
Learning rate: 0.001
Batch size: 32
Maximum epochs: 150
Dropout: 0.1
L2: 0.0001
Patience: 20
Minimum improvement: 0.00001
Uncertainty: Monte Carlo dropout
MC passes: 50

Important interpretation warning

Hyperparameters do not have universally optimal values. Their effects depend on:

• Training-set size
• Feature scaling
• Target transformation
• Noise level
• Number of processed features
• Split strategy
• Sources and regimes represented in each split

Use the defaults as starting points, not guaranteed recommendations. Final choices should be supported by validation performance and confirmed using the untouched test dataset.

v0.3.0 comparison-set note

The Step 5 comparison batch uses fixed manual baseline values for each selected model. It does not copy the currently visible Step 4 hyperparameters to every model family. To include a custom or tuned candidate, configure that model in Step 4 and select Train and evaluate. The result is saved in the same comparison workspace and is marked comparable when its data and evaluation setup matches.

v1.0 deployment and governance

v1.0 preserves the documented modelling behaviour and adds separate deployment editions, versioned governance lifecycle records, monitoring and revalidation, model-change assessments, local recovery, stricter import/export validation, and optional verification of organizational signatures against deployment-pinned public keys. See the package-root DEPLOYMENT_GUIDE.md, GOVERNANCE_GUIDE.md, MONITORING_GUIDE.md, SECURITY.md, and MIGRATION_GUIDE.md.