Validation and Applicability Guide

Application: Local Regression Studio v1.0.11

Purpose

The validation layer helps users identify common data risks, define measurable acceptance requirements, inspect performance by group and recognize prediction inputs that differ from the fitted training domain.

These tools support review. They do not certify a model, establish causality or guarantee reliable predictions under distribution shift.

Data-quality assistant

The assistant updates when the target or selected features change. It currently checks:

• Exact duplicate rows
• Small datasets
• Missing or nonnumeric target values
• Substantial missingness in selected features
• Constant and near-constant selected columns
• Identifier-like selected features
• High-cardinality categorical features
• Features exactly equal to the target
• Extremely high numeric correlation that may indicate target leakage

Each finding is labelled critical, warning or information and includes a suggested response.

Interpretation

A critical finding should normally be resolved before model approval. A warning requires investigation and documentation. Information items explain the current state or limitations of the check.

The absence of findings does not prove that the dataset is representative, unbiased or suitable for the intended use.

Acceptance criteria

Step 6 supports these optional requirements:

• Maximum test RMSE
• Minimum test R²
• Minimum test interval coverage
• Maximum test interval coverage
• Maximum RMSE in any selected test group
• No critical data-quality findings

Blank numeric fields are not evaluated. The result is:

• Passed: all configured measurable requirements passed
• Failed: at least one configured requirement failed
• Not evaluated: no measurable requirement could be evaluated

Acceptance settings and results are stored in the fitted artifact, experiment record and project file.

Performance by group

Choose a group column such as source, site, instrument, batch, regime or a selected feature. The app reports:

• Test rows
• RMSE
• MAE
• R²
• Mean residual bias
• Interval coverage, when intervals are available

Small groups can produce unstable statistics. Group results should be interpreted with sample size and domain knowledge.

Prediction applicability

When a prediction CSV is uploaded, each row receives one of these statuses:

• within-domain
• near-boundary
• warning
• outside-domain

Checks include:

• Numeric values outside fitted training ranges
• Numeric values within 5% of a training-range boundary
• Missing numeric or categorical values
• Categorical values not observed during training
• Rows dropped by fitted preprocessing rules

The downloaded prediction CSV includes:

• applicability_status
• applicability_issue_count
• applicability_notes

Limitations

The current applicability checks are transparent marginal checks. They do not fully detect multivariate extrapolation, covariate shift, concept drift or changes in the relationship between inputs and target.

Conformal calibration

Conformal interval calibration is not included in v1.0.11. A statistically defensible implementation should use a calibration set that is separate from both hyperparameter selection and final test assessment. Reusing the same validation rows for tuning and calibration would weaken the coverage claim. A future release can add an explicit calibration split or nested workflow.

Relationship to approval

In v1.0.11 the acceptance result is an input to the human approval workflow. A full Approved decision requires the configured criteria to pass. Approved with conditions can document restricted use and requires written conditions. Rejected, retired and draft models cannot be exported as approved prediction packages.

Acceptance and approval remain different concepts:

• Acceptance evaluates user-configured measurable criteria.
• Approval records a responsible human decision, intended use, limitations and review date.

v1.0 deployment and governance

v1.0 preserves the documented modelling behaviour and adds separate deployment editions, versioned governance lifecycle records, monitoring and revalidation, model-change assessments, local recovery, stricter import/export validation, and optional verification of organizational signatures against deployment-pinned public keys. See the package-root DEPLOYMENT_GUIDE.md, GOVERNANCE_GUIDE.md, MONITORING_GUIDE.md, SECURITY.md, and MIGRATION_GUIDE.md.