Repository navigation
Mitokic/09202026/custom model logic - #292
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Custom finalization can lose CSV series identities, and the hierarchy authoring example passes an unsupported argument.
2 open findings
What changed in this PR
Adds approved custom forecasting rules to FinnTS’s standard and Agent workflows, alongside built-in models.
Changes:
- Introduces custom-model authoring, enrollment, raw C1 recipes and runtime integration.
- Adds regression coverage for selection, hierarchy, workers, updates and restart integrity.
- Expands documentation and clarifies targeted validation guidance.
| File | Description |
|---|---|
| vignettes/models-used-in-finnts.Rmd | Explains C1 and custom drivers. |
| vignettes/hierarchical-forecasting.Rmd | Documents custom hierarchy workflows. |
| vignettes/ai-agent.Rmd | Documents custom enrollment and updates. |
| tests/testthat/test-custom-model-standard.R | Tests training and version identity. |
| tests/testthat/test-custom-model-standard-workers.R | Checks parallel training parity. |
| tests/testthat/test-custom-model-standard-selection.R | Tests averages and component integrity. |
| tests/testthat/test-custom-model-standard-global.R | Tests global output and flooring. |
| tests/testthat/test-custom-model-standard-errors.R | Tests runtime failure boundaries. |
| tests/testthat/test-custom-model-standard-enrollment.R | Tests explicit enrollment rules. |
| tests/testthat/test-custom-model-standard-authorized.R | Tests automatic approval envelopes. |
| tests/testthat/test-custom-model-hierarchy-runtime.R | Exercises standard hierarchy forecasts. |
| tests/testthat/test-custom-model-hierarchy-grouped.R | Exercises grouped hierarchy forecasts. |
| tests/testthat/test-custom-model-example-table.R | Tests example-table conversion. |
| tests/testthat/test-custom-model-contract.R | Tests definitions and content hashes. |
| tests/testthat/test-custom-model-c1-drivers.R | Tests independent C1 drivers. |
| tests/testthat/test-artifact-io-parallel.R | Updates worker helper dependencies. |
| tests/testthat/test-agent-update-forecast.R | Checks lightweight worker context. |
| tests/testthat/test-agent-custom-updates.R | Tests replay and identifier preservation. |
| tests/testthat/test-agent-custom-update-workers.R | Checks real update workers. |
| tests/testthat/test-agent-custom-update-restart.R | Tests restart without refitting. |
| tests/testthat/test-agent-custom-update-integrity.R | Tests immutable update provenance. |
| tests/testthat/test-agent-custom-update-global.R | Tests shared global replay. |
| tests/testthat/test-agent-custom-update-execution.R | Tests recoverable error classification. |
| tests/testthat/test-agent-custom-update-defaults.R | Tests new-series default forecasts. |
| tests/testthat/test-agent-custom-update-cohort.R | Tests changed-cohort recovery. |
| tests/testthat/test-agent-custom-storage.R | Tests exact-path backend reads. |
| tests/testthat/test-agent-custom-iteration.R | Tests iteration and restart behavior. |
| tests/testthat/test-agent-custom-iteration-workers.R | Checks independent Agent workers. |
| tests/testthat/test-agent-custom-iteration-global.R | Tests global iteration identity. |
| tests/testthat/test-agent-custom-integration.R | Tests reasoning and runtime integration. |
| tests/testthat/test-agent-custom-fallback.R | Tests bounded custom recovery. |
| tests/testthat/test-agent-custom-enrollment.R | Tests enrollment persistence and references. |
| tests/testthat/test-agent-custom-driver-update.R | Tests driver preservation during updates. |
| tests/testthat/test-agent-custom-driver-iteration.R | Tests driver-independent Agent settings. |
| tests/testthat/helper-custom-model-standard.R | Supplies standard forecasting fixtures. |
| tests/testthat/helper-custom-model-hierarchy.R | Supplies hierarchy fixtures and assertions. |
| tests/testthat/helper-custom-model-authoring.R | Supplies authoring and console fixtures. |
| tests/testthat/helper-custom-drivers.R | Supplies proportional-driver fixtures. |
| tests/testthat/helper-agent-custom-models.R | Supplies offline Agent fixtures. |
| scott_cherry_custom_acr_logic_playground_prompt.md | Adds a business-rule generation prompt. |
| R/utility.R | Registers the custom engine. |
| R/read_write_data.R | Adds C1 artifact handling. |
| R/hierarchy.R | Validates custom reconciliation coverage. |
| R/forecast_time_series.R | Integrates custom enrollment and C1. |
| R/forecast_selection.R | Supports raw C1 history. |
| R/final_models.R | Audits custom artifacts before selection. |
| R/ensemble_models.R | Aligns mixed-model ensemble targets. |
| R/custom_model_registry.R | Adds version-bound model storage. |
| R/agent_summarize_models.R | Adds custom metadata summaries. |
| R/agent_info.R | Persists custom Agent enrollment. |
| NEWS.md | Records custom-model features and fixes. |
| NAMESPACE | Registers new exports and printing. |
| man/update_forecast.Rd | Documents custom replay requirements. |
| man/train_models.Rd | Documents custom training behavior. |
| man/set_agent_info.Rd | Documents custom Agent setup. |
| man/print.finnts_custom_model.Rd | Documents custom-model printing. |
| man/prep_models.Rd | Documents enrollment contracts. |
| man/prep_data.Rd | Documents C1 preparation. |
| man/iterate_forecast.Rd | Documents custom iteration behavior. |
| man/get_summarized_models.Rd | Documents custom summary metadata. |
| man/forecast_time_series.Rd | Documents integrated custom forecasts. |
| man/final_models.Rd | Documents custom selection and reconciliation. |
| man/ensemble_models.Rd | Documents custom ensemble inputs. |
| man/custom_model_predict_impl.Rd | Documents prediction contracts. |
| man/custom_model_fit_impl.Rd | Documents fitting contracts. |
| man/custom_model_feedback.Rd | Documents authoring feedback. |
| man/custom_model_examples_template.Rd | Documents worked-example tables. |
| man/custom_model_control.Rd | Documents authoring controls. |
| man/create_custom_model.Rd | Documents creation and execution boundaries. |
| DESCRIPTION | Advances the development version. |
| CLAUDE.md | References targeted validation scope. |
| AGENTS.md | Defines validation and implementation scope. |
| .vscode/settings.json | Sets the Python environment manager. |
| .github/agent-guides/validation-matrix.md | Updates validation scope and regression guidance. |
| .claude/skills/finnts-validation/SKILL.md | Updates canonical validation guidance. |
| .claude/rules/r-package.md | Clarifies deferred full-validation checks. |
| .agents/skills/finnts-validation/SKILL.md | Synchronizes generated validation guidance. |
| _pkgdown.yml | Adds the Custom Models reference section. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
smdp2000ofc
reviewed
Oct 19, 2026
smdp2000ofc
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add custom forecasting models to finnts.
create_custom_model()turns plain-language business rules plus a simple example table into a validated, approved local or global model. That model then runs in standard forecasts (prep_models()→final_models()) and in Agent forecasts and updates, side by side with the built-in models.Changes
create_custom_model()uses an LLM to generate fit/predict code from instructions and checks it against worked examples invalidation_examples. It approves the model manually in interactive sessions and automatically otherwise, and returns afinnts_custom_model. Draft/response round-trips are removed, so it is a single call.example,Date, target, drivers,expected, and optionaledge_case,expected_error,howandtolerance. A new exportedcustom_model_examples_template()gives a starting table that users and coding agents can fill in.custom_model_feedback(),custom_model_control()andprint()for custom models.C1recipe (recipes_to_run) feeds custom models raw history and every eligible column as a driver. It works with built-in recipes, multistep, feature selection, averages/ensembles and Spark.prep_models(models_to_run = NULL)runs all built-in models plus every supplied custom model. An explicitmodels_to_runmust name each custom model, or the call errors.update_forecast(), including restart, integrity checks, cohorts and worker paths.Notable / Breaking
create_custom_model,custom_model_examples_template,custom_model_feedback,custom_model_control.prep_models()errors when custom models are supplied but missing from an explicitmodels_to_run.ellmerchat object.Testing
devtools::check(), the full suite) was not run.