An experiment tracks a model training run: it records parameters and stepped metric series as the model trains, and integrates with MLflow. An evaluation captures a single snapshot of how a finished model version performs against a set of test data. The two use different APIs: experiments are written from a ModelOutput, while evaluations are written from a ModelInput.
Evaluations are authored in Code Repositories, using the @pm.transforms.evaluation decorator together with a ModelInput. See Author and run evaluations for a complete example.
Evaluations can only be written to a model that is in the same project as the code repository where the evaluation logic runs. If the model and the repository are in different projects, the write is denied. Move the model or the repository so that they share a project.
An evaluation is always linked to the single model version that the ModelInput resolves and loads into the transform. By default this is the latest version on the build's branch; you can pin a version with model_version or target a branch with branch. See Determine which model version is evaluated.
To evaluate a new model version, you can either setup a schedule that will run the evaluation logic whenever a new model version is published, or navigate to the evaluations view and run the evaluation using the Run option at the top of the page.
No. Reused names are automatically made unique, so the same code can run multiple times without renaming. If you do not provide a name, the evaluation set name is used.
Nothing is published. The evaluation is committed to its evaluation set only when the build succeeds; if the build fails, the evaluation is aborted.
Use a separate evaluation set for each methodology. Because the set name is fixed in the transform definition by the decorator, define a separate transform with a different set name for each distinct analysis. A model can have up to 100 evaluation sets.
An evaluation set holds all of the evaluations produced by the same methodology across model versions. Because every evaluation is tied to a specific version, you can compare how a metric evolves on the model page as the model is retrained.
Metrics (numeric values), images (PNG or Pillow images), plots (Plotly figures), and tables (pandas or Polars DataFrames). You can group related logs into subsets using a prefix/name convention. See the Limits for size and length constraints.
Only if you use the built-in evaluators. Logging your own metrics, images, plots, and tables does not require scikit-learn.