qualia_core.evaluation.host.Qualia module

class qualia_core.evaluation.host.Qualia.Result(metrics: 'dict[str, float]', time: 'float')[source]

Bases: object

metrics: dict[str, float]

time: float

class qualia_core.evaluation.host.Qualia.Qualia(chunks: int | None = None, max_workers: int | None = None, save_preds: bool = False)[source]

Bases: Evaluator

Custom evaluation loop for Qualia host implementations like Qualia-CodeGen Linux example.

Parameters:: save_preds – Save predictions to CSV file in logs/evaluate/<target>

evaluate(framework: LearningFramework[Any], model_kind: str, dataset: RawDataModel, target: str, tag: str, limit: int | None = None, dataaugmentations: list[DataAugmentation] | None = None) → Stats[source]