qualia_core.evaluation package

Subpackages

• qualia_core.evaluation.host package
  • Submodules
    • qualia_core.evaluation.host.Keras module
      • benchmark()
      • Keras
    • qualia_core.evaluation.host.Qualia module
      • Result
      • Qualia
    • qualia_core.evaluation.host.STM32CubeAI module
      • STM32CubeAI
    • qualia_core.evaluation.host.TFLite module
      • TFLite
  • Module contents
    • Keras
      • Keras.evaluate()
    • Qualia
      • Qualia.evaluate()
    • STM32CubeAI
    • TFLite
      • TFLite.evaluate()
• qualia_core.evaluation.target package
  • Submodules
    • qualia_core.evaluation.target.Qualia module
      • Result
      • Qualia
    • qualia_core.evaluation.target.STM32CubeAI module
      • STM32CubeAI
  • Module contents
    • Qualia
      • Qualia.evaluate()
    • STM32CubeAI

Submodules

• qualia_core.evaluation.Evaluator module
  • Evaluator
    • Evaluator.apply_dataaugmentations()
    • Evaluator.shuffle_dataset()
    • Evaluator.limit_dataset()
    • Evaluator.compute_accuracy()
    • Evaluator.evaluate()
• qualia_core.evaluation.STM32CubeAI module
  • STM32CubeAI
    • STM32CubeAI.evaluate()
• qualia_core.evaluation.Stats module
  • StatsFields
    • StatsFields.name
    • StatsFields.i
    • StatsFields.quantization
    • StatsFields.params
    • StatsFields.mem_params
    • StatsFields.accuracy
    • StatsFields.avg_time
    • StatsFields.rom_size
    • StatsFields.ram_size
    • StatsFields.metrics
  • Stats
    • Stats.name
    • Stats.i
    • Stats.quantization
    • Stats.params
    • Stats.mem_params
    • Stats.accuracy
    • Stats.avg_time
    • Stats.rom_size
    • Stats.ram_size
    • Stats.metrics
    • Stats.fieldnames()
    • Stats.asdict()
    • Stats.astuple()
    • Stats.asnamedtuple()

Module contents

class qualia_core.evaluation.Evaluator

Bases: ABC

apply_dataaugmentations(framework: LearningFramework[Any], dataaugmentations: list[DataAugmentation] | None, test_x: numpy.typing.NDArray[np.float32], test_y: numpy.typing.NDArray[np.int32]) → tuple[numpy.typing.NDArray[np.float32], numpy.typing.NDArray[np.int32]]

Apply evaluation qualia_core.dataaugmentation.DataAugmentation.DataAugmentation to dataset.

Only the qualia_core.dataaugmentation.DataAugmentation.DataAugmentation with qualia_core.dataaugmentation.DataAugmentation.DataAugmentation.evaluate set are applied. This should not be used to apply actual data augmentation to the data, but rather use the conversion or transform qualia_core.dataaugmentation.DataAugmentation.DataAugmentation modules.

Parameters:

• framework – The qualia_core.learningframework.LearningFramework.LearningFramework instance providing the qualia_core.learningframework.LearningFramework.LearningFramework.apply_dataaugmentation() method compatible with the provided qualia_core.dataaugmentation.DataAugmentation.DataAugmentation

• dataaugmentations – List of qualia_core.dataaugmentation.DataAugmentation.DataAugmentation objects

• test_x – Input data to apply data augmentation to

• test_y – Input labels to apply data augmentation to

Returns:

Tuple of data and labels after applying qualia_core.dataaugmentation.DataAugmentation.DataAugmentation sequentially

compute_accuracy(preds: list[int], truth: numpy.typing.NDArray[np.int32]) → float

Compute accuracy from the target results.

Parameters:

• results – List of Result from inference on the target

• test_y – Array of ground truth one-hot encoded

Returns:

Accuracy (micro) between 0 and 1

abstractmethod evaluate(framework: LearningFramework[Any], model_kind: str, dataset: RawDataModel, target: str, tag: str, limit: int | None = None, dataaugmentations: list[DataAugmentation] | None = None) → Stats | None

limit_dataset(test_x: numpy.typing.NDArray[np.float32], test_y: numpy.typing.NDArray[np.int32], limit: int | None) → tuple[numpy.typing.NDArray[np.float32], numpy.typing.NDArray[np.int32]]

Truncate dataset to limit samples.

Parameters:

• test_x – Input data

• test_y – Input labels

• limit – Number of samples to truncate to, data is returned as-is if None or 0

Returns:

Tuple of data and labels limited to limit samples

shuffle_dataset(test_x: numpy.typing.NDArray[np.float32], test_y: numpy.typing.NDArray[np.int32]) → tuple[numpy.typing.NDArray[np.float32], numpy.typing.NDArray[np.int32]]

Shuffle the input data, keeping the labels in the same order as the shuffled data.

Shuffling uses the seeded shared random generator from qualia_core.random.shared.

Parameters:

• test_x – Input data

• test_y – Input labels

Returns:

Tuple of shuffled data and labels

class qualia_core.evaluation.Stats(name: str = '', i: int = -1, quantization: str = 'float32', params: int = -1, mem_params: int = -1, accuracy: float = -1, avg_time: float = -1, rom_size: int = -1, ram_size: int = -1, metrics: dict[str, float] = <factory>)

Bases: object

accuracy: float = -1

asdict() → dict[str, Any]

asnamedtuple() → StatsFields

astuple() → tuple[Any, ...]

avg_time: float = -1

classmethod fieldnames() → tuple[str, ...]

i: int = -1

mem_params: int = -1

name: str = ''

params: int = -1

quantization: str = 'float32'

ram_size: int = -1

rom_size: int = -1

metrics: dict[str, float]

class qualia_core.evaluation.STM32CubeAI(stm32cubeai_args=(), mode='')

Bases: object

evaluate(framework, model_kind, dataset, target: str, tag: str, limit: int = None, dataaugmentations=[])