Skip to content

Base

MetricBase

Base class for metrics.

Metrics can be local or global; local means score are calculated per example. Global means score is calculated by looking at the entire dataset, e.g. fluency.

Source code in ragfoundry/evaluation/base.py 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class MetricBase:
    """
    Base class for metrics.

    Metrics can be local or global; local means score are calculated per example.
    Global means score is calculated by looking at the entire dataset, e.g. fluency.
    """

    def __init__(self, key_names, **kwargs):
        self.key_names = key_names
        self.kwargs = kwargs
        self.field = self.key_names["generated"]
        self.target = self.key_names["label"]

    def measure(self, example: dict) -> dict[str, float]:
        """
        Measure the performance of the model on a given example.

        Parameters:
            example (dict): The example to evaluate the model on.

        Returns:
            dict[str, float]: A dictionary containing the performance metrics.

        """
        pass

measure(example: dict) -> dict[str, float]

Measure the performance of the model on a given example.

Parameters:

  • example (dict) –

    The example to evaluate the model on.

Returns:

  • dict[str, float] –

    dict[str, float]: A dictionary containing the performance metrics.

Source code in ragfoundry/evaluation/base.py 
15
16
17
18
19
20
21
22
23
24
25
26
def measure(self, example: dict) -> dict[str, float]:
    """
    Measure the performance of the model on a given example.

    Parameters:
        example (dict): The example to evaluate the model on.

    Returns:
        dict[str, float]: A dictionary containing the performance metrics.

    """
    pass