dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general

Module Contents

Functions

check_include_exclude
exact_match
match_in_list
literal_match
is_in_list
diff_text_file
fuzzy_match
fuzzy_place_math
check_csv	Args: result (str): path to csv file rules (Dict[str, List[Dict[str, str]]]): dict like { “expect”: [{key: value}] “unexpect”: [{key: value}] }
check_list	Args: result (str): path to list file rules (Dict[str, List[str]]): dict like { “expect”: list of str as regexes “unexpect”: list of str as regexes }
check_accessibility_tree	Args: result (str): XML of GNOME Accessibility Tree rules (List[Dict[str, Any]]): list of dict like { “selectors”: list of str as CSS selectors, will be connected by “, “ to form a composite selector. Only one from selectors and xpath is needed. If both are present, xpath takes the priority. “xpath”: str as xpath. Only one from selectors and xpath is needed. If both are present, xpath takes the priority. “text”: str as the expected text content of the selected element. “exact”: bool specifying whether exact match or fuzzy match should be performed. defaults to True. } osname (str): “ubuntu” | “windows” | “macos”. “ubuntu” by default.
run_sqlite3
check_json	Args: result (str): path to json file rules (Dict[str, List[Dict[str, Union[List[str], str]]]]): dict like { “expect”: [ { “key”: list of str “method”: str “ref”: something } ], “unexpect”: <the same as expect } is_yaml (bool): yaml rather than json
check_direct_json_object	One of the most commonly used function to evalute. Compare two json objects directly.
compare_time_in_speedtest_results
is_included_all_json_objects
is_gold_text_included_in_pdf
file_contains
check_line_number
compare_terminal_and_txt
compare_python_pure_text

Data

logger

API

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.logger

‘getLogger(…)’

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_include_exclude(result: str, rules: Dict[str, List[str]]) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.exact_match(result, rules) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.match_in_list(result, rules) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.literal_match(result: Any, expected: Any, **options) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.is_in_list(result, rules) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.diff_text_file(result: str, expect: str) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.fuzzy_match(result, rules) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.fuzzy_place_math(result_file_path, rules) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_csv(result: str, rules: Dict[str, List[Dict[str, str]]]) → float

Args: result (str): path to csv file rules (Dict[str, List[Dict[str, str]]]): dict like { “expect”: [{key: value}] “unexpect”: [{key: value}] }

Returns: float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_list(result: str, rules: Dict[str, List[str]]) → float

Args: result (str): path to list file rules (Dict[str, List[str]]): dict like { “expect”: list of str as regexes “unexpect”: list of str as regexes }

Returns: float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_accessibility_tree(result: str, rules: List[Dict[str, Any]], osname: str = 'ubuntu') → float

Args: result (str): XML of GNOME Accessibility Tree rules (List[Dict[str, Any]]): list of dict like { “selectors”: list of str as CSS selectors, will be connected by “, “ to form a composite selector. Only one from selectors and xpath is needed. If both are present, xpath takes the priority. “xpath”: str as xpath. Only one from selectors and xpath is needed. If both are present, xpath takes the priority. “text”: str as the expected text content of the selected element. “exact”: bool specifying whether exact match or fuzzy match should be performed. defaults to True. } osname (str): “ubuntu” | “windows” | “macos”. “ubuntu” by default.

Returns: float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.run_sqlite3(result: str, rules: Dict[str, Any]) → float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_json(result: str, rules: Dict[str, List[Dict[str, Union[List[str], str]]]], is_yaml: bool = False) → float

Args: result (str): path to json file rules (Dict[str, List[Dict[str, Union[List[str], str]]]]): dict like { “expect”: [ { “key”: list of str “method”: str “ref”: something } ], “unexpect”: <the same as expect } is_yaml (bool): yaml rather than json

Returns: float

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_direct_json_object(result, rules) → float

One of the most commonly used function to evalute. Compare two json objects directly.

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.compare_time_in_speedtest_results(speedtest_result_path, time_diff)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.is_included_all_json_objects(gold_file_path, result_file_path)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.is_gold_text_included_in_pdf(pdf_file_path, gold_text_path)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.file_contains(file_path, config)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.check_line_number(file_path, line_number)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.compare_terminal_and_txt(txt_file_path, terminal_output)

dockyard_rl.experience.cua.vendor.desktop_env_eval.metrics.general.compare_python_pure_text(py_file_path, gold_file_path)