Helpers

`create_label_to_name_list_node_mapping`

Often, data sources contain list values named after the "label" of the JSON project list node, instead of the "name" which is needed for the dsp-tools xmlupload. To create a correct XML, you need a dictionary that maps the "labels" to their correct "names".

Parameters:

Name	Type	Description	Default
`project_json_path`	`str`	path to a JSON project file (a.k.a. ontology)	required
`list_name`	`str`	name of a list in the JSON project	required
`language_of_label`	`str`	which language of the label to choose	required

Returns:

Type	Description
`dict[str, str]`	a dictionary of the form {label: name}

Examples:

"lists": [
    {
        "name": "listName",
        "labels": {
            "en": "List",
            "de": "Liste"
        },
        "comments": { ... },
        "nodes": [
            {
                "name": "n1",
                "labels": {
                    "en": "Node 1",
                    "de": "Knoten 1"
                }
            },
            {
                "name": "n2",
                "labels": {
                    "en": "Node 2",
                    "de": "Knoten 2"
                }
            }
        ]
    }
]

result = xmllib.create_label_to_name_list_node_mapping(
    project_json_path="project.json",
    list_name="listName",
    language_of_label="de",
)
# result == {"Knoten 1": "n1", "knoten 1": "n1", "Knoten 2": "n2", "knoten 2": "n2"}

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def create_label_to_name_list_node_mapping(
    project_json_path: str,
    list_name: str,
    language_of_label: str,
) -> dict[str, str]:
    """
    Often, data sources contain list values named after the "label" of the JSON project list node, instead of the "name"
    which is needed for the `dsp-tools xmlupload`.
    To create a correct XML, you need a dictionary that maps the "labels" to their correct "names".

    Args:
        project_json_path: path to a JSON project file (a.k.a. ontology)
        list_name: name of a list in the JSON project
        language_of_label: which language of the label to choose

    Returns:
        a dictionary of the form {label: name}

    Examples:
        ```json
        "lists": [
            {
                "name": "listName",
                "labels": {
                    "en": "List",
                    "de": "Liste"
                },
                "comments": { ... },
                "nodes": [
                    {
                        "name": "n1",
                        "labels": {
                            "en": "Node 1",
                            "de": "Knoten 1"
                        }
                    },
                    {
                        "name": "n2",
                        "labels": {
                            "en": "Node 2",
                            "de": "Knoten 2"
                        }
                    }
                ]
            }
        ]
        ```

        ```python
        result = xmllib.create_label_to_name_list_node_mapping(
            project_json_path="project.json",
            list_name="listName",
            language_of_label="de",
        )
        # result == {"Knoten 1": "n1", "knoten 1": "n1", "Knoten 2": "n2", "knoten 2": "n2"}
        ```
    """
    with open(project_json_path, encoding="utf-8") as f:
        json_file = json.load(f)
    json_subset = [x for x in json_file["project"]["lists"] if x["name"] == list_name]
    # json_subset is a list containing one item, namely the json object containing the entire json-list
    res = {}
    for label, name in _name_label_mapper_iterator(json_subset, language_of_label):
        if name != list_name:
            res[label] = name
            res[label.strip().lower()] = name
    return res

`escape_reserved_xml_characters`

From richtext strings (encoding="xml"), escape the reserved characters <, > and &, but only if they are not part of a standard standoff tag or escape sequence.

See the documentation for the standard standoff tags allowed by DSP-API, which will not be escaped.

Parameters:

Name	Type	Description	Default
`text`	`str`	the richtext string to be escaped	required

Returns:

Type	Description
`str`	The escaped richtext string

Examples:

result = xmllib.escape_reserved_xml_characters("Text <unknownTag>")
# result == "Text &lt;unknownTag&gt;"

result = xmllib.escape_reserved_xml_characters("Text <br/> text after")
# result == "Text <br/> text after"

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def escape_reserved_xml_characters(text: str) -> str:
    """
    From richtext strings (encoding="xml"), escape the reserved characters `<`, `>` and `&`,
    but only if they are not part of a standard standoff tag or escape sequence.

    [See the documentation for the standard standoff tags allowed by DSP-API,
    which will not be escaped.](https://docs.dasch.swiss/latest/DSP-API/03-endpoints/api-v2/text/standard-standoff/)

    Args:
        text: the richtext string to be escaped

    Returns:
        The escaped richtext string

    Examples:
        ```python
        result = xmllib.escape_reserved_xml_characters("Text <unknownTag>")
        # result == "Text &lt;unknownTag&gt;"
        ```

        ```python
        result = xmllib.escape_reserved_xml_characters("Text <br/> text after")
        # result == "Text <br/> text after"
        ```
    """
    allowed_tags = [
        "a( [^>]+)?",  # <a> is the only tag that can have attributes
        "p",
        "em",
        "strong",
        "u",
        "sub",
        "sup",
        "strike",
        "h1",
        "ol",
        "ul",
        "li",
        "tbody",
        "table",
        "tr",
        "td",
        "br",
        "hr",
        "pre",
        "cite",
        "blockquote",
        "code",
    ]
    allowed_tags_regex = "|".join(allowed_tags)
    lookahead = rf"(?!/?({allowed_tags_regex})/?>)"
    illegal_lt = rf"<{lookahead}"
    lookbehind = rf"(?<!</?({allowed_tags_regex})/?)"
    illegal_gt = rf"{lookbehind}>"
    illegal_amp = r"&(?![#a-zA-Z0-9]+;)"
    text = regex.sub(illegal_lt, "&lt;", text)
    text = regex.sub(illegal_gt, "&gt;", text)
    text = regex.sub(illegal_amp, "&amp;", text)
    return text

`find_date_in_string`

Checks if a string contains a date value (single date, or date range), and returns the first found date as DSP-formatted string, see XML documentation for details Returns None if no date was found.

Notes

All dates are interpreted in the Christian era and the Gregorian calendar.
BC dates are only supported in French notation (e.g. 1000-900 av. J.-C.).
The years 0000-2999 are supported, in 3/4-digit form.
Dates written with slashes are always interpreted in a European manner: 5/11/2021 is the 5th of November.
In the European notation, 2-digit years are expanded to 4 digits, with the current year as watershed:
- 30.4.24 -> 30.04.2024
- 30.4.25 -> 30.04.1925

Currently supported date formats

0476-09-04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
0476_09_04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
30.4.2021 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
30.4.21 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
5/11/2021 -> GREGORIAN:CE:2021-11-05:CE:2021-11-05
Jan 26, 1993 -> GREGORIAN:CE:1993-01-26:CE:1993-01-26
28.2.-1.12.1515 -> GREGORIAN:CE:1515-02-28:CE:1515-12-01
25.-26.2.0800 -> GREGORIAN:CE:0800-02-25:CE:0800-02-26
1.9.2022-3.1.2024 -> GREGORIAN:CE:2022-09-01:CE:2024-01-03
1848 -> GREGORIAN:CE:1848:CE:1848
1849/1850 -> GREGORIAN:CE:1849:CE:1850
1849/50 -> GREGORIAN:CE:1849:CE:1850
1845-50 -> GREGORIAN:CE:1845:CE:1850
840-50 -> GREGORIAN:CE:840:CE:850
840-1 -> GREGORIAN:CE:840:CE:841
1000-900 av. J.-C. -> GREGORIAN:BC:1000:BC:900
45 av. J.-C. -> GREGORIAN:BC:45:BC:45

Parameters:

Name	Type	Description	Default
`string`	`str`	string to check	required

Returns:

Type	Description
`str \| None`	DSP-formatted date string, or None

Examples:

result = xmllib.find_date_in_string("1849/1850")
# result == "GREGORIAN:CE:1849:CE:1850"

result = xmllib.find_date_in_string("not a valid date")
# result == None

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def find_date_in_string(string: str) -> str | None:
    """
    Checks if a string contains a date value (single date, or date range),
    and returns the first found date as DSP-formatted string,
    [see XML documentation for details](https://docs.dasch.swiss/latest/DSP-TOOLS/file-formats/xml-data-file/#date)
    Returns None if no date was found.

    Notes:
        - All dates are interpreted in the Christian era and the Gregorian calendar.
        - BC dates are only supported in French notation (e.g. 1000-900 av. J.-C.).
        - The years 0000-2999 are supported, in 3/4-digit form.
        - Dates written with slashes are always interpreted in a European manner: 5/11/2021 is the 5th of November.
        - In the European notation, 2-digit years are expanded to 4 digits, with the current year as watershed:
            - 30.4.24 -> 30.04.2024
            - 30.4.25 -> 30.04.1925

    Currently supported date formats:
        - 0476-09-04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
        - 0476_09_04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
        - 30.4.2021 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
        - 30.4.21 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
        - 5/11/2021 -> GREGORIAN:CE:2021-11-05:CE:2021-11-05
        - Jan 26, 1993 -> GREGORIAN:CE:1993-01-26:CE:1993-01-26
        - 28.2.-1.12.1515 -> GREGORIAN:CE:1515-02-28:CE:1515-12-01
        - 25.-26.2.0800 -> GREGORIAN:CE:0800-02-25:CE:0800-02-26
        - 1.9.2022-3.1.2024 -> GREGORIAN:CE:2022-09-01:CE:2024-01-03
        - 1848 -> GREGORIAN:CE:1848:CE:1848
        - 1849/1850 -> GREGORIAN:CE:1849:CE:1850
        - 1849/50 -> GREGORIAN:CE:1849:CE:1850
        - 1845-50 -> GREGORIAN:CE:1845:CE:1850
        - 840-50 -> GREGORIAN:CE:840:CE:850
        - 840-1 -> GREGORIAN:CE:840:CE:841
        - 1000-900 av. J.-C. -> GREGORIAN:BC:1000:BC:900
        - 45 av. J.-C. -> GREGORIAN:BC:45:BC:45

    Args:
        string: string to check

    Returns:
        DSP-formatted date string, or None

    Examples:
        ```python
        result = xmllib.find_date_in_string("1849/1850")
        # result == "GREGORIAN:CE:1849:CE:1850"
        ```

        ```python
        result = xmllib.find_date_in_string("not a valid date")
        # result == None
        ```
    """

    # sanitise input, just in case that the method was called on an empty or N/A cell
    if not is_nonempty_value(string):
        return None
    try:
        return _find_date_in_string_throwing(string)
    except ValueError:
        return None

`make_xsd_compatible_id`

An xsd:ID may not contain all types of special characters, and it must start with a letter or underscore. Replace illegal characters with _, and prepend a leading _ if necessary.

The string must contain at least one Unicode letter (matching the regex \p{L}), _, !, ?, or number, but must not be None, <NA>, N/A, or -.

Parameters:

Name	Type	Description	Default
`input_value`	`str \| float \| int`	input value	required

Returns:

Type	Description
`str`	An xsd ID compatible string based on the input value

Examples:

result = xmllib.make_xsd_compatible_id("0_Universität_Basel")
# result == "_0_Universit_t_Basel"

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def make_xsd_compatible_id(input_value: str | float | int) -> str:
    """
    An xsd:ID may not contain all types of special characters,
    and it must start with a letter or underscore.
    Replace illegal characters with `_`, and prepend a leading `_` if necessary.

    The string must contain at least one Unicode letter (matching the regex ``\\p{L}``),
    `_`, `!`, `?`, or number, but must not be `None`, `<NA>`, `N/A`, or `-`.

    Args:
        input_value: input value

    Raises:
        InputError: if the input cannot be transformed to an xsd:ID

    Returns:
        An xsd ID compatible string based on the input value

    Examples:
        ```python
        result = xmllib.make_xsd_compatible_id("0_Universität_Basel")
        # result == "_0_Universit_t_Basel"
        ```
    """
    if not is_nonempty_value(input_value):
        raise InputError(f"The input '{input_value}' cannot be transformed to an xsd:ID")
    # if the start of string is neither letter nor underscore, add an underscore
    res = regex.sub(r"^(?=[^A-Za-z_])", "_", str(input_value))
    # replace all illegal characters by underscore
    res = regex.sub(r"[^\w_\-.]", "_", res, flags=regex.ASCII)
    return res

`make_xsd_compatible_id_with_uuid`

An xsd:ID may not contain all types of special characters, and it must start with a letter or underscore. Replace illegal characters with _, and prepend a leading _ if necessary. Additionally, add a UUID at the end. The UUID will be different each time the function is called.

The string must contain at least one Unicode letter (matching the regex \p{L}), _, !, ?, or number, but must not be None, <NA>, N/A, or -.

Parameters:

Name	Type	Description	Default
`input_value`	`str \| float \| int`	input value	required

Returns:

Type	Description
`str`	an xsd ID based on the input value, with a UUID attached.

Examples:

result = xmllib.make_xsd_compatible_id_with_uuid("Universität_Basel")
# result == "Universit_t_Basel_88f5cd0b-f333-4174-9030-65900b17773d"

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def make_xsd_compatible_id_with_uuid(input_value: str | float | int) -> str:
    """
    An xsd:ID may not contain all types of special characters,
    and it must start with a letter or underscore.
    Replace illegal characters with `_`, and prepend a leading `_` if necessary.
    Additionally, add a UUID at the end.
    The UUID will be different each time the function is called.

    The string must contain at least one Unicode letter (matching the regex ``\\p{L}``),
    `_`, `!`, `?`, or number, but must not be `None`, `<NA>`, `N/A`, or `-`.

    Args:
        input_value: input value

    Raises:
        InputError: if the input cannot be transformed to an xsd:ID

    Returns:
        an xsd ID based on the input value, with a UUID attached.

    Examples:
        ```python
        result = xmllib.make_xsd_compatible_id_with_uuid("Universität_Basel")
        # result == "Universit_t_Basel_88f5cd0b-f333-4174-9030-65900b17773d"
        ```
    """
    res = make_xsd_compatible_id(input_value)
    _uuid = uuid.uuid4()
    res = f"{res}_{_uuid}"
    return res

`create_list_from_string`

Creates a list from a string. Trailing and leading whitespaces are removed from the list items.

Parameters:

Name	Type	Description	Default
`string`	`str`	input string	required
`separator`	`str`	The character that separates the different values in the string. For example, a comma or newline.	required

Returns:

Type	Description
`list[str]`	The list that results from splitting the input string. If the original string is empty or consists only of whitespace characters, the resulting list will be empty.

Examples:

result = xmllib.create_non_empty_list_from_string(" One/  Two\n/", "/")
# result == ["One", "Two"]

result = xmllib.create_list_from_string("   \n    ", "\n")
# result == []

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def create_list_from_string(string: str, separator: str) -> list[str]:
    """
    Creates a list from a string.
    Trailing and leading whitespaces are removed from the list items.

    Args:
        string: input string
        separator: The character that separates the different values in the string.
            For example, a comma or newline.

    Returns:
        The list that results from splitting the input string.
            If the original string is empty or consists only of whitespace characters, the resulting list will be empty.

    Raises:
        InputError: If the input value is not a string.

    Examples:
        ```python
        result = xmllib.create_non_empty_list_from_string(" One/  Two\\n/", "/")
        # result == ["One", "Two"]
        ```

        ```python
        result = xmllib.create_list_from_string("   \\n    ", "\\n")
        # result == []
        ```
    """
    if not isinstance(string, str):
        raise InputError(f"The input for this function must be a string. Your input is a {type(string).__name__}.")
    return [strpd for x in string.split(separator) if (strpd := x.strip())]

`create_non_empty_list_from_string`

Creates a list from a string. Trailing and leading whitespaces are removed from the list items.

If the resulting list is empty it will raise an InputError.

Parameters:

Name	Type	Description	Default
`string`	`str`	input string	required
`separator`	`str`	The character that separates the different values in the string. For example, a comma or newline.	required
`resource_id`	`str \| None`	If the ID of the resource is provided, a better error message can be composed	`None`
`prop_name`	`str \| None`	If the name of the property is provided, a better error message can be composed	`None`

Returns:

Type	Description
`list[str]`	The list that results from splitting the input string.

Examples:

result = xmllib.create_non_empty_list_from_string("One\nTwo   ", "\n")
# result == ["One", "Two"]

result = xmllib.create_non_empty_list_from_string("   \n/    ", "/")
# raises InputError

Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py

def create_non_empty_list_from_string(
    string: str, separator: str, resource_id: str | None = None, prop_name: str | None = None
) -> list[str]:
    """
    Creates a list from a string.
    Trailing and leading whitespaces are removed from the list items.

    If the resulting list is empty it will raise an `InputError`.

    Args:
        string: input string
        separator: The character that separates the different values in the string.
            For example, a comma or newline.
        resource_id: If the ID of the resource is provided, a better error message can be composed
        prop_name: If the name of the property is provided, a better error message can be composed

    Returns:
        The list that results from splitting the input string.

    Raises:
        InputError: If the resulting list is empty.

    Examples:
        ```python
        result = xmllib.create_non_empty_list_from_string("One\\nTwo   ", "\\n")
        # result == ["One", "Two"]
        ```

        ```python
        result = xmllib.create_non_empty_list_from_string("   \\n/    ", "/")
        # raises InputError
        ```
    """
    lst = create_list_from_string(string, separator)
    if len(lst) == 0:
        msg = "The input for this function must result in a non-empty list. Your input"
        details = []
        if resource_id:
            details.append(f"resource with the ID '{resource_id}'")
        if prop_name:
            details.append(f"property '{prop_name}'")
        details_msg = "for the " + " and ".join(details) + " " if details else ""
        msg += " " + details_msg + "results in an empty list."
        raise InputError(msg)
    return lst