Skip to content

Helpers

create_label_to_name_list_node_mapping

Often, data sources contain list values named after the "label" of the JSON project list node, instead of the "name" which is needed for the dsp-tools xmlupload. To create a correct XML, you need a dictionary that maps the "labels" to their correct "names".

Parameters:

Name Type Description Default
project_json_path str

path to a JSON project file (a.k.a. ontology)

 required
list_name str

name of a list in the JSON project

 required
language_of_label str

which language of the label to choose

 required

Returns:

Type Description
dict[str, str]

a dictionary of the form {label: name}
Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py 
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def create_label_to_name_list_node_mapping(
    project_json_path: str,
    list_name: str,
    language_of_label: str,
) -> dict[str, str]:
    """
    Often, data sources contain list values named after the "label" of the JSON project list node, instead of the "name"
    which is needed for the `dsp-tools xmlupload`.
    To create a correct XML, you need a dictionary that maps the "labels" to their correct "names".

    Args:
        project_json_path: path to a JSON project file (a.k.a. ontology)
        list_name: name of a list in the JSON project
        language_of_label: which language of the label to choose

    Returns:
        a dictionary of the form {label: name}
    """
    with open(project_json_path, encoding="utf-8") as f:
        json_file = json.load(f)
    json_subset = [x for x in json_file["project"]["lists"] if x["name"] == list_name]
    # json_subset is a list containing one item, namely the json object containing the entire json-list
    res = {}
    for label, name in _name_label_mapper_iterator(json_subset, language_of_label):
        if name != list_name:
            res[label] = name
            res[label.strip().lower()] = name
    return res

escape_reserved_xml_characters

From richtext strings (encoding="xml"), escape the reserved characters <, > and &, but only if they are not part of a standard standoff tag or escape sequence.

See the documentation for the standard standoff tags allowed by DSP-API, which will not be escaped.

Parameters:

Name Type Description Default
text str

the richtext string to be escaped

 required

Returns:

Type Description
str

The escaped richtext string
Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def escape_reserved_xml_characters(text: str) -> str:
    """
    From richtext strings (encoding="xml"), escape the reserved characters `<`, `>` and `&`,
    but only if they are not part of a standard standoff tag or escape sequence.

    [See the documentation for the standard standoff tags allowed by DSP-API,
    which will not be escaped.](https://docs.dasch.swiss/latest/DSP-API/03-endpoints/api-v2/text/standard-standoff/)

    Args:
        text: the richtext string to be escaped

    Returns:
        The escaped richtext string
    """
    allowed_tags = [
        "a( [^>]+)?",  # <a> is the only tag that can have attributes
        "p",
        "em",
        "strong",
        "u",
        "sub",
        "sup",
        "strike",
        "h1",
        "ol",
        "ul",
        "li",
        "tbody",
        "table",
        "tr",
        "td",
        "br",
        "hr",
        "pre",
        "cite",
        "blockquote",
        "code",
    ]
    allowed_tags_regex = "|".join(allowed_tags)
    lookahead = rf"(?!/?({allowed_tags_regex})/?>)"
    illegal_lt = rf"<{lookahead}"
    lookbehind = rf"(?<!</?({allowed_tags_regex})/?)"
    illegal_gt = rf"{lookbehind}>"
    illegal_amp = r"&(?![#a-zA-Z0-9]+;)"
    text = regex.sub(illegal_lt, "&lt;", text)
    text = regex.sub(illegal_gt, "&gt;", text)
    text = regex.sub(illegal_amp, "&amp;", text)
    return text

find_date_in_string

Checks if a string contains a date value (single date, or date range), and returns the first found date as DSP-formatted string, see XML documentation for details Returns None if no date was found.

Notes
  • All dates are interpreted in the Christian era and the Gregorian calendar.
  • BC dates are only supported in French notation (e.g. 1000-900 av. J.-C.).
  • The years 0000-2999 are supported, in 3/4-digit form.
  • Dates written with slashes are always interpreted in a European manner: 5/11/2021 is the 5th of November.
  • In the European notation, 2-digit years are expanded to 4 digits, with the current year as watershed:
    • 30.4.24 -> 30.04.2024
    • 30.4.25 -> 30.04.1925
Currently supported date formats
  • 0476-09-04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
  • 0476_09_04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
  • 30.4.2021 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
  • 30.4.21 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
  • 5/11/2021 -> GREGORIAN:CE:2021-11-05:CE:2021-11-05
  • Jan 26, 1993 -> GREGORIAN:CE:1993-01-26:CE:1993-01-26
  • 28.2.-1.12.1515 -> GREGORIAN:CE:1515-02-28:CE:1515-12-01
  • 25.-26.2.0800 -> GREGORIAN:CE:0800-02-25:CE:0800-02-26
  • 1.9.2022-3.1.2024 -> GREGORIAN:CE:2022-09-01:CE:2024-01-03
  • 1848 -> GREGORIAN:CE:1848:CE:1848
  • 1849/1850 -> GREGORIAN:CE:1849:CE:1850
  • 1849/50 -> GREGORIAN:CE:1849:CE:1850
  • 1845-50 -> GREGORIAN:CE:1845:CE:1850
  • 840-50 -> GREGORIAN:CE:840:CE:850
  • 840-1 -> GREGORIAN:CE:840:CE:841
  • 1000-900 av. J.-C. -> GREGORIAN:BC:1000:BC:900
  • 45 av. J.-C. -> GREGORIAN:BC:45:BC:45

Parameters:

Name Type Description Default
string str

string to check

 required

Returns:

Type Description
str | None

DSP-formatted date string, or None
Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py 
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def find_date_in_string(string: str) -> str | None:
    """
    Checks if a string contains a date value (single date, or date range),
    and returns the first found date as DSP-formatted string,
    [see XML documentation for details](https://docs.dasch.swiss/latest/DSP-TOOLS/file-formats/xml-data-file/#date)
    Returns None if no date was found.

    Notes:
        - All dates are interpreted in the Christian era and the Gregorian calendar.
        - BC dates are only supported in French notation (e.g. 1000-900 av. J.-C.).
        - The years 0000-2999 are supported, in 3/4-digit form.
        - Dates written with slashes are always interpreted in a European manner: 5/11/2021 is the 5th of November.
        - In the European notation, 2-digit years are expanded to 4 digits, with the current year as watershed:
            - 30.4.24 -> 30.04.2024
            - 30.4.25 -> 30.04.1925

    Currently supported date formats:
        - 0476-09-04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
        - 0476_09_04 -> GREGORIAN:CE:0476-09-04:CE:0476-09-04
        - 30.4.2021 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
        - 30.4.21 -> GREGORIAN:CE:2021-04-30:CE:2021-04-30
        - 5/11/2021 -> GREGORIAN:CE:2021-11-05:CE:2021-11-05
        - Jan 26, 1993 -> GREGORIAN:CE:1993-01-26:CE:1993-01-26
        - 28.2.-1.12.1515 -> GREGORIAN:CE:1515-02-28:CE:1515-12-01
        - 25.-26.2.0800 -> GREGORIAN:CE:0800-02-25:CE:0800-02-26
        - 1.9.2022-3.1.2024 -> GREGORIAN:CE:2022-09-01:CE:2024-01-03
        - 1848 -> GREGORIAN:CE:1848:CE:1848
        - 1849/1850 -> GREGORIAN:CE:1849:CE:1850
        - 1849/50 -> GREGORIAN:CE:1849:CE:1850
        - 1845-50 -> GREGORIAN:CE:1845:CE:1850
        - 840-50 -> GREGORIAN:CE:840:CE:850
        - 840-1 -> GREGORIAN:CE:840:CE:841
        - 1000-900 av. J.-C. -> GREGORIAN:BC:1000:BC:900
        - 45 av. J.-C. -> GREGORIAN:BC:45:BC:45

    Args:
        string: string to check

    Returns:
        DSP-formatted date string, or None
    """

    # sanitise input, just in case that the method was called on an empty or N/A cell
    if not is_nonempty_value(string):
        return None
    try:
        return _find_date_in_string_throwing(string)
    except ValueError:
        return None

make_xsd_compatible_id

An xsd:ID may not contain all types of special characters, and it must start with a letter or underscore. Replace illegal characters with "", and prepend a leading "" if necessary.

The string must contain at least one Unicode letter (matching the regex \p{L}), _, !, ?, or number, but must not be None, <NA>, N/A, or -.

Parameters:

Name Type Description Default
input_value str | float | int

input value

 required

Returns:

Type Description
str

An xsd ID compatible string based on the input value
Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py 
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
def make_xsd_compatible_id(input_value: str | float | int) -> str:
    """
    An xsd:ID may not contain all types of special characters,
    and it must start with a letter or underscore.
    Replace illegal characters with "_", and prepend a leading "_" if necessary.

    The string must contain at least one Unicode letter (matching the regex ``\\p{L}``),
    _, !, ?, or number, but must not be `None`, `<NA>`, `N/A`, or `-`.

    Args:
        input_value: input value

    Raises:
        InputError: if the input cannot be transformed to an xsd:ID

    Returns:
        An xsd ID compatible string based on the input value
    """
    if not is_nonempty_value(input_value):
        raise InputError(f"The input '{input_value}' cannot be transformed to an xsd:ID")
    # if the start of string is neither letter nor underscore, add an underscore
    res = regex.sub(r"^(?=[^A-Za-z_])", "_", str(input_value))
    # replace all illegal characters by underscore
    res = regex.sub(r"[^\w_\-.]", "_", res, flags=regex.ASCII)
    return res

make_xsd_compatible_id_with_uuid

An xsd:ID may not contain all types of special characters, and it must start with a letter or underscore. Replace illegal characters with "", and prepend a leading "" if necessary. Additionally, add a UUID at the end. The UUID will be different each time the function is called.

The string must contain at least one Unicode letter (matching the regex \p{L}), _, !, ?, or number, but must not be None, <NA>, N/A, or -.

Parameters:

Name Type Description Default
input_value str | float | int

input value

 required

Returns:

Type Description
str

an xsd ID based on the input value, with a UUID attached.
Source code in dsp/dsp-tools/src/dsp_tools/xmllib/helpers.py 
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
def make_xsd_compatible_id_with_uuid(input_value: str | float | int) -> str:
    """
    An xsd:ID may not contain all types of special characters,
    and it must start with a letter or underscore.
    Replace illegal characters with "_", and prepend a leading "_" if necessary.
    Additionally, add a UUID at the end.
    The UUID will be different each time the function is called.

    The string must contain at least one Unicode letter (matching the regex ``\\p{L}``),
    _, !, ?, or number, but must not be `None`, `<NA>`, `N/A`, or `-`.

    Args:
        input_value: input value

    Raises:
        InputError: if the input cannot be transformed to an xsd:ID

    Returns:
        an xsd ID based on the input value, with a UUID attached.
    """
    res = make_xsd_compatible_id(input_value)
    _uuid = uuid.uuid4()
    res = f"{res}_{_uuid}"
    return res