[Bug] `unknown` value type is escaped/unescaped instead of kept verbatim, breaking round-trips

[lcampanella98]

Describe the bug

When a property's value type is unknown (unrecognized properties, including
X- properties without a VALUE parameter), the value seems to be run through
RFC 5545 TEXT escaping rules instead of being kept as written. As I read
RFC 7265 §5.1/§5.2, an unknown value should be passed through "without
processing."

The problem does not look limited to jCal. In my testing, the value changes in
three places:

1. Just reading it — the value I get back from from_ical is already
   different from the source, before I write anything out.
2. iCal → iCal round-trip — iCalendar in, iCalendar out, no jCal involved;
   backslashes are added or removed.
3. iCal → jCal — the case RFC 7265 §5.1 warns "breaks round-tripping
   values."

I think the jCal case (3) is a clear RFC 7265 violation. I am less sure about
the plain-iCal cases (1, 2).

To reproduce

from icalendar.cal import Component

orig = r"""BEGIN:VEVENT
UID:1
X-MYSTERY:a;b\,c\:d\ne
END:VEVENT
"""
ev = Component.from_ical(orig)

print(repr(str(ev["X-MYSTERY"])))      # just reading it
print(ev.to_ical().decode())           # iCal -> iCal
print(ev.to_jcal())                     # iCal -> jCal

Output:

'a;b,c:d\ne'             # read:  \, -> ,   \: -> :   \n -> real newline
BEGIN:VEVENT
UID:1
X-MYSTERY:a\;b\,c:d\ne   # iCal:  ; -> \;   \, kept   \: -> :
END:VEVENT

['vevent', [['uid', {}, 'text', '1'], ['x-mystery', {}, 'unknown', 'a;b,c:d\ne']], []]
# jCal:  \, -> ,  (kept in iCal above, lost here -> breaks round-tripping, RFC 7265 §5.3)

The source value a;b\,c\:d\ne is preserved by none of the three. The escaped
\, is the clearest case — it survives the iCal → iCal round-trip but loses its
backslash in jCal. Across my tests the affected characters were ,, ;, \,
:, and newlines; others came through unchanged, though I have not tried every
input.

Expected behavior

As I read RFC 7265 §5.1/§5.2 and the §5.3 example, an unknown value should
come through exactly as written, in both directions. For the input above that
would mean:

>>> str(ev["X-MYSTERY"])
'a;b\\,c\\:d\\ne'
>>> print(ev.to_ical().decode())
...
X-MYSTERY:a;b\,c\:d\ne
...
>>> ev.to_jcal()
['vevent', [['uid', {}, 'text', '1'], ['x-mystery', {}, 'unknown', 'a;b\\,c\\:d\\ne']], []]

i.e. no RFC 5545 escaping or unescaping for unknown values either way (only
JSON string escaping, which the JSON layer does on its own).

The RFC 7265 §5.3 example shows the same idea — the decoded jCal value matches
the iCalendar value exactly:

	value
iCalendar	Stenophylla;Guinea\,Africa
jCal (decoded)	Stenophylla;Guinea\,Africa

Environment

• Operating system: Ubuntu (WSL2)
• Python version: Python 3.14.5
• icalendar version: 7.1.2

Additional context

One case I have not worked out: a jCal unknown string could hold a character
(such as a raw line break) that RFC 5545 may not be able to put in a bare
value, so passing it through unchanged could clash with producing valid
iCalendar. The RFC examples do not seem to cover this.

• I tested it with the latest version (current main).
• I attached the .ics sample file, or there is no .ics sample file. (Reproducer is inline above.)

[stevepiercy]

@lcampanella98 first, is that a valid iCalendar TEXT value for the property? It doesn't look compliant to me. See the description in https://datatracker.ietf.org/doc/html/rfc5545.html#section-3.3.11.

Next, did you try running this through a debugger?

I see that Python will read the string, replace line endings with \n, and escape backslashes as \\. So far so good.

'BEGIN:VEVENT
UID:1
X-MYSTERY:a;b\\,c\\:d\\ne
END:VEVENT
'

Then icalendar.parser.content_line.Contentline.parts, in its try block, converts the string. I'm not sure whether this behavior aligns with the RFC.

icalendar/src/icalendar/parser/content_line.py

Lines 171 to 191 in 24b7db5

	try:
	name_split: int | None = None
	value_split: int | None = None
	in_quotes: bool = False
	escaped: bool = False
	for i, ch in enumerate(self):
	if ch == '"' and not escaped:
	in_quotes = not in_quotes
	elif ch == "\\" and not in_quotes:
	escaped = True
	continue
	elif not in_quotes and not escaped:
	# Find first delimiter for name
	if ch in ":;" and name_split is None:
	name_split = i
	# Find value delimiter (first colon)
	if ch == ":" and value_split is None:
	value_split = i
	escaped = False

Then in vText.__new__, there's more escaping.

Note that it doesn't matter whether the property is unknown. If you use DESCRIPTION instead, you'll get the same result, with only the property name being different.

>>> from icalendar.cal import Component
>>> orig = r"""BEGIN:VEVENT
UID:1
X-MYSTERY:a;b\,c\:d\ne
END:VEVENT
"""
>>> ev = Component.from_ical(orig)
>>> print(ev.to_ical().decode())
BEGIN:VEVENT
UID:1
X-MYSTERY:a\;b\,c:d\ne
END:VEVENT

>>> from icalendar.cal import Component
>>> orig = r"""BEGIN:VEVENT
UID:1
DESCRIPTION:a;b\,c\:d\ne
END:VEVENT
"""
>>> ev = Component.from_ical(orig)
>>> print(ev.to_ical().decode())
BEGIN:VEVENT
UID:1
DESCRIPTION:a\;b\,c:d\ne
END:VEVENT

I'm not sure how to go forward from here. @lcampanella98 what do you think?

[lcampanella98]

I ran it through the debugger and see the same two steps you did: Contentline.parts unescapes the value on the way in, and on the way out vText.to_ical re-escapes via _escape_char (which vUnknown inherits).

On your first question: you're right, a;b\,c\:d\ne is not valid TEXT — the bare ; should be \;, and \: isn't allowed at all. My example was a poor choice, sorry about that.

But I think that's the core of the issue. The point of the unknown type in RFC 7265 is that for an unrecognized property we don't know it's TEXT, so we shouldn't apply TEXT's escaping. §5.1/§5.2 say an unknown value must be taken "without processing," and the §5.3 example makes the point:

iCalendar:  X-COFFEE-DATA:Stenophylla;Guinea\,Africa
jCal:       ["x-coffee-data", {}, "unknown", "Stenophylla;Guinea\\,Africa"]

That value has a bare ; too, so it isn't valid TEXT either — yet the RFC requires it to round-trip byte-for-byte, and §5.1 warns that treating it as TEXT means "additional escaping may occur, which breaks round-tripping values." A vendor X- property may use ; or , as its own delimiters, and rewriting ; → \; silently changes its data.

Using the RFC's own §5.3 example as the reference case:

>>> from icalendar.cal import Component
>>> orig = r"""BEGIN:VEVENT
UID:1
X-COFFEE-DATA:Stenophylla;Guinea\,Africa
END:VEVENT
"""
>>> ev = Component.from_ical(orig)
>>> print(ev.to_ical().decode())
BEGIN:VEVENT
UID:1
X-COFFEE-DATA:Stenophylla\;Guinea\,Africa
END:VEVENT
>>> ev.to_jcal()
['vevent', [['uid', {}, 'text', '1'], ['x-coffee-data', {}, 'unknown', 'Stenophylla;Guinea,Africa']], []]

In iCal the ; gains a backslash that wasn't there; in jCal the \, loses its backslash, where RFC §5.3 gives the value as "Stenophylla;Guinea\\,Africa", preserved exactly. Both are the round-trip break §5.1 describes. And you're right that DESCRIPTION behaves the same way today, since the escaping code is shared; I'd keep any fix scoped to unknown values so known TEXT properties are unaffected. Let me know if you see it the same way — happy to open a PR along those lines.

[niccokunzmann]

Thanks for catching this.

Any PR would need to obey the two requirements:

To correctly implement this format, it is critical that the type
"unknown" be used if the default type is not known. If this
requirement is ignored and, for example, "text" is used, additional
escaping may occur, which breaks round-tripping values.

Any unrecognized property parameter MUST be converted to a string
value, with its content set to the property parameter value text, and
treated as if it were a "TEXT" value.
from https://datatracker.ietf.org/doc/html/rfc7265.html#autoid-32

Otherwise, the next bug is introduced.

It might be a good idea to remove the inheritance from vUnknown to vText just to stress the point.

Your PR would be very welcome! I especially welcome test parametrization and lots of examples and example files.

[stevepiercy]

It might be a good idea to remove the inheritance from vUnknown to vText just to stress the point.

I'd suggest duplicating vText's class as a starting point, instead of inheriting it. What do you think? Contrasting vUnknown to vText in the docstrings will also add clarity and emphasis.

@niccokunzmann where did you get those quotations of the requirements? A citation or reference link would help with context.

@lcampanella98 please let us know if you need any guidance. Thank you!

[niccokunzmann]

Yes, for duplicating some of vText, also the tests should be parametrized where these two overlap and vText is tested.

[lcampanella98]

Thanks for the guidance. I've opened a PR #1450, noting these requirements of rfc 7265, testing, and duplicating the vText class removing the inheritance between vText and vUnknown.

[niccokunzmann]

This has been released with v7.3.0.