Repository navigation
bug(sql): sqlglot corrupts PostgreSQL Unicode escape string literals (U&'...') #38009
Description
Bug description
Superset silently corrupts PostgreSQL Unicode escape string literals (U&'...'), breaking charts with "Data error". sqlglot misparses U& as identifier U + bitwise AND &, adding spaces during SQL regeneration. Multiple code paths route SQL through Dialect.generate(). This affects expression sanitization, virtual dataset RLS, SQL execution, and dialect transpilation.
Reproduction steps
- Connect a PostgreSQL database
- Create a virtual dataset with SQL containing a Unicode escape string, e.g.:
SELECT (U&'\FE01' || 'Test literal') AS label- Create a chart using this dataset
- Observe the chart fails with:
Error: column "u" does not exist
Expected behavior
The Unicode escape literal U&'\FE01' should be preserved as-is and sent to PostgreSQL without modification.
Actual behavior
The U&'\FE01' expression is corrupted to U & '\FE01' (spaces added around &), which PostgreSQL interprets as: column u (bitwise AND) string literal '\FE01'.
Screenshots/recordings
No response
Superset version
6.0.0
Python version
3.11
Node version
18 or greater
Browser
Chrome
Additional context
Sqlglot does not have tokenizer/parser support for PostgreSQL's U&'...' syntax. When Superset passes SQL through Dialect.generate(), it reconstructs the SQL with the U and & treated as separate tokens.
This can be confirmed directly:
import sqlglot
from sqlglot.dialects.dialect import Dialect
sql = r"SELECT (U&'\FE01' || 'test')"
dialect = Dialect.get_or_raise('postgres')
for stmt in sqlglot.parse(sql, dialect='postgres'):
result = dialect.generate(stmt, copy=True, pretty=True)
print(result)
# Output: SELECT (U & '\FE01' || 'test') -- CORRUPTEDPossible fixes
- Workaround on SQL level: replace
U&'\FE01' || '...'withE'\uFE01' || '...'orchr(65025) || '...'. - Upstream fix in sqlglot: Add
U&'...'support to sqlglot's PostgreSQL tokenizer/parser (the proper long-term fix). - Avoid unnecessary reformatting: Where
Dialect.generate()is called only for validation (not transformation), return the original SQL instead of the regenerated version. Only use the regenerated SQL when actual transformations (RLS injection, transpilation) were applied. - Make known query limitations transparent by adding context help to SQL query input fields.
Checklist
- I have searched Superset docs and Slack and didn't find a solution to my problem.
- I have searched the GitHub issue tracker and didn't find a similar bug report.
- I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
Activity
rusackas commentedon Jul 12, 2026
Nice writeup @goldjee, the fix options at the bottom are basically the map here.
Digging into where we stand: #41125 (which was actually for #36113) covers part of this. It stopped
sanitize_clausefrom round-tripping adhoc expressions through sqlglot, so aU&'...'literal in a metric/column/adhoc filter is now returned verbatim and survives. The basic virtual-dataset repro in this issue is OK too, but that one's from #38683 ('apply RLS conservatively'), which only regenerates the FROM clause when RLS was actually applied.So the common cases are handled by avoiding the re-render. What's still broken is anywhere sqlglot genuinely has to regenerate the SQL: a virtual dataset (or adhoc clause) with RLS actually applied, cross-dialect transpilation, or a clause that contains a comment (that last one takes
sanitize_clause's re-render branch). Those still come back asU & '...'and break.The root cause is upstream and still unfixed. I checked the latest sqlglot (30.12.0, and we're pinned
<31) and PostgresU&'\FE01'still tokenizes asU+&. sqlglot does have aUNICODE_STRINGtoken and aUnicodeStringexpression, and it supports this for Trino/Snowflake, but the Postgres dialect'sUNICODE_STRINGSlist is empty, soU&was never wired up there. A version bump won't help until that lands.So I don't think a Superset feature PR is the right shape for the real fix. The clean path is upstream: teach sqlglot's Postgres dialect to tokenize/generate
U&'...'(the scaffolding is already there for other dialects), then we bump once it's released. If we need relief sooner, the interim option is a mask/restore shim around our sqlglot round-trips (protect the literal before parse, restore after generate) so the RLS/transpile paths survive, but I'd rather not take on more SQL string-munging than we have to, which your own note calls out. Keeping this open.goldjee commentedon Jul 18, 2026
Posted a bug report to sqlglot's GitHub: tobymao/sqlglot#7898.
goldjee commentedon Jul 21, 2026
Looks like it's fixed in sqlglot v30.13.0.
@rusackas
rusackas commentedon Sep 4, 2026
Thanks @goldjee. sqlglot 30.13.0 had the upstream fix and #42772 bumped us past it (30.17.0 on master now), with #41125 covering the
sanitize_clausepath. Closing, but holler if it still bites and I'll reopen.