What is an SQL parser?什么是 SQL 解析器?
An SQL parser converts SQL text into a structured representation by recognizing tokens and applying a selected dialect grammar. The result is commonly a concrete syntax tree or abstract syntax tree containing statements, projections, table references, joins, predicates, expressions, groups, windows, order clauses, and source locations. If input violates the grammar, the parser reports a syntax error or produces a recovered partial tree according to its policy.
SQL 解析器通过识别 Token 并应用指定数据库方言语法,把 SQL 文本转换为结构化语法树。输出通常是具体语法树或 AST,包含语句、字段投影、表引用、JOIN、条件、表达式、分组、窗口、排序和源码位置。输入违反语法时,解析器会报告错误,或按照明确策略生成部分恢复的语法树。
Parsing is structural, not execution. A successful parse does not prove that a table exists, a column is unambiguous, types are compatible, a function signature is valid, permissions allow access, a query is read-only, a result is correct, or execution is affordable. Those claims require binding, semantic validation, schema context, policy analysis, planning, or controlled execution.
解析关注结构,而不是执行。解析成功不能证明表存在、列无歧义、类型兼容、函数签名有效、权限允许、查询只读、结果正确或执行成本可接受。这些声明需要绑定、语义验证、模式上下文、策略分析、规划或受控执行。
Trace the pipeline from raw text to a usable tree追踪从原始文本到可用语法树的流水线
“Parse SQL” often hides several stages. An application may normalize line endings, remove a byte-order mark, expand templates, split a script, tokenize characters, apply grammar rules, build a parse tree, transform it into an AST, attach source metadata, recover from errors, and run post-parse normalization. Each stage can change what downstream tools see.
“解析 SQL”经常隐藏多个阶段。应用可能先规范换行、移除字节序标记、展开模板、切分脚本,再把字符 Token 化、应用语法规则、构建解析树、转换为 AST、附加源码元数据、从错误恢复,并执行解析后规范化。每个阶段都会改变下游工具看到的内容。
Encoding, bytes, line endings, BOM, length, source identity.
编码、字节、换行、BOM、长度与来源身份。
Templates, variables, directives, includes, delimiter commands.
模板、变量、指令、包含文件与分隔命令。
Keywords, identifiers, literals, parameters, punctuation, trivia.
关键字、标识符、字面值、参数、标点与附属文本。
Dialect productions, precedence, associativity, statement forms.
方言产生式、优先级、结合性与语句形式。
Concrete nodes or semantic AST, spans, comments, recovered nodes.
具体节点或语义 AST、范围、注释与恢复节点。
Statements, diagnostics, coverage, confidence, unsupported constructs.
语句、诊断、覆盖、置信与不支持结构。
PostgreSQL’s official parser-stage documentation distinguishes lexical recognition and grammar parsing from a later transformation process. That separation is broadly useful: a raw syntax tree and a semantically augmented query representation answer different questions and should not be labeled interchangeably.
PostgreSQL 官方的解析阶段文档区分词法识别、语法解析与后续转换过程。这种区分具有普遍价值:原始语法树与经过语义增强的查询表示回答不同问题,不应使用相同标签混淆。
Separate parsing, validation, planning, and execution claims区分解析、验证、规划与执行声明
A parser answers “Can this text be represented by this grammar?” A validator may answer “Do referenced names and types make sense in this schema and policy context?” A planner asks “How could the database execute the validated operation?” An executor performs work. Tools sometimes collapse these stages into one green check, which creates false confidence.
解析器回答“这段文本能否由该语法表示?”验证器可能回答“引用名称与类型在此模式和策略上下文中是否合理?”规划器回答“数据库如何执行已验证操作?”执行器真正完成工作。部分工具把这些阶段合并为一个绿色勾选,从而制造错误信心。
| Stage阶段 | Needs需要 | Can establish可建立 | Cannot establish alone单独无法建立 |
|---|---|---|---|
| Lex and parse词法与解析 | Text, dialect grammar, parser configuration文本、方言语法与解析器配置 | Token and grammatical structure; syntax diagnosticsToken 与语法结构,以及语法诊断 | Object existence, types, privileges, meaning, result对象存在、类型、权限、含义与结果 |
| Bind and validate绑定与验证 | AST, catalog, functions, types, scopes, policiesAST、目录、函数、类型、作用域与策略 | Resolved references, compatible operations, selected policy checks解析引用、兼容操作与所选策略检查 | Runtime result, plan cost, data-dependent behavior运行结果、计划成本与数据依赖行为 |
| Plan or compile规划或编译 | Validated representation, catalog, statistics, settings已验证表示、目录、统计与设置 | Candidate execution plan, estimates, unsupported runtime features候选执行计划、估算与不支持运行功能 | Actual rows, time, locks, side effects, business correctness实际行、时间、锁、副作用与业务正确性 |
| Execute执行 | Database, identity, data, resources, transaction, network数据库、身份、数据、资源、事务与网络 | Observed results, errors, resource use, and side effects for one run一次运行的结果、错误、资源与副作用 | Universal correctness or future performance普遍正确性或未来性能 |
A syntax-only parser should display “parsed,” not “valid” or “safe.” If a tool enriches the tree with schema or executes a dry run, state the data source, identity, freshness, permissions, and side-effect boundary. The same-batch SQL query validator guide covers the broader validation contract.
纯语法解析器应显示“已解析”,而不是“有效”或“安全”。如果工具使用模式增强语法树或执行 dry run,应说明数据来源、身份、新鲜度、权限与副作用边界。同批次的 SQL 查询验证器指南将覆盖更广泛的验证契约。
Define the exact text and preprocessing boundary定义准确文本与预处理边界
The parser may never see the text visible to the user. Applications can expand `${variable}` placeholders, replace templated blocks, strip comments, normalize quotes, resolve notebook cells, include other files, convert named parameters, rewrite client commands, or split batches first. Parse diagnostics then refer to transformed text unless the system maintains a source map.
解析器可能从未看到用户眼中的原始文本。应用可能先展开 `${variable}` 占位符、替换模板块、删除注释、规范引号、解析笔记本单元、包含其他文件、转换命名参数、重写客户端命令或切分批次。如果系统没有维护源码映射,解析诊断就会指向转换后文本。
| Input property输入属性 | Question问题 | Evidence证据 |
|---|---|---|
| Bytes and encoding字节与编码 | Which encoding, BOM, normalization, invalid-byte policy?使用何种编码、BOM、规范化与无效字节策略? | Byte length, decoded length, encoding label, error location字节长度、解码长度、编码标签与错误位置 |
| Line and column convention行列约定 | Are columns bytes, code units, code points, or display cells?列按字节、代码单元、码点还是显示单元计算? | Unicode and tab fixtures with documented indexing baseUnicode 与制表符夹具,以及索引起点说明 |
| Template expansion模板展开 | Which variables, includes, conditionals, and escaping rules?哪些变量、包含、条件与转义规则? | Original text, expanded text, source map, redacted values原始文本、展开文本、源码映射与脱敏值 |
| Script splitting脚本切分 | Are delimiters understood inside strings, comments, and procedural bodies?能否理解字符串、注释与过程体中的分隔符? | Statement ranges, delimiters, client directives, incomplete tail语句范围、分隔符、客户端指令与不完整尾部 |
| Source identity来源身份 | Which file, cell, request, or generated fragment owns each span?每个范围属于哪个文件、单元、请求或生成片段? | Stable source URI, version, content hash, mapping chain稳定来源 URI、版本、内容哈希与映射链 |
Preserve both raw and normalized input where policy permits, with secrets redacted. If a parser accepts pre-tokenized input, document token provenance. Diagnostics and rewrite tools cannot be trusted when source locations silently drift.
在策略允许时保留原始与规范化输入,并去除密钥。如果解析器接受预先 Token 化输入,应记录 Token 来源。源码位置静默漂移时,诊断与重写工具都不可信。
Evaluate the lexer before judging the syntax tree评价语法树前先评价词法器
The lexer turns characters into tokens such as keywords, identifiers, quoted identifiers, string and numeric literals, parameter markers, operators, punctuation, comments, whitespace, and dialect-specific constructs. If token boundaries or kinds are wrong, the grammar receives the wrong language. Lexing is especially sensitive to quoting, escape modes, nested comments, dollar-quoted bodies, Unicode, multi-character operators, and keywords that can also act as identifiers.
词法器把字符转换为关键字、标识符、引用标识符、字符串与数字字面值、参数标记、运算符、标点、注释、空白及方言特有结构等 Token。如果边界或类别错误,语法器接收到的就是错误语言。引用、转义模式、嵌套注释、美元引用过程体、Unicode、多字符运算符,以及可同时充当标识符的关键字尤其敏感。
| Lexer fixture词法夹具 | Expected distinction预期区分 | Failure impact失败影响 |
|---|---|---|
| Quoted and unquoted identifiers引用与未引用标识符 | Quote style, original spelling, escape, case-folding flag引用样式、原始拼写、转义与大小写折叠标志 | Wrong object identity or unsafe reserialization错误对象身份或不安全重新序列化 |
| Strings and escape modes字符串与转义模式 | Delimiter, prefix, decoded value, raw text, termination分隔符、前缀、解码值、原始文本与终止 | Premature statement end, changed literal, false parameter语句提前结束、字面值改变或误识别参数 |
| Numeric forms数字形式 | Integer, decimal, exponent, sign, separator, suffix整数、小数、指数、符号、分隔符与后缀 | Different AST, overflow before validation, source driftAST 改变、验证前溢出与源码漂移 |
| Parameters and variables参数与变量 | Question mark, positional, named, template, client variable问号、位置、命名、模板与客户端变量 | Literal confusion, wrong count, unsafe substitution字面值混淆、数量错误与不安全替换 |
| Comments and hints注释与 Hint | Line, block, nested, optimizer hint, directive行、块、嵌套、优化器 Hint 与指令 | Lost policy marker, wrong split, changed execution hint丢失策略标记、错误切分与执行 Hint 改变 |
| Operators and punctuation运算符与标点 | Longest valid operator, cast, JSON path, range, delimiter最长有效运算符、转换、JSON 路径、范围与分隔符 | Changed precedence or entirely different grammar path优先级改变或进入完全不同语法路径 |
Expose token kind, raw slice, normalized value, quote metadata, start and end offsets, line and column, and channel or trivia status. Redact literal values only in a separate presentation layer; altering tokens before parsing changes the input.
应暴露 Token 类别、原始切片、规范值、引用元数据、起止偏移、行列与通道或附属状态。字面值脱敏应在独立展示层完成;解析前改变 Token 会改变输入。
Treat dialect configuration as part of the parse result把方言配置视为解析结果的一部分
A grammar defines valid statement forms, clause order, expression precedence, associativity, reserved words, extensions, and ambiguity resolution. “ANSI SQL” is rarely sufficient because products support different standard levels and extensions. Some parsers offer strict, lenient, or vendor conformance modes; leniency can increase coverage while hiding unsupported syntax.
语法定义有效语句形式、子句顺序、表达式优先级、结合性、保留字、扩展与歧义解决方式。“ANSI SQL”通常不够,因为产品支持不同标准级别与扩展。部分解析器提供严格、宽松或厂商一致性模式;宽松模式可以提高覆盖,却也可能隐藏不支持语法。
| Configuration配置 | Record记录 | Test测试 |
|---|---|---|
| Dialect and version方言与版本 | Parser dialect name, target engine/version, parser build解析器方言名、目标引擎与版本、解析器构建 | Version-specific keyword and syntax fixtures版本特有关键字与语法夹具 |
| Conformance level一致性级别 | Strict, pragmatic, vendor, lenient, custom flags严格、实用、厂商、宽松与自定义标志 | Construct accepted only under one level只在某一级别接受的结构 |
| Lexical policy词法策略 | Quote styles, case sensitivity, casing, escapes, comments引用样式、大小写敏感、折叠、转义与注释 | Same text tokenizes differently under two policies同一文本在两种策略下 Token 化不同 |
| Extensions扩展 | Custom statements, clauses, functions, operators, hints自定义语句、子句、函数、运算符与 Hint | Round-trip every extension and reject disabled ones每个扩展往返,并拒绝禁用扩展 |
| Recovery policy恢复策略 | Fail fast, collect errors, insert, delete, skip, partial tree快速失败、收集错误、插入、删除、跳过与部分树 | Malformed fixtures with known recovered regions带已知恢复区域的畸形夹具 |
Apache Calcite’s official SQL language reference describes the grammar accepted by its default parser and identifies constructs allowed only at certain conformance levels. A parser result should carry this configuration so another system cannot silently reinterpret the same text under a different grammar.
Apache Calcite 官方的 SQL 语言参考描述其默认解析器接受的语法,并指出部分结构只在特定一致性级别允许。解析结果应携带这些配置,避免另一个系统在不同语法下静默重新解释相同文本。
Define what the abstract syntax tree preserves and discards定义抽象语法树保留与丢弃什么
An AST represents meaningful constructs rather than every character. Parentheses may be represented through tree shape; keywords and commas may disappear; aliases, quote flags, comments, hints, and source spans may be attached selectively; equivalent syntax may normalize to one node form. This is useful for analysis but risky for exact rewriting unless the fidelity contract is explicit.
AST 表示有意义的结构,而不是每个字符。括号可能通过树形表示;关键字与逗号可能消失;别名、引用标志、注释、Hint 与源码范围可能只选择性附加;等价语法还可能规范成一种节点形式。这有利于分析,但若没有明确保真契约,用于精确重写会有风险。
| Tree property树属性 | Decision决策 | Downstream consequence下游影响 |
|---|---|---|
| Concrete versus abstract具体树与抽象树 | Preserve grammar punctuation or only semantic nodes保留语法标点,还是只保留语义节点 | Exact editing versus easier semantic traversal精确编辑与更容易语义遍历之间的取舍 |
| Node identity节点身份 | Stable IDs, object references, paths, or structural equality稳定 ID、对象引用、路径或结构相等 | Incremental updates, comments, diffs, and cache validity增量更新、评论、差异与缓存有效性 |
| Source fidelity源码保真 | Offsets, line/column, full ranges, token references, source map偏移、行列、完整范围、Token 引用与源码映射 | Precise diagnostics, highlights, safe edits, refactoring精确诊断、高亮、安全编辑与重构 |
| Trivia and comments附属文本与注释 | Discard, token channel, attach leading/trailing, standalone nodes丢弃、Token 通道、前后附加或独立节点 | Formatting, hints, directives, documentation, round trip格式化、Hint、指令、文档与往返 |
| Normalization规范化 | Preserve spelling and syntax or canonicalize nodes and values保留拼写与语法,还是规范节点与值 | Stable analysis versus loss of original author intent稳定分析与原始作者意图丢失之间的取舍 |
| Recovered and unknown nodes恢复与未知节点 | Typed placeholder, raw fragment, error node, omitted region类型占位、原始片段、错误节点或省略区域 | Whether downstream analysis can distinguish certainty下游能否区分确定与不确定区域 |
Version the AST schema independently from the parser package if other systems persist or exchange it. Adding a node kind, changing child order, normalizing aliases, or altering source-span semantics can break policies and lineage even when parsing still succeeds.
如果其他系统会持久化或交换 AST,应独立于解析器包对 AST 模式进行版本控制。新增节点类型、改变子节点顺序、规范别名或修改源码范围语义,即使解析仍成功,也可能破坏策略与血缘。
Require precise source spans for every material node为每个关键节点提供精确源码范围
Source spans connect the tree back to the text. They power diagnostics, syntax highlighting, hover details, code actions, policy findings, diffs, refactoring, and review. A node may need a full range, a name range, keyword ranges, delimiter ranges, and ranges for child lists. One coarse statement span is not enough for a safe automated edit.
源码范围把语法树重新连接到文本,为诊断、语法高亮、悬停详情、代码操作、策略发现、差异、重构与审查提供基础。一个节点可能需要完整范围、名称范围、关键字范围、分隔符范围与子列表范围。只有粗粒度语句范围不足以支持安全自动编辑。
| Span requirement范围要求 | Hard case困难案例 | Test测试 |
|---|---|---|
| Half-open offset convention半开偏移约定 | Empty node, insertion point, final token, EOF空节点、插入点、最后 Token 与文件结束 | Slice source by start and end and compare expected raw text按起止切片源码并比较预期原文 |
| Unicode-aware line and column感知 Unicode 的行列 | Surrogate pairs, combining marks, wide characters, tabs代理对、组合字符、宽字符与制表符 | Cross-check editor position using documented unit and tab width按说明单位与制表宽度交叉检查编辑器位置 |
| Parent and child containment父子包含 | Implicit nodes, normalized operators, detached comments隐式节点、规范化运算符与分离注释 | Define exceptions; assert every ordinary child lies within parent定义例外,并断言普通子节点位于父节点内 |
| Original versus expanded source原始与展开源码 | Template variable expands to multiple tokens or lines模板变量展开为多个 Token 或行 | Map generated range back to one or more original ranges把生成范围映射回一个或多个原始范围 |
| Recovered and missing token恢复与缺失 Token | Parser inserts an expected token that has no source characters解析器插入源码中不存在的预期 Token | Mark synthetic range and insertion point explicitly明确标记合成范围与插入位置 |
Store offset units and indexing base in the API contract. Do not infer them from examples. Verify spans after every preprocessing step and AST normalization. A diagnostic that highlights the wrong identifier can cause a developer to “fix” valid text.
API 契约中应记录偏移单位与索引起点,不能从示例猜测。每个预处理步骤与 AST 规范化后都要验证范围。高亮错误标识符的诊断可能导致开发者“修复”原本有效的文本。
Preserve comments, hints, and whitespace according to purpose按用途保留注释、Hint 与空白
Whitespace and comments are often called trivia because they do not change the core grammar. They can still carry optimizer hints, tool directives, lineage annotations, suppression markers, ticket references, ownership notes, and review rationale. Removing them may change execution or governance even when the AST looks equivalent.
空白与注释常被称为附属文本,因为它们通常不改变核心语法。但它们仍可能包含优化器 Hint、工具指令、血缘注解、抑制标记、工单引用、归属说明与审查理由。即使 AST 看似等价,删除它们也可能改变执行或治理。
| Preservation model保留模型 | Strength优势 | Risk风险 |
|---|---|---|
| Discard trivia丢弃附属文本 | Small tree and simple semantic analysis语法树较小,语义分析简单 | Cannot round-trip comments, formatting, directives, or hints无法往返注释、格式、指令或 Hint |
| Separate token channel独立 Token 通道 | Exact lexical order and source fidelity保持准确词法顺序与源码保真 | Downstream tool must associate trivia with semantic nodes下游工具必须把附属文本关联到语义节点 |
| Leading and trailing attachment前后附加 | Convenient node-based formatting and editing便于基于节点格式化与编辑 | Ambiguous ownership between siblings and delimiters兄弟节点与分隔符之间归属歧义 |
| Standalone comment nodes独立注释节点 | Comments can be queried and transformed explicitly注释可被显式查询与转换 | Tree traversal must distinguish semantic and nonsemantic children树遍历必须区分语义与非语义子节点 |
| Concrete syntax tree具体语法树 | Maximum round-trip and token ownership fidelity最大化往返与 Token 归属保真 | More complex and grammar-coupled analysis分析更复杂且与语法紧耦合 |
Classify hints and directives separately from ordinary prose comments. If an unparser relocates a hint, test whether the target engine still interprets it. Never claim semantic equivalence after comment removal without understanding dialect behavior.
Hint 与指令应和普通说明注释分开分类。如果反解析器移动 Hint,应测试目标引擎是否仍按原意解释。不了解方言行为时,不能在删除注释后声称语义等价。
Parse scripts with grammar-aware statement boundaries使用感知语法的边界解析脚本
Splitting on every semicolon fails when semicolons appear inside strings, comments, procedural bodies, dynamic SQL, or vendor constructs. Some clients support custom delimiters, batch separators, meta-commands, includes, or notebook cell boundaries. The script parser must distinguish server SQL from client directives and report incomplete trailing input.
简单按每个分号切分会在字符串、注释、过程体、动态 SQL 或厂商结构中失败。部分客户端支持自定义分隔符、批次分隔符、元命令、包含文件或笔记本单元边界。脚本解析器必须区分服务器 SQL 与客户端指令,并报告不完整尾部输入。
| Boundary case边界案例 | Required result必要结果 | Failure mode失败模式 |
|---|---|---|
| Semicolon in string or comment字符串或注释中的分号 | Remain inside one statement token stream保持在同一语句 Token 流中 | False extra statement and misleading error cascade产生虚假额外语句与误导性错误级联 |
| Procedural body过程体 | Respect body quoting and nested language grammar尊重过程体引用与嵌套语言语法 | Body fragments parsed as top-level SQL过程体片段被当作顶层 SQL |
| Custom delimiter自定义分隔符 | Track directive scope and delimiter changes跟踪指令作用域与分隔符变化 | Old delimiter remains active or directive reaches server tree旧分隔符仍生效,或指令进入服务器语法树 |
| Batch separator批次分隔符 | Represent batch boundaries and repetition semantics表示批次边界与重复语义 | Identifier mistaken for separator inside a statement语句内标识符被误认成分隔符 |
| Incomplete final statement不完整末尾语句 | Report expected continuation at EOF with partial tree在 EOF 报告预期后续,并提供部分树 | Silently discard tail or attach it to previous statement静默丢弃尾部或附加到前一语句 |
Return statement and batch ranges, delimiter metadata, client-directive nodes, and an explicit completeness flag. A formatter or policy engine must know whether it received one complete statement, a script, or a fragment.
输出应包含语句与批次范围、分隔符元数据、客户端指令节点与显式完整性标志。格式化器或策略引擎必须知道收到的是完整单语句、脚本还是片段。
Return structured diagnostics that identify cause and location返回能够识别原因与位置的结构化诊断
A useful syntax diagnostic is more than “parse failed.” It identifies an unexpected token or input region, expected token classes or grammar context, severity, stage, source range, related ranges, recovery action, and whether a tree remains usable. The human message should be derived from structured fields so interfaces can localize, sort, suppress, and test it.
有用语法诊断不只是“解析失败”,还应指出意外 Token 或输入区域、预期 Token 类别或语法上下文、严重级别、阶段、源码范围、相关范围、恢复动作,以及语法树是否仍可用。人类消息应由结构化字段生成,使界面能够本地化、排序、抑制与测试。
| Diagnostic field诊断字段 | Purpose目的 | Quality check质量检查 |
|---|---|---|
| Stable code and stage稳定代码与阶段 | Automation, suppression, metrics, documentation自动化、抑制、指标与文档 | Code does not change with localized message text代码不随本地化消息变化 |
| Primary source range主要源码范围 | Highlight the smallest responsible region or insertion point高亮最小责任区域或插入点 | Unicode, tabs, EOF, templates, and recovered tokens测试 Unicode、制表符、EOF、模板与恢复 Token |
| Unexpected and expected意外与预期 | Explain grammar mismatch and support completion解释语法不匹配并支持补全 | Expected set is relevant, bounded, and dialect-aware预期集合相关、有界且感知方言 |
| Context path上下文路径 | Show statement, clause, expression, and nested construct显示语句、子句、表达式与嵌套结构 | Does not expose internal parser stack as user guidance不会把内部解析栈直接当作用户指导 |
| Recovery and certainty恢复与确定性 | Tell downstream tools what was inserted, deleted, skipped, or guessed告知下游插入、删除、跳过或猜测了什么 | Recovered regions are explicit and never reported as clean恢复区域明确,绝不报告为干净解析 |
| Related information相关信息 | Point to opening delimiter, paired clause, previous declaration指向起始分隔符、配对子句或前一声明 | Ranges remain valid after source mapping源码映射后范围仍有效 |
Do not promise the one true fix. After an unexpected token, several repairs may be valid. Offer grammar-grounded suggestions and label them as candidates. Preserve the raw parser error for debugging while presenting a concise user message.
不要承诺唯一正确修复。遇到意外 Token 后可能存在多种合法修复。建议应基于语法,并明确标为候选。保留原始解析错误用于调试,同时展示简洁用户消息。
Make recovered trees visibly different from clean trees让恢复语法树与干净语法树明显不同
Editors need partial trees while a user is typing; batch validators may prefer fail-fast behavior. Recovery strategies can insert an expected token, delete an unexpected token, skip to a synchronization point, wrap raw text in an error node, or choose one branch of an ambiguity. Recovery enables continued analysis but introduces parser-created structure not present in the source.
编辑器在用户输入时需要部分语法树,批量验证器则可能更适合快速失败。恢复策略可以插入预期 Token、删除意外 Token、跳到同步点、把原始文本包进错误节点,或在歧义中选择一个分支。恢复使分析得以继续,却会引入源码中不存在的解析器结构。
| Recovery action恢复动作 | Benefit好处 | Required marker必要标记 |
|---|---|---|
| Insert missing token插入缺失 Token | Continue a common incomplete construct继续常见不完整结构 | Synthetic token, insertion offset, expected kind, diagnostic合成 Token、插入偏移、预期类别与诊断 |
| Delete unexpected token删除意外 Token | Recover from stray punctuation or duplicated keyword从多余标点或重复关键字恢复 | Skipped source range retained as error trivia or node跳过源码范围保留为错误附属文本或节点 |
| Skip to synchronization point跳到同步点 | Avoid cascades and parse later clauses or statements避免级联并解析后续子句或语句 | Omitted range, synchronization token, incomplete ancestor flags省略范围、同步 Token 与不完整祖先标志 |
| Error or unknown node错误或未知节点 | Preserve raw fragment and tree position保留原始片段与树中位置 | Distinct node kind, raw text, expected role, uncertainty独立节点类别、原文、预期角色与不确定性 |
| Best-effort dialect fallback尽力方言回退 | Analyze unsupported vendor construct approximately近似分析不支持的厂商结构 | Fallback grammar, unsupported warning, affected subtree回退语法、不支持警告与受影响子树 |
Downstream security, lineage, and rewrite systems should default to refusing consequential conclusions from recovered regions. A formatter may print them, and an editor may navigate them, but a policy engine must not treat guessed structure as proven syntax.
下游安全、血缘与重写系统默认不应从恢复区域得出重大结论。格式化器可以打印它们,编辑器可以导航它们,但策略引擎不能把猜测结构当作已证明语法。
Measure incremental parsing by correctness before latency增量解析应先衡量正确性,再衡量延迟
Interactive tools reparse as a user types. Incremental parsers reuse unchanged tokens or subtrees, while simpler systems reparse the whole document. Reuse reduces latency but requires valid invalidation when an edit changes lexical mode, delimiter balance, statement boundary, alias scope, or dialect interpretation far from the cursor.
交互工具会在用户输入时持续重解析。增量解析器复用未变化 Token 或子树,而简单系统重解析整个文档。复用降低延迟,却要求在编辑改变词法模式、分隔符平衡、语句边界、别名作用域或远处方言解释时正确失效。
Insert, delete, replace, paste, undo, and redo at beginning, middle, and end.
在开头、中间与结尾测试插入、删除、替换、粘贴、撤销与重做。
Change an opening quote, comment marker, dollar tag, or multi-token operator.
改变起始引号、注释标记、美元标签或多 Token 运算符。
Move delimiter, parenthesis, CTE boundary, alias, or nested query terminator.
移动分隔符、括号、CTE 边界、别名或嵌套查询终止符。
Define which unchanged nodes retain stable identity and which must be recreated.
定义哪些未变节点保留稳定身份,哪些必须重建。
Incremental result must equal a clean full parse after every edit sequence.
每个编辑序列后,增量结果必须等于干净完整解析。
Track median and tail latency, allocations, reused nodes, and cancellation.
跟踪中位与尾延迟、分配、复用节点与取消。
Build a differential test that applies random and recorded editor operations, compares the incremental tree and diagnostics with a fresh full parse, and checks source spans. Fast stale trees are worse than slower correct trees.
应构建差分测试:应用随机与记录的编辑操作,把增量语法树和诊断与全新完整解析比较,并检查源码范围。快速但过期的语法树比稍慢但正确的树更糟。
Preserve identifier, parameter, and literal distinctions保留标识符、参数与字面值之间的区别
Many downstream errors begin when a parser normalizes away distinctions that matter later. An identifier needs original spelling, quote style, quote escapes, case-folding status, and qualification parts. A parameter needs marker style, position or name, and source range. A literal needs raw text, literal kind, prefixes, and decoded value without losing precision or executing conversion logic unsafely.
许多下游错误始于解析器过度规范化,丢失后续重要区别。标识符需要原始拼写、引用样式、引用转义、大小写折叠状态与限定部分。参数需要标记样式、位置或名称以及源码范围。字面值需要原始文本、类型、前缀与解码值,同时不能丢失精度或不安全执行转换逻辑。
| Construct结构 | Preserve保留 | Do not assume不要假定 |
|---|---|---|
| Unquoted identifier未引用标识符 | Raw spelling, normalized lookup form, dialect case rule原始拼写、规范查找形式与方言大小写规则 | Displayed case equals catalog identity显示大小写等于目录身份 |
| Quoted identifier引用标识符 | Delimiter, escaped delimiters, exact content, quote flag分隔符、转义分隔符、准确内容与引用标志 | Quotes can be removed during formatting格式化时可移除引号 |
| Qualified name限定名称 | Each component, separators, wildcard or omitted component每个组成、分隔符、通配符或省略组成 | Number of parts maps universally to server, database, schema, table组成数量普遍映射到服务器、数据库、模式与表 |
| Parameter marker参数标记 | Style, ordinal or name, repeated identity, surrounding cast样式、序号或名称、重复身份与周围转换 | Every colon or question mark is a parameter每个冒号或问号都是参数 |
| String or numeric literal字符串或数字字面值 | Raw lexeme, kind, prefix, escape policy, arbitrary precision representation原始词素、类别、前缀、转义策略与任意精度表示 | Host-language number or string preserves target semantics宿主语言数字或字符串保留目标语义 |
Redaction should replace literal presentation after parsing, not mutate the AST used for analysis unless the transformation is explicit and type-aware. A redacted tree is a different artifact and needs its own provenance.
脱敏应在解析后替换字面值展示,不应静默修改分析使用的 AST,除非转换显式且感知类型。脱敏语法树是不同制品,需要自己的来源记录。
Measure coverage beyond simple SELECT statements衡量简单 SELECT 之外的覆盖
A parser can advertise a dialect while supporting only common queries. Real repositories contain DDL, DML, MERGE, COPY or load commands, grants, comments, transactions, session settings, explain commands, procedural functions, triggers, dynamic SQL, scripting variables, warehouse clauses, hints, semi-structured paths, and vendor extensions. Coverage must be stated by construct and version.
解析器可能宣称支持某方言,却只支持常见查询。真实仓库还包含 DDL、DML、MERGE、COPY 或加载命令、授权、注释、事务、会话设置、执行计划命令、过程函数、触发器、动态 SQL、脚本变量、仓库子句、Hint、半结构化路径与厂商扩展。覆盖必须按结构与版本说明。
| Coverage level覆盖级别 | Meaning含义 | Evidence证据 |
|---|---|---|
| Recognized可识别 | Lexer and grammar accept the construct词法器与语法接受该结构 | Fixture parses cleanly in strict target mode夹具在严格目标模式下干净解析 |
| Structured已结构化 | AST exposes meaningful typed nodes and propertiesAST 暴露有意义的类型化节点与属性 | Node assertions for all material clauses所有关键子句的节点断言 |
| Source-faithful源码保真 | Spans, comments, hints, quotes, and raw fragments are preserved范围、注释、Hint、引用与原始片段得到保留 | Token and span checks plus lossless or defined-loss round tripToken 与范围检查,以及无损或定义损失往返 |
| Analyzable可分析 | Visitors, name collectors, policies, and rewrites understand it访问器、名称收集、策略与重写理解该结构 | Downstream semantic tests, not just parser snapshots下游语义测试,而不只是解析快照 |
| Round-trippable可往返 | Unparser can emit accepted target SQL with defined fidelity反解析器可按定义保真输出目标可接受 SQL | Parse–unparse–parse structural and execution-aware checks解析—反解析—再解析结构与执行感知检查 |
Publish unsupported constructs and fallback behavior. Returning a generic raw node can be acceptable when clearly marked; silently coercing a vendor clause into a superficially similar standard node is more dangerous because downstream tools assume understanding.
应公开不支持结构与回退行为。若明确标记,返回通用原始节点可以接受;把厂商子句静默强制成表面相似的标准节点更危险,因为下游工具会假定解析器理解其含义。
Evaluate AST traversal with scope and context在作用域与上下文中评价 AST 遍历
A tree visitor can collect every table-looking or column-looking node, but the same syntax has different roles in CTE definitions, subqueries, aliases, function arguments, windows, insert targets, update assignments, merge branches, and correlated references. Useful traversal APIs expose parent, child role, statement, clause, scope boundary, and source range.
语法树访问器可以收集每个看起来像表或列的节点,但相同语法在 CTE 定义、子查询、别名、函数参数、窗口、插入目标、更新赋值、MERGE 分支与相关引用中扮演不同角色。有用遍历 API 应暴露父节点、子角色、语句、子句、作用域边界与源码范围。
| Traversal task遍历任务 | Context required所需上下文 | Naive failure朴素失败 |
|---|---|---|
| Collect relation references收集关系引用 | CTE names, aliases, subquery boundaries, table functions, targetsCTE 名、别名、子查询边界、表函数与目标 | CTE reported as physical table or write target missedCTE 被报告为物理表,或遗漏写入目标 |
| Collect column references收集列引用 | Qualifier, alias visibility, star, nested fields, scope限定符、别名可见性、星号、嵌套字段与作用域 | Alias definition mistaken for input column别名定义被误认成输入列 |
| Classify statement effects分类语句影响 | Top-level and nested statements, procedures, functions, external commands顶层与嵌套语句、过程、函数与外部命令 | SELECT wrapper hides mutation or side-effecting callSELECT 外壳隐藏变更或副作用调用 |
| Find predicates查找条件 | WHERE, JOIN, HAVING, FILTER, QUALIFY, policy, merge branchWHERE、JOIN、HAVING、FILTER、QUALIFY、策略与 MERGE 分支 | Filter role and row stage collapsed together筛选角色与作用阶段被混为一谈 |
| Rewrite identifiers重写标识符 | Definition versus reference, quote semantics, namespace, source map定义与引用、引用语义、命名空间与源码映射 | Renames unrelated alias or breaks quoting重命名无关别名或破坏引用 |
Provide typed child roles instead of relying only on list position. Visitors should be able to skip recovered or unknown subtrees, preserve traversal order, and report uncertainty. Test nested scopes and shadowed names even if the parser itself does not bind them.
应提供类型化子角色,而不是只依赖列表位置。访问器应能跳过恢复或未知子树、保持遍历顺序并报告不确定性。即使解析器本身不绑定名称,也要测试嵌套作用域与名称遮蔽。
Do not confuse table-shaped nodes with resolved catalog objects不要把表形节点与已解析目录对象混淆
The AST can say that the source contains a relation reference
named orders. Without a catalog and scope rules, it
cannot prove whether that name refers to a CTE, local temporary
table, view, materialized view, synonym, table function, remote
object, or physical table; nor which database and schema resolve
an unqualified name.
AST 可以说明源码包含名为
orders
的关系引用,但没有目录与作用域规则时,无法证明它指向
CTE、本地临时表、视图、物化视图、同义词、表函数、远程对象还是物理表,也无法确定未限定名称解析到哪个数据库和模式。
| Question问题 | Parser-only answer仅解析器答案 | Additional context额外上下文 |
|---|---|---|
| Is this identifier a table?这个标识符是表吗? | It occupies a relation-reference grammar position它位于关系引用语法位置 | CTE scope, aliases, catalog object types, table functionsCTE 作用域、别名、目录对象类型与表函数 |
Which column does id mean?id 指哪个列?
|
It is an unqualified column-reference node它是未限定列引用节点 | Visible relations, projection aliases, lateral and correlation rules可见关系、投影别名、横向与相关规则 |
| Is this function safe?这个函数安全吗? | It is a call with a name and arguments它是带名称与参数的调用 | Function registry, overload resolution, volatility, privileges, extensions函数注册表、重载解析、易变性、权限与扩展 |
| What type is the expression?表达式是什么类型? | Its syntactic operator and literal forms are known已知其语法运算符与字面值形式 | Column types, function signatures, coercion, parameter types, settings列类型、函数签名、强制转换、参数类型与设置 |
| Is the query read-only?查询是只读吗? | Top-level statement and nested syntactic forms are known已知顶层语句与嵌套语法形式 | Function side effects, procedures, external sources, engine semantics函数副作用、过程、外部来源与引擎语义 |
Downstream APIs should name syntax references differently from resolved symbols. Preserve the original node and attach resolution separately with catalog version, scope, identity, and confidence. This prevents stale binding from being mistaken for parser truth.
下游 API 应把语法引用与已解析符号使用不同名称。保留原始节点,并把解析结果以目录版本、作用域、身份与置信信息单独附加,防止过期绑定被误认为解析器事实。
Use the AST as lineage input, not lineage proof把 AST 当作血缘输入,而不是血缘证明
Parsing reveals syntactic data flow: sources, projections, aliases, joins, expressions, insert targets, and nested statements. Complete lineage also needs name resolution, star expansion, view and routine definitions, dynamic SQL, macros, temporary objects, external tables, session defaults, policy rewrites, runtime branches, and engine-specific semantics. Recovered or unsupported syntax creates explicit gaps.
解析揭示语法数据流:来源、投影、别名、连接、表达式、插入目标与嵌套语句。完整血缘还需要名称解析、星号展开、视图与例程定义、动态 SQL、宏、临时对象、外部表、会话默认值、策略重写、运行分支与引擎特定语义。恢复或不支持语法会形成明确缺口。
Parser can enumerate syntax sources and targets, but must distinguish CTEs, aliases, and physical relations.
解析器可枚举语法来源与目标,但必须区分 CTE、别名与物理关系。
Requires resolved columns, star expansion, expression semantics, set alignment, and nested scope.
需要解析列、星号展开、表达式语义、集合对齐与嵌套作用域。
Dynamic SQL, conditional scripts, stored procedures, generated names, and external systems may only resolve during execution.
动态 SQL、条件脚本、存储过程、生成名称与外部系统可能只能运行时解析。
Every edge should record source stage, catalog version, unresolved nodes, and whether it is syntactic, resolved, or observed.
每条边应记录来源阶段、目录版本、未解析节点,以及它是语法、已解析还是观察所得。
Never turn an incomplete parse into a complete-looking lineage graph by omitting unknown regions. Show gaps, affected outputs, and the evidence needed to resolve them.
不能通过省略未知区域,把不完整解析转换成看似完整的血缘图。应显示缺口、受影响输出与解决缺口所需证据。
Test parse–unparse–parse at syntax and semantic levels在语法与语义层测试解析—反解析—再解析
An unparser or formatter turns a tree back into SQL. The result may intentionally normalize keyword case, indentation, parentheses, aliases, identifier quotes, or literal spelling. The critical question is which changes are allowed and whether the output reparses under the same dialect into an equivalent structure without losing hints, comments, or unsupported fragments.
反解析器或格式化器把语法树重新转换成 SQL,可能有意规范关键字大小写、缩进、括号、别名、标识符引用或字面值拼写。关键问题是允许哪些变化,以及输出能否在相同方言下重新解析为等价结构,同时不丢失 Hint、注释或不支持片段。
| Fidelity level保真级别 | Required equality必要相等 | Use case用途 |
|---|---|---|
| Lossless text无损文本 | Original bytes or characters reproduced exactly原始字节或字符完全重现 | Archival, minimal edits, source-preserving tools归档、最小编辑与源码保留工具 |
| Trivia-preserving syntax保留附属文本的语法 | Tokens, comments, hints, and structure preserved; whitespace may normalizeToken、注释、Hint 与结构保留;空白可规范 | Formatter and review-friendly rewrite格式化与便于审查的重写 |
| Structural AST结构 AST | Reparsed semantic tree equivalent under defined normalization重新解析语义树在定义规范化下等价 | Canonicalization, diff, cache keys, analysis规范化、差异、缓存键与分析 |
| Validated semantics已验证语义 | Resolved objects, types, effects, and selected invariants equivalent解析对象、类型、影响与所选不变量等价 | Refactoring and dialect-aware transformation重构与感知方言转换 |
| Observed behavior观察行为 | Controlled execution gives equivalent typed outcomes and side effects受控执行产生等价类型化结果与副作用 | High-risk rewrite verification in disposable environment一次性环境中的高风险重写验证 |
Structural equality needs a documented normalizer: alias omission, redundant parentheses, literal forms, commutative expressions, and clause ordering cannot be treated casually. A parser round trip proves its own representation consistency, not universal query equivalence.
结构相等需要有文档的规范器:别名省略、冗余括号、字面值形式、可交换表达式与子句顺序不能随意处理。解析往返证明的是自身表示一致性,而不是普遍查询等价。
Treat dialect transpilation as a typed, lossy transformation把方言转译视为有类型且可能有损的转换
A multi-dialect parser can normalize source syntax and emit a target dialect, but syntax similarity does not guarantee semantic equivalence. Functions, types, null behavior, date arithmetic, identifier resolution, collation, regex, arrays, JSON, intervals, merge semantics, transactions, and optimizer hints may lack exact mappings. Some transformations require schema and type information unavailable to the parser.
多方言解析器可以规范源语法并输出目标方言,但语法相似不保证语义等价。函数、类型、空值行为、日期运算、标识符解析、排序规则、正则、数组、JSON、间隔、MERGE 语义、事务与优化器 Hint 可能没有精确映射。部分转换需要解析器无法获得的模式与类型信息。
| Outcome结果 | Meaning含义 | Required disclosure必要披露 |
|---|---|---|
| Exact supported mapping精确支持映射 | Construct has a documented target equivalent under stated context结构在声明上下文中拥有文档化目标等价 | Source and target dialect/version, rule, tests源与目标方言及版本、规则与测试 |
| Context-dependent mapping上下文依赖映射 | Requires types, schema, settings, or business assumption需要类型、模式、设置或业务假设 | Required context and behavior when absent所需上下文与缺失时行为 |
| Approximation近似 | Output is usable but may change precision, edge behavior, or performance输出可用,但可能改变精度、边界行为或性能 | Loss description, affected node, severity, alternative损失说明、受影响节点、严重度与替代方案 |
| Unsupported不支持 | No trustworthy target representation没有可信目标表示 | Hard error or explicit preserved raw fragment硬错误或显式保留原始片段 |
| Dropped silently静默丢弃 | Source property disappears without evidence源属性在没有证据时消失 | Unacceptable for trustworthy transformation对可信转换不可接受 |
The SQLGlot project documentation describes parser errors, AST inspection, custom dialects, and unsupported transpilation behavior, including cases that need schema information. Regardless of library, configure unsupported mappings to fail or surface prominently for consequential use.
SQLGlot 项目文档描述了解析错误、AST 检查、自定义方言与不支持转译行为,包括需要模式信息的情况。无论使用哪个库,重大用途都应让不支持映射失败或显著展示。
Parse untrusted SQL without turning analysis into execution解析不受信 SQL,但不能把分析变成执行
Parsing should not contact a database or execute functions, but it still processes attacker-controlled input. Deep nesting, huge token counts, pathological ambiguity, long literals, recursive grammar paths, comment nesting, generated diagnostics, AST serialization, extension hooks, and downstream visitors can consume CPU, memory, stack, disk, or logs. Dependencies and native bindings add their own attack surface.
解析不应联系数据库或执行函数,但仍会处理攻击者控制输入。深层嵌套、海量 Token、病态歧义、超长字面值、递归语法路径、嵌套注释、生成诊断、AST 序列化、扩展 Hook 与下游访问器都可能消耗 CPU、内存、栈、磁盘或日志。依赖与原生绑定还会增加攻击面。
| Control控制项 | Protects against防护风险 | Evidence证据 |
|---|---|---|
| Byte and character limit字节与字符限制 | Oversized input, decoding and logging amplification超大输入、解码与日志放大 | Reject before allocation with structured size diagnostic分配前拒绝,并返回结构化大小诊断 |
| Token, statement, and nesting limitsToken、语句与嵌套限制 | Huge trees, stack exhaustion, excessive ambiguity巨大语法树、栈耗尽与过度歧义 | Boundary tests at, below, and above every limit每个限制的低于、等于与高于边界测试 |
| Time and cancellation budget时间与取消预算 | Pathological parse, stalled worker, batch starvation病态解析、工作线程停滞与批处理饥饿 | Hard deadline, cooperative or process cancellation, no orphan work硬截止、协作或进程取消且无孤儿工作 |
| Process or worker isolation进程或工作线程隔离 | Crash, memory leak, native bug, untrusted extension崩溃、内存泄漏、原生缺陷与不受信扩展 | Resource cap, restart, health check, sanitized IPC资源上限、重启、健康检查与脱敏 IPC |
| No resolution or execution callback禁止解析或执行回调 | Network access, file access, function execution, secret retrieval网络访问、文件访问、函数执行与密钥取回 | Architecture and tests prove parse-only dependency graph架构与测试证明纯解析依赖图 |
| Pinned and reviewed dependencies固定并审查依赖 | Supply-chain changes and vulnerable parser runtimes供应链变化与存在漏洞的解析运行时 | Lockfile, integrity, notices, update and rollback process锁文件、完整性、公告、更新与回退流程 |
Do not log full untrusted SQL by default. Diagnostics can include secrets or personal data in context windows. Use content hashes, bounded redacted excerpts, secure access, retention limits, and deletion procedures.
默认不要记录完整不受信 SQL。诊断上下文可能包含密钥或个人数据。应使用内容哈希、有界脱敏片段、安全访问、保留限制与删除流程。
Benchmark realistic syntax, errors, and tail latency用真实语法、错误与尾延迟进行基准测试
A benchmark of short valid SELECT statements says little about migration files, deeply nested views, procedural definitions, malformed editor buffers, multi-megabyte generated SQL, or mixed-dialect repositories. Measure throughput and latency by corpus class, input length, token count, node count, nesting, error count, recovery mode, comments, and source-map requirements.
短小合法 SELECT 基准无法代表迁移文件、深层嵌套视图、过程定义、畸形编辑缓冲区、多 MB 生成 SQL 或混合方言仓库。应按语料类别、输入长度、Token 数、节点数、嵌套、错误数、恢复模式、注释与源码映射需求衡量吞吐与延迟。
| Metric指标 | Segment by分段维度 | Why it matters重要原因 |
|---|---|---|
| Median and p95/p99 latency中位与 p95/p99 延迟 | Valid, incomplete, invalid, recovered, dialect, size有效、不完整、无效、恢复、方言与大小 | Interactive experience and batch capacity depend on tails交互体验与批处理容量取决于尾部 |
| Peak and retained memory峰值与保留内存 | Tokens, AST, trivia, diagnostics, serialization, repeated parseToken、AST、附属文本、诊断、序列化与重复解析 | Leaks and large trees can exhaust services or editors泄漏与大树会耗尽服务或编辑器 |
| Allocations and node reuse分配与节点复用 | Full parse versus incremental edit classes完整解析与增量编辑类别 | Explains latency and garbage-collection pressure解释延迟与垃圾回收压力 |
| Cancellation latency取消延迟 | Tokenizer, grammar, recovery, post-processing phase词法、语法、恢复与后处理阶段 | A timeout is ineffective if work ignores cancellation如果工作忽略取消,超时就无效 |
| Correctness under load负载下正确性 | Parallel parses, parser reuse, dialect switching, cancellation并行解析、解析器复用、方言切换与取消 | Shared mutable state can corrupt trees and diagnostics共享可变状态会破坏语法树与诊断 |
Pin runtime and hardware, warm-up policy, garbage collector, parser options, input corpus hash, and output mode. A parser that discards spans and comments should not be compared directly with one that preserves them without explaining the work difference.
固定运行时与硬件、预热策略、垃圾回收器、解析选项、输入语料哈希与输出模式。丢弃范围与注释的解析器不能在不解释工作差异时,直接与保留它们的解析器比较。
Build a layered parser test corpus构建分层解析器测试语料
Parser quality depends on breadth and adversarial depth. Golden AST snapshots are useful but can approve an incorrect tree if reviewed casually. Combine lexical fixtures, grammar examples, vendor documentation, real sanitized repositories, negative syntax cases, recovery cases, round trips, differential tests, fuzzing, metamorphic properties, and downstream behavior tests.
解析器质量取决于覆盖广度与对抗深度。黄金 AST 快照有用,但如果审查随意,也可能批准错误语法树。应组合词法夹具、语法示例、厂商文档、真实脱敏仓库、负面语法案例、恢复案例、往返、差分测试、模糊测试、蜕变性质与下游行为测试。
| Test layer测试层 | Oracle判定基准 | Finds发现 |
|---|---|---|
| Token fixturesToken 夹具 | Exact kinds, raw slices, values, quotes, spans, trivia准确类别、原始切片、值、引用、范围与附属文本 | Lexical boundary and source-location defects词法边界与源码位置缺陷 |
| Grammar positives and negatives语法正负例 | Target dialect accepts valid and rejects invalid constructs目标方言接受合法并拒绝非法结构 | Coverage gaps and overly lenient grammar覆盖缺口与过度宽松语法 |
| AST property assertionsAST 属性断言 | Typed nodes, child roles, spans, quote flags, recovery markers类型化节点、子角色、范围、引用标志与恢复标记 | Semantically wrong but syntactically accepted trees语法接受但语义结构错误的树 |
| Round-trip and metamorphic往返与蜕变测试 | Defined equality after formatting, whitespace, comments, parentheses格式化、空白、注释与括号变化后的定义相等 | Loss, unstable serialization, normalization mistakes损失、不稳定序列化与规范化错误 |
| Differential parsing差分解析 | Compare server parser or independent implementation with reconciled models与服务器解析器或独立实现比较,并对齐模型 | Dialect disagreement and hidden assumptions方言分歧与隐藏假设 |
| Fuzz and resource tests模糊与资源测试 | No crash, hang, unbounded memory, stale state, or unsafe callback无崩溃、卡死、无限内存、过期状态或不安全回调 | Robustness and security defects稳健性与安全缺陷 |
Keep corpus provenance, dialect and version, expected support level, sanitization method, license, and failure history. Add every production parser defect as a minimal regression plus the original sanitized case.
保留语料来源、方言与版本、预期支持级别、脱敏方法、许可证与失败历史。每个生产解析缺陷都应加入最小回归案例与原始脱敏案例。
Use parsing as the first structural gate for generated SQL把解析作为生成 SQL 的第一道结构门
AI-generated SQL should be parsed under the intended dialect before any database interaction. The AST can identify statement count and type, nested mutations, relation-shaped references, functions, joins, predicates, limits, comments, unsupported syntax, parameters, and recovered regions. Those findings support policy and review, but they remain syntactic until enriched with a trusted schema and engine semantics.
AI 生成 SQL 在任何数据库交互前,应按预期方言解析。AST 可以识别语句数量与类型、嵌套变更、关系形引用、函数、连接、条件、限制、注释、不支持语法、参数与恢复区域。这些发现支持策略与审查,但在可信模式与引擎语义增强前仍属于语法层。
Require target engine/version and reject silent generic fallback.
要求目标引擎与版本,拒绝静默通用回退。
Record exact text, parser version, diagnostics, unsupported and recovered regions.
记录准确文本、解析器版本、诊断、不支持与恢复区域。
Count statements and identify top-level and nested operation forms.
统计语句并识别顶层与嵌套操作形式。
Reject disallowed statement kinds, unresolved fragments, or risky constructs.
拒绝不允许语句类别、未解析片段或风险结构。
Resolve names, types, functions, permissions, and semantics using approved metadata.
使用获批元数据解析名称、类型、函数、权限与语义。
Use deterministic fixtures, assertions, read-only boundaries, plans, and limits.
使用确定性夹具、断言、只读边界、计划与限制。
Inspect an NL2SQL candidate before database use数据库使用前先检查 NL2SQL 候选
Use the InfiniSynapse NL2SQL Query Tester with a sanitized schema and explicit target dialect, then keep parsing, schema validation, policy checks, and controlled execution as separate evidence gates.
使用 InfiniSynapse NL2SQL Query Tester 输入脱敏模式与明确目标方言,并把解析、模式验证、策略检查与受控执行保持为独立证据门。
Open NL2SQL Query Tester打开 NL2SQL 查询测试器 Use sanitized SQL and metadata. Never paste credentials, secrets, personal data, or unrestricted production samples. 只使用脱敏 SQL 与元数据。切勿粘贴凭据、密钥、个人数据或未受限生产样本。Score an SQL parser with observable evidence用可观察证据评价 SQL 解析器
Parser choice should follow use cases. An editor needs recovery, incremental correctness, spans, and low tail latency; a migration analyzer needs script and DDL coverage; a policy engine needs strict dialect behavior and explicit unknown nodes; a formatter needs round-trip fidelity; a transpiler needs typed context and loss reporting. Set nonnegotiable gates before weighting convenience.
解析器选择应依据用途。编辑器需要恢复、增量正确性、范围与低尾延迟;迁移分析器需要脚本与 DDL 覆盖;策略引擎需要严格方言行为与显式未知节点;格式化器需要往返保真;转译器需要类型上下文与损失报告。便利性加权前先定义不可妥协门槛。
| Criterion标准 | Weight权重 | Evidence task证据任务 | Gate?门槛? |
|---|---|---|---|
| Target dialect and construct coverage目标方言与结构覆盖 | 18% | Run versioned positive, negative, procedural, DDL, DML, and script corpus运行版本化正负例、过程、DDL、DML 与脚本语料 | Yes是 |
| AST contract and source fidelityAST 契约与源码保真 | 16% | Assert nodes, roles, spans, quotes, trivia, unknown and recovery markers断言节点、角色、范围、引用、附属文本、未知与恢复标记 | Yes是 |
| Diagnostics and recovery honesty诊断与恢复诚实性 | 14% | Inject malformed constructs and verify location, code, expected, recovery, certainty注入畸形结构并验证位置、代码、预期、恢复与确定性 | Yes是 |
| Safety under untrusted input不受信输入安全 | 14% | Exercise size, nesting, token, time, memory, cancellation, fuzz, isolation演练大小、嵌套、Token、时间、内存、取消、模糊与隔离 | Yes是 |
| Round-trip and transformation behavior往返与转换行为 | 12% | Test parse–unparse–parse, comments, hints, quotes, unsupported and losses测试解析往返、注释、Hint、引用、不支持与损失 | Depends视用途 |
| Incremental and concurrency correctness增量与并发正确性 | 10% | Differential editor sequences, parallel dialect switching, cancellation差分编辑序列、并行方言切换与取消 | Depends视用途 |
| Performance and operational fit性能与运营适配 | 8% | Measure realistic median/tails, memory, allocations, serialization, recovery衡量真实中位与尾部、内存、分配、序列化与恢复 | No否 |
| Versioning, API, documentation, maintenance版本、API、文档与维护 | 8% | Upgrade, AST schema diff, regression history, security notices, rollback升级、AST 模式差异、回归历史、安全公告与回退 | No否 |
Score 0–4: absent, claimed, demonstrated once, repeatably verified, or enforced and monitored. Weighting cannot rescue a failed gate. Store corpus hashes, parser and runtime versions, options, output schemas, test results, owners, and dates.
按 0–4 评分:缺失、仅声明、单次演示、可重复验证、强制并监控。加权不能挽救失败门槛。保存语料哈希、解析器与运行时版本、选项、输出模式、测试结果、负责人与日期。
Use a twelve-step trustworthy parsing workflow使用十二步可信解析工作流
State whether the tree supports editing, policy, lineage, formatting, migration, or review.
说明语法树用于编辑、策略、血缘、格式化、迁移还是审查。
Record source, encoding, content hash, original text, preprocessing, and source maps.
记录来源、编码、内容哈希、原始文本、预处理与源码映射。
Specify target engine/version, conformance, lexical policy, extensions, build, and options.
指定目标引擎与版本、一致性、词法策略、扩展、构建与选项。
Set bytes, tokens, statements, nesting, time, memory, diagnostics, and serialization limits.
设置字节、Token、语句、嵌套、时间、内存、诊断与序列化限制。
Preserve kind, raw text, value, quote, trivia, and precise source spans.
保留类别、原文、值、引用、附属文本与精确源码范围。
Report statement boundaries, clean or recovered status, errors, unknown and unsupported regions.
报告语句边界、干净或恢复状态、错误、未知与不支持区域。
Check node schema, child roles, span containment, token coverage, source identity, and recovery markers.
检查节点模式、子角色、范围包含、Token 覆盖、来源身份与恢复标记。
Classify every statement and construct as structured, raw, recovered, unsupported, or omitted.
把每个语句与结构分类为已结构化、原始、恢复、不支持或省略。
Attach binding, types, lineage, effects, and policies with catalog versions and confidence.
以目录版本与置信信息附加绑定、类型、血缘、影响与策略。
Run actual visitors, formatters, policies, diffs, and refactors on hard corpus cases.
在困难语料上运行真实访问器、格式化、策略、差异与重构。
Compare clean full parse, incremental parse, independent parser, and target engine where meaningful.
在有意义时比较干净完整解析、增量解析、独立解析器与目标引擎。
Store versions, options, corpus, AST schema, diagnostics, gaps, metrics, and upgrade evidence.
保存版本、选项、语料、AST 模式、诊断、缺口、指标与升级证据。
Stop consequential analysis when the dialect is unknown, input transformation cannot be mapped, the tree contains unbounded recovered or unknown regions, source spans fail checks, required constructs are unsupported, resource bounds are absent, or downstream logic treats syntax references as resolved facts.
如果方言未知、输入转换无法映射、语法树包含无边界恢复或未知区域、源码范围检查失败、必要结构不支持、缺少资源边界,或下游把语法引用当作已解析事实,就应停止重大分析。
Avoid common SQL parser failure patterns避免常见 SQL 解析器失败模式
| Pattern模式 | Why it fails失败原因 | Better control更好控制 |
|---|---|---|
| Parsed means valid and safe解析成功等于有效且安全 | Grammar says nothing about schema, types, permissions, meaning, or cost语法无法证明模式、类型、权限、含义或成本 | Stage-specific status and separate validation gates阶段特定状态与独立验证门 |
| Generic SQL mode for vendor repositories厂商仓库使用通用 SQL 模式 | Lenient grammar accepts, misclassifies, or drops extensions宽松语法接受、误分类或丢弃扩展 | Pinned dialect/version and explicit unsupported regions固定方言与版本,并显式标明不支持区域 |
| Recovered tree treated as clean恢复语法树被当作干净树 | Inserted and skipped tokens become invented syntax facts插入与跳过 Token 变成虚构语法事实 | Recovery markers, certainty, and downstream refusal policy恢复标记、确定性与下游拒绝策略 |
| Regex statement splitting正则切分语句 | Strings, comments, procedures, delimiters, and client commands break it字符串、注释、过程、分隔符与客户端命令会破坏它 | Dialect-aware lexer and script grammar感知方言词法器与脚本语法 |
| AST snapshot as only test只测试 AST 快照 | Review can approve stable but incorrect structure and misses downstream impact审查可能批准稳定但错误结构,并遗漏下游影响 | Property, round-trip, differential, fuzz, and consumer tests属性、往返、差分、模糊与消费者测试 |
| Silent best-effort transpilation静默尽力转译 | Unsupported semantics disappear behind valid-looking target SQL不支持语义隐藏在看似合法目标 SQL 后 | Loss report, hard failure, typed context, controlled equivalence tests损失报告、硬失败、类型上下文与受控等价测试 |
Frequently asked questions about SQL parsers关于 SQL 解析器的常见问题
What is an SQL parser?什么是 SQL 解析器?
It tokenizes SQL text and applies a selected dialect grammar to produce a structured syntax tree or syntax errors. It should not execute the statement.
它把 SQL 文本 Token 化,并应用所选方言语法,产生结构化语法树或语法错误,不应执行语句。
Does successful parsing mean a query is valid?解析成功意味着 SQL 查询有效吗?
No. It proves grammar only. Object existence, name resolution, types, functions, permissions, policies, meaning, plan, resources, and results require later stages.
不。它只证明语法。对象存在、名称解析、类型、函数、权限、策略、含义、计划、资源与结果需要后续阶段。
Why does the dialect matter?为什么 SQL 解析器必须知道数据库方言?
Dialects differ in keywords, quoting, operators, functions, parameters, types, statements, procedural syntax, precedence, and extensions. The same text may parse differently or fail.
方言在关键字、引用、运算符、函数、参数、类型、语句、过程语法、优先级与扩展上不同。同一文本可能解析不同或失败。
What is an SQL AST?什么是 SQL AST?
It is a tree of meaningful constructs such as statements, projections, relations, joins, predicates, expressions, groups, windows, and order. Its contract determines which punctuation, comments, spelling, and source locations remain.
它是语句、投影、关系、连接、条件、表达式、分组、窗口与排序等有意义结构的树。其契约决定保留哪些标点、注释、拼写与源码位置。
Can a parser determine SQL lineage?SQL 解析器能确定完整数据血缘吗?
It provides syntax inputs, but complete lineage also needs catalog binding, star expansion, view and routine definitions, dynamic SQL, temporary state, engine semantics, and explicit gaps.
它提供语法输入,但完整血缘还需要目录绑定、星号展开、视图与例程定义、动态 SQL、临时状态、引擎语义与显式缺口。
Is parsing untrusted SQL safe?解析不受信任的 SQL 安全吗?
It avoids database execution but still needs input, nesting, token, time, memory, cancellation, isolation, dependency, and logging controls to resist denial of service and implementation defects.
它避免数据库执行,但仍需要输入、嵌套、Token、时间、内存、取消、隔离、依赖与日志控制,以抵御拒绝服务和实现缺陷。
Final SQL parser readiness checklistSQL 解析器最终就绪清单
Encoding, original and expanded text, source identity, script boundaries, preprocessing, and source maps are explicit.
编码、原始与展开文本、来源身份、脚本边界、预处理与源码映射明确。
Target engine/version, conformance, lexical rules, extensions, parser build, and recovery policy are pinned.
目标引擎与版本、一致性、词法规则、扩展、解析器构建与恢复策略固定。
Node schema, child roles, source spans, quote metadata, trivia, unknown, unsupported, and recovered regions are documented.
节点模式、子角色、源码范围、引用元数据、附属文本、未知、不支持与恢复区域已有文档。
Parsing, binding, validation, lineage, policy, planning, transformation, and execution results are labeled separately.
解析、绑定、验证、血缘、策略、规划、转换与执行结果分别标记。
Untrusted input limits, timeouts, cancellation, isolation, dependency integrity, redaction, retention, and no-execution boundaries are tested.
不受信输入限制、超时、取消、隔离、依赖完整性、脱敏、保留与禁止执行边界已测试。
Versioned corpora cover tokens, grammar, errors, recovery, source fidelity, round trip, downstream consumers, differential behavior, fuzzing, and performance tails.
版本化语料覆盖 Token、语法、错误、恢复、源码保真、往返、下游消费者、差分行为、模糊测试与性能尾部。
A trustworthy SQL parser does not merely produce a tree. It tells downstream systems exactly which language it recognized, which source regions support each node, where it recovered or guessed, what it intentionally discarded, what it cannot understand, and which claims still require schema or runtime evidence.
可信 SQL 解析器不只是生成一棵树,还会告诉下游:识别了哪种语言、每个节点由哪些源码区域支持、在哪里恢复或猜测、主动丢弃了什么、无法理解什么,以及哪些声明仍需要模式或运行时证据。