Downloads 21.7kthis week

GitHub stars 76

Extension descriptor (YAML)

Comprehensive processing extension for web markup languages (XML and HTML) with SAX streaming for large files, intelligent schema inference, XPath-based data extraction, and HTML table parsing.

Installing and Loading

INSTALL webbed FROM community;
LOAD webbed;

Example

-- Load the extension
LOAD webbed;

-- Read HTML files directly as structured document blocks
SELECT * FROM read_html_blocks('index.html');

-- Read XML files directly into tables
SELECT * FROM 'data.xml';
SELECT * FROM read_xml('config/*.xml');

-- Parse and extract from XML content using XPath
SELECT xml_extract_text('<book><title>Database Guide</title></book>', '//title');
-- Result: "Database Guide"

-- Parse and extract from HTML content
SELECT html_extract_text('<html><body><h1>Welcome</h1></body></html>', '//h1');
-- Result: "Welcome"

-- Extract HTML tables directly into DuckDB
SELECT * FROM html_extract_tables('<table><tr><th>Name</th><th>Age</th></tr><tr><td>John</td><td>25</td></tr></table>');

-- Extract links and images from HTML pages
SELECT html_extract_links('<a href="https://example.com">Click here</a>');
SELECT html_extract_images('<img src="photo.jpg" alt="Photo" width="800">');

-- Parse XML strings directly (without files)
SELECT * FROM parse_xml('<books><book><title>DuckDB</title><price>29.99</price></book></books>');

-- Convert between XML and JSON formats
SELECT xml_to_json('<person><name>John</name><age>30</age></person>');
SELECT json_to_xml('{"name":"John","age":"30"}');

-- Convert HTML to structured document blocks
SELECT html_to_duck_blocks('<h1>Title</h1><p>Paragraph with <strong>bold</strong> text.</p>');

About webbed

DuckDB XML is a comprehensive extension that brings powerful XML and HTML processing capabilities to DuckDB, enabling SQL-native analysis of structured documents. The extension provides four core areas of functionality:

XML Processing & Analysis: Parse, validate, and extract data from XML documents using full XPath 1.0 expressions. Functions include xml_extract_text(), xml_extract_elements(), xml_extract_attributes(), xml_valid(), and xml_stats() for comprehensive document analysis. The extension handles namespaces with intelligent 'auto' mode detection, comments, CDATA sections, and provides utilities like xml_pretty_print() and xml_minify().

HTML Processing & Web Scraping: Advanced HTML parsing capabilities with specialized functions for web data extraction. Extract text content with html_extract_text(), parse HTML tables into structured data with html_extract_tables(), extract links with metadata using html_extract_links(), and extract images with attributes using html_extract_images(). Perfect for web scraping and HTML document analysis workflows.

Document Block Processing: Convert HTML documents to and from structured block representations with read_html_blocks(), parse_html_blocks(), html_to_duck_blocks(), and duck_blocks_to_html(). Supports paragraphs, headings, code blocks, tables, lists, inline formatting (bold, italic, links, etc.), and frontmatter preservation. Integrates with the duck_block_utils extension for full Markdown support.

Smart Schema Inference & File Reading: Automatically flatten XML/HTML documents into relational tables with intelligent type detection for dates, numbers, booleans, and nested structures. Functions like read_xml() and read_html() provide direct file-to-table conversion with configurable options including datetime_format presets, nullstr for custom NULL values, and schema customization. SAX streaming (v2.0) automatically handles files over 16MB using a push parser that processes XML in 64KB chunks — peak memory stays proportional to a single record, not the file size.

Key XML Functions:

• read_xml(pattern) - Read XML files with automatic schema inference
• xml_extract_text(xml, xpath) - XPath-based text extraction
• xml_extract_elements(xml, xpath) - Extract structured elements
• xml_extract_attributes(xml, xpath) - Extract attributes as structs
• xml_to_json(xml) / json_to_xml(json) - Format conversions
• xml_stats(xml) - Document statistics and analysis
• xml_validate_schema(xml, xsd) - XSD schema validation
• xml_find_undefined_prefixes(xml, xpath) - Detect undeclared namespace prefixes
• xml_add_namespace_declarations(xml, map) - Inject namespace declarations
• parse_xml(content) - Parse XML strings with schema inference
• parse_xml_objects(content) - Parse XML strings to raw XML type

Key HTML Functions:

• read_html(pattern) - Read HTML files into tables
• read_html_blocks(pattern) - Read HTML files directly into flattened duck_block rows
• parse_html_blocks(content) - Parse HTML strings directly into flattened duck_block rows
• html_extract_tables(html) - Extract HTML tables as structured data
• html_extract_links(html) - Extract all links with metadata
• html_extract_images(html) - Extract images with attributes
• html_extract_text(html, xpath) - XPath-based HTML text extraction
• parse_html(content) - Parse HTML strings with schema inference
• parse_html_objects(content) - Parse HTML strings to raw HTML type
• html_to_duck_blocks(html) - Convert HTML to structured document blocks
• duck_blocks_to_html(blocks) - Convert document blocks back to HTML

See https://duckdb-webbed.readthedocs.io for comprehensive documentation and examples.

Built on libxml2 for robust, standards-compliant parsing with comprehensive error handling and memory-safe RAII implementation. 102 test suites with 3951 assertions, including DOM/SAX equivalence, adversarial/security, and multi-file parallelism testing. The extension supports mixed file systems, configurable schema inference, and efficient processing of large document collections.

Added Functions

function_name	function_type	description	comment	examples
duck_blocks_to_html	scalar	Convert a list of duck_block structures back into an HTML document string.	NULL	[duck_blocks_to_html(html_to_duck_blocks('<h1>Title</h1>'))]
html_escape	scalar	Escape special characters for HTML embedding.	NULL	[html_escape('<hello & world>')]
html_extract_images	scalar	Extract all image tags () from HTML as a list of structs.	NULL	[html_extract_images('')]
html_extract_links	scalar	Extract all hyperlink tags () from HTML as a list of structs.	NULL	[html_extract_links('DuckDB')]
html_extract_table_rows	scalar	Extract rows and cells from all tables in HTML as a list of structs.	NULL	[html_extract_table_rows('<table><tr><td>A</td></tr></table>')]
html_extract_tables	table	Extract all HTML tables from an HTML string as structured rows.	NULL	[SELECT * FROM html_extract_tables('<table><tr><td>A</td></tr></table>')]
html_extract_tables_json	scalar	Extract all tables from HTML as JSON structures.	NULL	[html_extract_tables_json('<table><tr><th>H</th></tr><tr><td>V</td></tr></table>')]
html_extract_text	scalar	NULL	NULL
html_to_duck_blocks	scalar	Parse an HTML document into a list of duck_block structures.	NULL	[html_to_duck_blocks('<h1>Title</h1><p>Text</p>'), html_to_duck_blocks('<table>…</table>', capture_attributes := true)]
html_unescape	scalar	Decode HTML entities in a string.	NULL	[html_unescape('<hello&world>')]
json_to_xml	scalar	Convert a JSON string to XML.	NULL	[json_to_xml('{"root": {"item": "Hello"}}')]
parse_html	scalar	Parse an HTML string into an HTML type.	NULL	[parse_html('<div>Hello</div>')]
parse_html	table	Parse an HTML string with automatic schema inference and return tabular data.	NULL	[SELECT * FROM parse_html('<div><p>Paragraph</p></div>')]
parse_html_blocks	table	Parse an HTML string and return each element as a duck_block row.	NULL	[SELECT * FROM parse_html_blocks('<h1>Hello</h1><p>World</p>')]
parse_html_objects	table	Parse an HTML string and return raw HTML object content.	NULL	[SELECT * FROM parse_html_objects('<div>A</div>')]
parse_xml	table	Parse an XML string with automatic schema inference and return tabular data.	NULL	[SELECT * FROM parse_xml('A')]
parse_xml_objects	table	Parse an XML string and return raw XML object content.	NULL	[SELECT * FROM parse_xml_objects('1')]
read_html	table	Read an HTML file with automatic schema inference and return tabular data.	NULL	[SELECT * FROM read_html('page.html')]
read_html_blocks	table	Read an HTML file and return each element as a duck_block row.	NULL	[SELECT * FROM read_html_blocks('page.html')]
read_html_objects	table	Read an HTML file and return raw HTML object content.	NULL	[SELECT * FROM read_html_objects('page.html')]
read_xml	table	Read an XML file with automatic schema inference and return tabular data.	NULL	[SELECT * FROM read_xml('data.xml'), SELECT * FROM read_xml('data.xml', record_element := '//item')]
read_xml_objects	table	Read an XML file and return raw XML object content.	NULL	[SELECT * FROM read_xml_objects('data.xml')]
to_xml	scalar	NULL	NULL
xml	scalar	Cast or convert a string to XML.	NULL	[xml('value')]
xml_add_namespace_declarations	scalar	Inject xmlns namespace declarations into an XML document root element.	NULL	[xml_add_namespace_declarations('', map(['ns'], ['http://example.com']))]
xml_common_namespaces	scalar	Return a map of common, well-known namespace prefixes and their URIs.	NULL	[xml_common_namespaces()]
xml_detect_prefixes	scalar	Detect namespace prefixes used within an XPath expression.	NULL	[xml_detect_prefixes('//ns:item/other:tag')]
xml_extract_all_text	scalar	Extract all concatenated text content from an XML document or fragment.	NULL	[xml_extract_all_text('')]
xml_extract_attributes	scalar	Extract attributes from elements matching an XPath expression as a list of structs.	NULL	[xml_extract_attributes('', '//item')]
xml_extract_cdata	scalar	Extract CDATA sections from an XML document as a list of structs with content and line numbers.	NULL	[xml_extract_cdata('some raw content')]
xml_extract_comments	scalar	Extract comments from an XML document as a list of structs with content and line numbers.	NULL	[xml_extract_comments('')]
xml_extract_elements	scalar	Extract XML fragments matching an XPath expression as a list of XML fragments.	NULL	[xml_extract_elements('AB', '//item')]
xml_extract_elements_string	scalar	Extract XML elements matching an XPath expression as a single concatenated string.	NULL	[xml_extract_elements_string('A', '//item')]
xml_extract_text	scalar	Extract text content matching an XPath expression from XML as a list of strings.	NULL	[xml_extract_text('Hello', '//item'), xml_extract_text('A', '//ns:item', map(['ns'], ['uri']))]
xml_find_undefined_prefixes	scalar	Find namespace prefixes used in an XPath expression that are not declared in the XML document.	NULL	[xml_find_undefined_prefixes('', '//ns:item')]
xml_libxml2_version	scalar	Return the linked libxml2 version.	NULL	[xml_libxml2_version('test')]
xml_lookup_namespace	scalar	Lookup URI for a well-known namespace prefix.	NULL	[xml_lookup_namespace('soap')]
xml_minify	scalar	Minify an XML string by removing unnecessary whitespace.	NULL	[xml_minify('\n value\n')]
xml_mock_namespaces	scalar	Generate mock namespace URI mappings for a list of prefixes.	NULL	[xml_mock_namespaces(['ns', 'other'])]
xml_namespaces	scalar	Extract all declared namespace prefixes and URIs from an XML document as a MAP.	NULL	[xml_namespaces('')]
xml_oom_selftest	scalar	Internal regression self-test for libxml2 memory handling.	NULL	[xml_oom_selftest()]
xml_pretty_print	scalar	Format and indent an XML string.	NULL	[xml_pretty_print('value')]
xml_stats	scalar	Compute statistics (element count, attribute count, max depth, size, namespace count) for an XML document.	NULL	[xml_stats('')]
xml_to_json	scalar	Convert an XML string to JSON.	NULL	[xml_to_json('Hello'), xml_to_json('Hello', attr_mode := 'prefixed')]
xml_valid	scalar	Check if an XML string or document is well-formed.	NULL	[xml_valid(''), xml_valid('')]
xml_validate_schema	scalar	Validate an XML string against an XSD schema string.	NULL	[xml_validate_schema(xml_doc, xsd_schema)]
xml_well_formed	scalar	Check if an XML string or document is well-formed.	NULL	[xml_well_formed('')]
xml_wrap_fragment	scalar	Wrap XML fragment content in an enclosing root tag.	NULL	[xml_wrap_fragment('12', 'root')]

Overloaded Functions

This extension does not add any function overloads.

Added Types

Added Settings

This extension does not add any settings.