Search Shortcut cmd + k | ctrl + k
observefs
Downloads 5.2kthis week
GitHub stars 18
Extension descriptor (YAML)

Provides IO observability to filesystem

Maintainer(s): dentiny, peterxcli

Installing and Loading 

INSTALL observefs FROM community;
LOAD observefs;

Example 

COPY (SELECT observefs_get_profile()) TO '/tmp/output.txt';

About observefs

This extension provides observability to duckdb filesystems. It supports a few key features:

  • 100% compatible with duckdb httpfs
  • Record latency for most of the IO operations (open, read, write, etc)
  • Provides both process-wise and bucket-wise latency stats (including histogram and quantile estimation) to all read operations
  • Provides cache access insight to duckdb external file cache
  • Allows registering ANY duckdb compatible filesystems (i.e., azure filesystem)

Added Functions

function_name function_type description comment examples
observefs_clear scalar Clears collected I/O observability metrics for all wrapped filesystems. NULL [observefs_clear()]
observefs_clear_external_file_cache_access_record scalar Clears the recorded external file cache hit and miss counts. NULL [observefs_clear_external_file_cache_access_record()]
observefs_external_file_cache_access_record table Returns recorded external file cache hit, miss, and partial-hit counts. NULL [observefs_external_file_cache_access_record()]
observefs_get_profile scalar Returns a human-readable profile of I/O metrics for wrapped filesystems. NULL [observefs_get_profile()]
observefs_list_registered_filesystems table Lists the filesystem implementations registered with DuckDB. NULL [observefs_list_registered_filesystems()]
observefs_wrap_filesystem scalar Wraps a registered filesystem with I/O observability instrumentation. NULL [observefs_wrap_filesystem('AzureBlobStorageFileSystem')]

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
observefs_enable_external_file_cache_stats Whether to enable stats record for external file cache. BOOLEAN GLOBAL []