With connectors available for databases and support for data sources, Prometheux can be used to seamlessly integrate and migrate data across various platforms. Moreover, it supports cloud and distributed file systems like S3 and HDFS, providing the flexibility needed for modern data lake and data migration scenarios.

​
@bind options

The bind command allows for the configuration of reading from and writing to database and datasources. The syntax is as follows:
Critical syntax requirements for @bind annotations:
  1. Options must be comma-separated: option1='value1', option2='value2'
  2. Values must be quoted: host='localhost' not host=localhost
  3. Use username= not user= for database authentication
  4. Must end with a dot: @bind(...).
Correct PostgreSQL example:
Common mistakes:
Supported datasource_type values: And the available configuration options are:
  • url: URL to use for the database connection (e.g. jdbc:postgresql://localhost:5432/prometheux)
  • protocol: Protocol to use for the database connection (e.g. jdbc, odbc, jdbc-odbc, bolt)
  • host: Database host (e.g. localhost)
  • port: Database port (e.g., 5432 for postgres, 7678 for neo4j)
  • database: Database name (e.g. prometheux)
  • username: Username to login with.
  • password: Password to login with.

​
Configuring credential access

Sensitive credentials such as database connection details (e.g., username, password) or AWS credentials (e.g., accessKey, secretKey) can be specified directly as options in the @bind annotations. Example:
This method allows for streamlined integration within the same code, ensuring that each datasource has its necessary credentials attached during the binding process. However, for better security and flexibility, these credentials can also be stored in external configurations:
  • Credentials can be stored the px.properties file to centralize sensitive information and allow reusability without hardcoding values within the @bind annotations.
  • REST APIs for dynamic configuration management, where you can set individual credentials or update multiple settings at once through API endpoints.

​
CSV Datasource

Prometheux supports CSV files as data source, both for reading and writing. The default CSV binding ("csv") is thus suitable for processing big CSV files. It does not make a guess about the input schema. Therefore, if no schema (@mapping) is provided, all fields are treated as strings. Values \N are treated as null values and interpreted as Labelled Nulls while reading the CSV file.

​
@bind options

The bind command allows for the configuration of reading and writing csv files. The syntax is as follows:
The options that are available are
  • useHeaders: Values can be true or false, depending on whether a header is available/output.
  • delimiter: Specifies the character that is used to separate single entries
  • recordSeparator: Specifies how the record are seperated
  • quoteMode: Defines quoting behavior. Possible values are:
    • all: Quotes all fields.
    • minimal: Quotes fields which contain special characters such as a the field delimiter, quote character or any of the characters in the line separator string.
    • non_numeric: Quotes all non-numeric fields.
    • none: Never quotes fields.
  • nullString: Value can be any string, which replaces null values in the csv file.
  • multiline: Handles multi-line fields. When multiline is set to true it handles fields that span multiple lines correctly
  • coalesce: Unifies the output in one partition. The output will be a single CSV file instead of partitioned CSV files. Supported only in standalone environments.
When specifying a configuaration, any subset of these options can be set at the same time. Each value has be surrounded by single quotation marks 'value'. An example for a csv bind command with configuration would be the following:

​
Examples

In the examples below we use a sample CSV file with the following content:
Simply reading a csv file into a relation:
showLineNumbers
We can also map the columns:
showLineNumbers
Selecting columns and filtering with SQL To select specific columns or filter rows, bind the full CSV with useHeaders=true and use a rule with a SQL body over the bound predicate. The SQL query can reference column names directly from the CSV header:
showLineNumbers
To store results into another CSV file, bind the output predicate:

​
Parquet Datasource

Parquet is a columnar storage format optimized for both storage efficiency and query performance, making it well-suited for large-scale data lake scenarios. In this section, we’ll explore how Parquet files can be integrated into Vadalog workflows, using the provided example:

​
Iceberg Datasource

Apache Iceberg is an open table format for huge analytic datasets. It is not a database and not a file format: it is a metadata layer that sits on top of plain Parquet/ORC/Avro files on object storage (S3, ADLS, GCS, HDFS, or a local filesystem) and adds catalog-managed schemas, snapshots, hidden partitioning, branches, tags, and ACID commits on top of them. In Prometheux you point an @bind at an Iceberg catalog + table identifier, not at a file path. The Prometheux runtime then takes care of resolving the identifier through the catalog, pruning files by metadata, and committing writes atomically — all of Iceberg’s planning-time benefits are inherited automatically:
  • Partition pruning — including hidden-partition transforms (bucket(N, col), truncate(N, col), days/hours/months/years(ts)). Predicates from Vadalog rules are pushed down to Iceberg’s manifest scan, so non-matching partitions are dropped before any file is opened.
  • File skipping via per-file min/max statistics — Iceberg stores lower_bound/upper_bound/null counts per column per data file in its manifest files, so files whose ranges are disjoint from the predicate are skipped at planning time.
  • Bloom filters — Parquet column bloom filters and Iceberg puffin/index bloom filters are read by the vectorized reader for equality / IN predicates on high-cardinality columns.
  • Cost-based optimization — Iceberg exposes record counts, null counts and NDV sketches as table statistics, which the Prometheux planner uses for join ordering and broadcast/shuffle decisions on every recursive reasoning round.
  • Time travel — read any historical snapshot by snapshot-id or as-of-timestamp.
  • Branches and tags — Iceberg lets you pin reads/writes to a named branch (e.g. dev, experiment_42) or tag, ideal for what-if reasoning pipelines that materialize results onto a non-main branch before promotion.

​
Catalog configuration

Before authoring an Iceberg @bind, the target catalog must be registered in the Prometheux engine configuration. All standard Iceberg catalog types are supported:
  • Hadoop (local / single-node, no metastore required) — point a warehouse path at a directory or object-storage prefix.
  • Hive (Hive Metastore), Glue (AWS Glue Data Catalog), REST (Iceberg REST catalog), Nessie (Project Nessie) — point a uri at the corresponding service.
The catalog name you register (e.g. prod_catalog) becomes the prefix of the table identifier used in @bind. Ask your administrator to register the catalog once at the engine level; see the Installation guide for the exact configuration properties.

​
@bind options for Iceberg

In addition to the shared options (saveMode, compression, selectedColumns, query, repartition, s3aAccessKey, s3aSecretKey, …) the Iceberg connector supports the following Iceberg-specific options:

​
Example 1: basic read

The schema field of the @bind is the catalog + namespace (e.g. prod_catalog.sales); the table field is the Iceberg table name:

​
Example 2: basic write

Prometheux commits Iceberg writes atomically through Iceberg’s snapshot protocol. On the first write the target table is created automatically (Hadoop catalogs create the namespace directory on demand; for Hive/Glue/REST catalogs the namespace must already exist):
The saveMode option maps to Iceberg commit semantics:

​
Example 3: time travel by snapshot id

Every Iceberg commit produces a new snapshot. The full history is queryable from the <table>.snapshots system table; pinning a read to a previous snapshot is a single bind option:

​
Example 4: time travel by timestamp

Same idea, but pinned to wall-clock time. Iceberg resolves the latest snapshot ≤ the given epoch-millisecond timestamp:

​
Example 5: reading from a branch (what-if reasoning)

Iceberg branches let you fork the table, run experimental reasoning on the fork, and discard or promote the result without touching main:

​
Example 6: column projection pushdown

The shared selectedColumns option works exactly as it does for Parquet and CSV — by index ranges, by name, or any mix. The Iceberg reader receives the projection and only opens the columns you ask for:

​
Example 7: SQL body over an Iceberg-bound predicate

Just like with CSV/Parquet, you can attach a SQL query to an Iceberg bind and the resulting filters / aggregations are pushed into the Iceberg scan whenever possible:

​
Cypher over Iceberg

Iceberg-bound predicates accept inline Cypher queries in rule bodies the same way they accept SQL bodies. You write a Cypher pattern; the engine produces an execution plan that pushes filters, projections, partition pruning and file skipping into the Iceberg scan, then runs the join / aggregation / graph algorithm over the pruned data. There is no Neo4j server in the loop — the engine evaluates the Cypher locally against the Iceberg tables, and snapshots, branches, schema evolution and statistics work exactly as they do for plain Iceberg reads. The full Cypher language surface — patterns, WHERE operators, RETURN expressions, WITH pipelines, aggregations, GDS algorithms — is documented once for every source in Cypher Integration. This section focuses on the Iceberg-specific aspects: how a Cypher pattern lines up with an Iceberg table, what pushdown looks like, and how time travel / branches / mixed-source joins compose with Cypher bodies.

​
Property-graph ↔ Iceberg layout

For a Cypher pattern to line up with an Iceberg table, follow the property-graph ↔ relational convention used by every non-Neo4j source (see Cypher Integration › How Cypher patterns map to your data for the full reference): The Iceberg-flavoured examples in this section use the canonical bindings below — assume they are present at the top of every snippet:

​
Predicate and projection pushdown

A single Cypher pattern with a WHERE is the most common case. The endpoint and property predicates and the RETURN projection are pushed into the Iceberg scan, so partitions, files and pages whose statistics are disjoint from the predicate are dropped at planning time — exactly as if you had written a SQL WHERE:
A pattern that reads node properties across a relationship keeps Iceberg’s pushdown on each side independently (column pruning + partition pruning + min/max file skipping), and the engine joins the projected results:

​
Graph algorithms over an Iceberg-backed graph

Graph Data Science calls work directly over an Iceberg edge table. When the projected node label is itself backed by an @bind’d Iceberg table, the engine restricts the graph to edges whose endpoints exist in that table — a node-induced edge filter applied before the algorithm runs:
See Cypher Integration › Graph algorithms for the full algorithm catalog and parameter options.

​
Cypher pinned to a historical snapshot

Time travel is a bind-level option on the Iceberg source, so the Cypher body stays unchanged — only the @bind changes. Pin a Cypher query to a snapshot id or an as-of timestamp:

​
Cypher on a what-if branch

The same trick with branch='…' lets you run Cypher (including GDS) on a non-main branch without touching production data:

​
Mixed-source joins with Iceberg

A single Cypher body can reference predicates bound to different data layers. The Iceberg side keeps its planning-time pushdown; the other side keeps its own connector-native pushdown (JDBC SQL, Cypher to Neo4j, Qdrant filter); the engine joins the projected results:
For the rest of the Cypher surface — list / map / temporal operations, WITH pipelines, the full WHERE and RETURN expression library, and recipes for composing Vadalog rules on top of a Cypher result — see Cypher Integration.

​
Vadalog, SQL, and Cypher together over Iceberg

The same Iceberg dataset can be queried with plain Vadalog rules, inline SQL bodies, or inline Cypher bodies. Pick the lens that best fits the query you’re writing — they all share the same Iceberg planning-time pushdown (partition pruning, file skipping, column pruning) and they all compose with the rest of your program. Suppose your catalog contains an Iceberg person table (id, name, age, city) and an Iceberg knows table (src, dst, since). Three identical questions, expressed three ways:

​
Lens 1 — Vadalog rules

Best when the logic involves derivations, recursion, or rule-based reasoning the other lenses cannot express:

​
Lens 2 — Inline SQL body

Best for analytical queries — joins, aggregations, window functions, CTEs:

​
Lens 3 — Inline Cypher body

Best when the question is naturally a graph pattern — multi-hop, variable-length, shortest path, GDS:
The three produce the same result, take the same Iceberg planning-time benefits, and can be freely mixed in the same program: one rule can derive its inputs with Vadalog, another can pre-aggregate with SQL, and a third can pattern-match with Cypher — all consuming the same Iceberg tables and producing predicates the others can consume.
Pick the right lens for the question
  • Vadalog — recursion, derivations, the chase, AI / vector functions, hashing, anything the other lenses cannot express. See Thinking in Vadalog.
  • SQL — joins, aggregations, window functions, CTEs, anything that is naturally relational. See SQL Integration.
  • Cypher — pattern matching, multi-hop traversals, shortestPath, GDS algorithms. See Cypher Integration.

​
Notes

  • All planning-time optimizations (partition pruning, min/max file skipping, bloom filters, CBO via table statistics) are owned by the Iceberg layer; Prometheux preserves the pushdown opportunity by keeping Vadalog-derived filters and projections adjacent to the Iceberg scan in the execution plan.
  • Recursive Vadalog rules over Iceberg-bound predicates benefit from the same planning-time pruning on every round of semi-naive evaluation, not just the first read.
  • Mixed-source joins (Iceberg ⨝ PostgreSQL, Iceberg ⨝ Neo4j, Iceberg ⨝ Qdrant, …) work transparently: each side keeps its own connector-native pushdown (JDBC SQL, Cypher, Qdrant filter), and the join itself is executed by the Prometheux engine.

​
Excel Datasource

Excel files are widely used for tabular data storage and exchange in business and analytics workflows. Prometheux can easily read from and write to Excel files, integrating them seamlessly into its data processing workflows. In this example, we will read data from a CSV file and populate an Excel file with the extracted data.
In this example, we will read data from the previously populated Excel file.
In this example, we will read data from a specific sheet of an Excel file.

​
JSON Datasource

JSON (JavaScript Object Notation) is a lightweight data interchange format that is widely used for APIs, configuration files, and structured data storage. Prometheux supports reading JSON files and querying their nested structures using two powerful approaches: SQL queries in rule bodies and the struct:get function for accessing nested fields. When Prometheux reads JSON files, nested objects are automatically inferred as struct types, allowing you to access nested fields using dot notation in SQL queries.

​
Example JSON Structure

Throughout this section, we’ll use examples based on an e-commerce orders JSON file with the following structure:
Understanding JSON Array Handling — Root-level arrays vs Nested arrays:
  • Root-level JSON array (like the example above): Each array element becomes a separate row automatically. No collections:explode needed.
  • Nested array field (e.g., {"data": [item1, item2, ...]}): The array becomes a single column value. Use collections:explode to convert array elements into multiple rows.
Example of nested array requiring explode:
This would give you 1 row with items as an array column. To get multiple rows (one per product), use collections:explode.

​
Simple Example: Reading JSON Files

This example demonstrates reading a JSON file without any query:

​
Accessing Nested Fields with SQL in Rule Bodies

The recommended approach for querying JSON data is to use SQL directly in rule bodies. This allows you to process nested structures using dot notation:

​
SQL Queries with Filtering and Aggregation

JSON datasources support full SQL capabilities including WHERE clauses, aggregations, and grouping:

​
SQL Queries with Ordering and Distinct

​
Using Parameters in SQL Queries

You can use @param to parameterize your JSON queries:

​
Alternative: Using struct:get Function

For accessing individual nested fields in Vadalog rules (without SQL), you can use the struct:get function. Note that the variable must be on the left side of the assignment:

​
Example: Filtering with struct:get

​
Example: Accessing Multiple Nested Structs

​
Using the query Option in @bind

Alternatively, you can specify SQL queries directly in the @bind annotation using the query option:

​
Real-World Example: E-Commerce Analytics

This example demonstrates a complete analytics workflow for processing order data from JSON files:

​
COBOL Datasource

Prometheux can read legacy COBOL / EBCDIC data files alongside the other file‑based datasources, so mainframe extracts (VSAM dumps, flat record files, variable‑length RDW/BDW streams) can be joined and transformed with the same Vadalog rules you already use for CSV, Parquet or JSON data. The connector is powered by the AbsaOSS Cobrix library, which the Prometheux engine integrates natively for distributed COBOL parsing. A COBOL binding always needs two inputs:
  1. a data file (binary, addressed by <filepath> + <filename>, exactly as for CSV or Parquet),
  2. a copybook describing the record layout — passed via the copybook option as a path to a .cpy file on the same file system (local, HDFS, or S3 via s3a://).
Why the copybook is always a pathThe engine‑side @bind tokenizer strips \n and \t characters from the option block and does not support escaped single quotes, which makes multi‑line copybook bodies and VALUE 'A' clauses structurally impossible to embed inline. Keep the copybook as a file next to (or accessible from) the data file and reference it by path.

​
@bind options

Supported options:
  • copybook: required, path to the COBOL copybook file describing the record layout (e.g. /data/layouts/customer_master.cpy).
  • cobolPreset: shortcut for common mainframe framings. One of:
    • flat-fixed-ebcdic (default) — fixed‑length records, EBCDIC encoding (IBM037 / IBM1140). Matches most VSAM ESDS and flat dump exports.
    • flat-fixed-ascii — fixed‑length records, ASCII encoding. Used when a vendor has already transcoded the extract.
    • mainframe-v — IBM variable‑length records with a 4‑byte big‑endian RDW header.
    • mainframe-vb — IBM variable‑blocked: BDW‑prefixed blocks of RDW‑prefixed records.
    • custom — no preset; every Cobrix flag is supplied via cobrix.* options.
  • encoding: override the encoding inferred by the preset. Typical values are ebcdic (with ebcdic_code_page='common', 'cp037', 'cp1140', …) or ascii.
  • record_format: Cobrix record format. F for fixed, V for variable, VB for variable‑blocked.
  • cobolFlattenPolicy: how nested COBOL groups (e.g. CUST-ADDRESS in the example below) are exposed in the Vadalog schema.
    • dotted (default) — nested groups are flattened and leaf columns are prefixed with their parent group name (CUST_ADDRESS_CUST_CITY).
    • keepNested — nested structs are preserved and exposed as map values in the Vadalog predicate, consumed via struct:get.
  • cobolFileExtensions: optional comma‑separated extension filter when the filepath points to a directory (e.g. .dat,.bin).
  • cobrix.<flag>: passthrough escape hatch — any option starting with cobrix. is forwarded to the underlying Cobrix reader verbatim (e.g. cobrix.is_rdw_big_endian='true'). Useful when a preset does not cover a particular vendor quirk.

​
Example: reading customer_master.dat

The copybook describes a 119‑byte fixed‑length record with a nested address group, COMP‑3 packed decimals, a COMP binary counter, and zoned‑decimal identifiers — a realistic shape for a customer master extract:
Given the copybook customer_master.cpy and the binary extract customer_master.dat sitting in /data/cobol/, the binding below reads the records, flattens the CUST-ADDRESS group into top‑level columns, and projects the customers registered in Italy:

​
Example: variable‑length mainframe file

For extracts shipped with IBM’s variable‑length framing (RDW‑prefixed records), switch the preset:

​
Example: advanced Cobrix overrides

When the file uses a non‑standard framing (e.g. RDW whose length field is not part of the record length, or a big‑endian BDW with non‑zero adjustment), drop down to the custom preset and forward the Cobrix flags unchanged via the cobrix.* passthrough:

​
Keeping nested groups as structs

If you prefer to query nested groups with struct:get (as you already do for JSON), flip the flatten policy:

​
Querying COBOL data with SQL

Like any file-based datasource, COBOL binds support SQL queries via the <- operator. The COBOL connector reads the binary file into an in-memory dataset and runs the SQL over it, which is useful for previewing, paginating, counting, or filtering mainframe data without writing COBOL-specific logic:
End-to-end exampleFor a complete example showing COBOL data preview, cross-source JOINs with PostgreSQL, and compliance rules, see From Mainframe to Modern: Migrating Card Clearing.

​
Tips

  • PIC S9(n)V99 COMP-3 (packed decimal) is decoded as a decimal at the connector level and exposed as double in the Vadalog model — use double in @model/@mapping if you declare the schema explicitly.
  • PIC 9(n) COMP binary counters are exposed as integers (int/long depending on width).
  • OCCURS n TIMES groups become Vadalog list columns and can be expanded with collections:explode, just like JSON arrays.
  • Point filepath to a directory (not a single file) to read every candidate extract in one go; combine with cobolFileExtensions='.dat,.bin' to narrow the selection.

​
PostgreSQL Database

PostgreSQL is a robust open-source relational database that supports a wide range of data types and advanced querying capabilities. In this section, we will explore how to integrate PostgreSQL with Vadalog by first populating a customer table from a CSV file and then reading data from it using two approaches: full table read and a custom query. In this example, we read data from a CSV file and populate the customer table in a PostgreSQL database.
This example demonstrates reading the full customer table from PostgreSQL.
In this example, we read specific columns and filter data using a SQL query.

​
PostgreSQL with Supabase

Supabase is an open-source Firebase alternative that provides a hosted PostgreSQL database. To connect Prometheux to your Supabase database, you can use the Transaction Pooler connection method with a JDBC URL.

​
How to Retrieve Your Supabase Connection String

  1. Log in to your Supabase Dashboard.
  2. Select your project.
  3. Navigate to Project Settings → Database.
  4. Under Connection string, select the JDBC tab.
  5. Choose Transaction Pooler as the connection mode (recommended for serverless and short-lived connections).
  6. Copy the JDBC connection string.
The connection string will be in the following format:

​
Example: Connecting to Supabase with JDBC URL

This example demonstrates how to read data from a Supabase PostgreSQL table using the Transaction Pooler and JDBC connection.
Alternatively, instead of using the url parameter, you can specify the connection details individually:
Connection ModesSupabase offers different connection modes:
  • Transaction Pooler (port 6543): Best for serverless functions and short-lived connections. Uses PgBouncer in transaction mode.
  • Session Pooler (port 5432): For long-lived connections that need session-level features.
  • Direct Connection (port 5432): Direct connection to the database without pooling.
For most Prometheux use cases, the Transaction Pooler is recommended as it efficiently manages connection pooling.
For enhanced security, instead of using your main database password, you can create a dedicated read-only user with access limited to specific tables. This follows the principle of least privilege and minimizes security risks.
Steps to Create a Read-Only User in Supabase
  1. Log in to your Supabase Dashboard.
  2. Select your project.
  3. Navigate to SQL Editor and execute the following SQL commands:
Setup TimeAfter creating the user and granting permissions, it may take a few minutes for the changes to propagate and become active. If you encounter connection issues immediately after setup, wait 5 minutes and try again.
Once the read-only user is created, use it in your connection string:

​
Alternative: Connecting via Supabase REST API

RecommendationThe JDBC connection method (shown above) is heavily recommended for production use. It provides full PostgreSQL capabilities, better performance, and more reliable connections. The REST API method below is primarily suitable for one-time access to simple tables or quick prototyping scenarios.
Supabase also exposes a REST API (powered by PostgREST) that allows you to access your database tables directly via HTTP. This approach uses your Supabase API keys for authentication.
How to Retrieve Your Supabase API Credentials
  1. Log in to your Supabase Dashboard.
  2. Select your project.
  3. Navigate to Project Settings → API.
  4. Copy your Project URL (e.g., https://yourprojectid.supabase.co).
  5. Copy your service_role key (secret) or anon key depending on your security requirements.
API keys: the publishable key / anon (legacy) respects Row Level Security (RLS) policies if enabled for your tables, while the secret key bypasses RLS policies.
Example: Connecting to Supabase via REST API
This example demonstrates how to read data from a simple Supabase table using the REST API with bearer token authentication. This method is best suited for quick, one-time data access or prototyping scenarios.

​
MariaDB Database

MariaDB is a popular open-source relational database, highly compatible with MySQL. It supports various SQL features and is commonly used in web applications and data platforms. In this example, we will explore how to interact with a MariaDB database in Prometheux, focusing on reading data from the order_customer table to test if the data has been populated correctly.

​
Neo4j Database

Neo4j is a graph database designed for efficiently storing and querying highly connected data. In this example, we’ll explore how to populate Neo4j with nodes representing Person and Order entities, as well as relationships between them. We’ll also cover how to query this data using a Cypher query. This example shows how to read data from a CSV file, populate Neo4j with Person and Order nodes, and create a relationship between them.
In this example, we query Neo4j to retrieve the relationship between Person and Order nodes.

​
Querying Neo4j with Cypher in rule bodies

Instead of placing the Cypher query inside a @qbind annotation, you can write it directly in a rule body. When the body starts with a Cypher keyword (MATCH, OPTIONAL, UNWIND, CALL, CREATE, MERGE), it is interpreted as a Cypher query and pushed down to Neo4j automatically. You still need a @bind annotation to provide the Neo4j connection details — Prometheux uses it to know where to execute the query.
Not just Neo4jThe same inline Cypher also runs over non-Neo4j sources — CSV/Parquet/Iceberg files, relational databases, vector stores, and derived predicates. See Cypher Integration for the full language surface, or Cypher over non-Neo4j sources below for the layout convention.
This is equivalent to writing a separate @qbind, but keeps the query inline with the rule — useful when you want the Cypher logic visible alongside the rest of the program.

​
Filtering and aggregating with Cypher

Any valid Cypher query can be used in the body:
When to use Cypher vs SQL
  • Cypher in rule body — when you need Neo4j-specific features like graph traversals, shortestPath, variable-length patterns, or APOC procedures.
  • SQL in rule body — when you want familiar SQL syntax for previewing, counting, filtering, or paginating Neo4j data. Prometheux translates it to Cypher for you.
  • Both approaches push the query down to the Neo4j server — no data is loaded into memory unnecessarily.

​
Cypher over non-Neo4j sources (CSV, files, databases)

A Cypher body is not limited to Neo4j. The same inline Cypher also runs when the predicates it references are bound to CSV / Parquet / Iceberg files, a relational database (PostgreSQL, MariaDB, Snowflake, …), in-memory facts, or even a derived predicate produced by another rule. The full Cypher language surface — patterns, WHERE operators, RETURN expressions, WITH pipelines, aggregations, GDS algorithms — is documented in Cypher Integration. This section focuses on what is specific to non-Neo4j sources: how a Cypher pattern lines up with relational tables and files.

​
How to lay out your data

For a Cypher pattern to line up with relational tables / files, name things as follows: The field names used in Cypher must match a CSV header or a table column — if there is no corresponding name, the query errors with column not found. Use @mapping to alias source columns into this layout when the underlying schema does not match.

​
A representative example

A pattern that reads node properties across a relationship works over files and databases alike. The endpoints of the relationship reference the node id, so the node table is joined in to resolve its properties:
The exact same query runs unchanged against a database — just swap the binding:
Endpoints reference the node idFor a join to line up, the relationship’s src/dst must hold the values of the node id column (here, knows.src/knows.dst are person.id values). A label used on both endpoints (such as Person above) is joined once per endpoint automatically.
Where the join executesWhen every predicate the pattern references is bound to the same database (PostgreSQL, Snowflake, …), the whole join is pushed down as a single statement to that database. When the sources are heterogeneous — files (CSV / Parquet / Iceberg), or a mix of connectors (Iceberg ⨝ PostgreSQL ⨝ Neo4j ⨝ Qdrant) — each source keeps its own connector-native pushdown (predicate / projection / partition pruning, JDBC SQL, Cypher to Neo4j, Qdrant filter) and the engine performs the join over the pruned results.

​
Reading from an intermediate result (positional columns)

A node or relationship can resolve to a derived predicate (the head of another rule), to facts, or to a headerless CSV. These have no header names — only positional columns named <predicate>_<index> (_0, _1, …). Reference them positionally:
Header names vs positional
  • Use property names (RETURN n.region) when the source has headers — a CSV bound with useHeaders=true, or a database table.
  • Use positional columns (RETURN airport_concept_1) when the source is derived, headerless, or in-memory facts.
For the full Cypher surface — WHERE operators, projection / value transformation, aggregation, WITH pipelines, graph algorithms, and worked examples over each source kind — see Cypher Integration.

​
Querying Neo4j with SQL

You can also use SQL queries directly against Neo4j-bound predicates. Prometheux automatically translates the SQL into optimized Cypher and pushes it down to the Neo4j server — no data is loaded into memory unnecessarily. This is useful when you want to preview, paginate, count, or filter Neo4j data using familiar SQL syntax.

​
Reading all nodes

Bind a Neo4j node label and read all its data:

​
Preview with LIMIT and OFFSET

Paginate through Neo4j data using SQL LIMIT and OFFSET. These are translated to Cypher SKIP/LIMIT and pushed down to the Neo4j server:

​
Counting records

Use SQL COUNT(*) to count nodes or relationships. The count is computed entirely by the Neo4j server:

​
Querying relationships with SQL

SQL works on relationship patterns as well:

​
Combining preview and count

A common pattern is to run both a paginated preview and a count in the same program:
How SQL-to-Neo4j translation worksWhen you write SQL against a Neo4j @bind predicate, Prometheux transparently converts the query:
  • SELECT * with LIMIT/OFFSET → The @bind pattern is used to generate optimized Cypher with WITH ... SKIP n LIMIT n RETURN ..., fully pushed down to Neo4j.
  • SELECT COUNT(*) and other aggregations → Converted to a Cypher @qbind query (e.g., MATCH (n:Person) RETURN count(n) AS total) and executed server-side.
  • No data is loaded into memory for filtering or pagination — everything is handled by the Neo4j server.

​
Amazon DynamoDB Database

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. Prometheux supports both reading from and writing to DynamoDB tables, with automatic table creation capabilities and support for PartiQL queries.

​
@bind options for DynamoDB

The DynamoDB connector supports the following configuration options:
  • region: AWS region for the DynamoDB instance (e.g., us-east-1, eu-west-1)
  • username: AWS Access Key ID for authentication
  • password: AWS Secret Access Key for authentication
  • sessionToken: AWS Session Token for temporary credentials (optional)
  • endpointOverride: Custom endpoint URL (useful for DynamoDB Local testing)
  • partitionKey: The partition key attribute name for table creation
  • sortKey: The sort key attribute name for table creation (optional)
  • billingMode: Either PAY_PER_REQUEST (default) or PROVISIONED
  • readCapacity: Read capacity units for provisioned billing mode (default: 5)
  • writeCapacity: Write capacity units for provisioned billing mode (default: 5)
  • writeBatchSize: Number of items to write per batch (1-25, default: 25)
  • readPageSize: Page size for read operations (default: 100)
  • totalSegments: Number of segments for parallel scanning (default: 8)
  • inferSampleLimit: Number of items to sample for schema inference (default: 64)
  • secondaryIndexName: Name of Global Secondary Index to create (optional)
  • secondaryIndexPartitionKey: Partition key for the GSI (optional)
  • secondaryIndexSortKey: Sort key for the GSI (optional)

​
Example 1: Writing Data to DynamoDB

This example shows how to read data from a CSV file and write it to a DynamoDB table with automatic table creation.

​
Example 2: Reading Data from DynamoDB

This example demonstrates reading data from an existing DynamoDB table.

​
Example 3: Using PartiQL Queries

This example shows how to use PartiQL (SQL-compatible query language) to query DynamoDB data.

​
Example 4: Advanced Configuration with Sort Key and GSI

This example demonstrates creating a table with both partition and sort keys, plus a Global Secondary Index.

​
Example 5: Using Session Credentials

This example shows how to use temporary AWS credentials with session tokens.

​
Example 6: Using DynamoDB Local for Development

This example shows how to connect to DynamoDB Local for development and testing.

​
Advanced PartiQL Features

DynamoDB supports powerful PartiQL functions that can be used in queries:

​
Configuration Best Practices

  1. Credentials Management: Store sensitive credentials in pmtx.properties or environment variables rather than hardcoding them in bind annotations.
  2. Billing Mode: Use PAY_PER_REQUEST for variable workloads and PROVISIONED for predictable traffic patterns.
  3. Batch Sizes: Adjust writeBatchSize based on item size - use smaller batches for larger items.
  4. Parallel Scanning: Increase totalSegments for faster reads on large tables, but be mindful of consumed read capacity.

​
Qdrant Vector Database

Qdrant is a high-performance vector similarity search engine designed for storing, indexing, and querying dense vector embeddings. Prometheux integrates with Qdrant via gRPC, supporting concept-aware embedding generation, raw vector search, payload filtering, and structured result decomposition — all from within Vadalog rules. Qdrant collections store points, each consisting of:
  • A vector (dense float array) used for similarity search
  • A payload (key-value metadata) attached to the point
  • A point ID (integer or UUID)
Prometheux supports three read modes and a concept-aware write mode, making it suitable for both text-based semantic search and domain-specific embeddings (e.g., molecular fingerprints, image features).

​
Vadalog + Qdrant = Ontology-grounded RAG

Most “RAG over a vector database” pipelines work in one direction only: you embed your text, you embed your query, you hope cosine similarity returns something useful, and an LLM stitches an answer together. There is no ground truth about what each row means, no way to express domain rules, and no audit trail. Vadalog + Qdrant changes that. The @model annotation lets you attach a small ontology to every Qdrant-backed predicate: a description template that says, in your domain’s language, what a row represents. The same ontology is then used everywhere: The result is ontology-grounded RAG with reasoning — a pipeline where vector search and rule-based inference share the same domain model. Concept-aware retrieval gives you semantic recall; rules give you precision, explainability, and the ability to combine retrieved knowledge with everything else in your program. If you want to see this end-to-end, jump to:
  • Example 11 — the minimal write → semantic search → reason loop in ~20 lines.
  • Example 12 — a richer case study unifying PDF + TXT + CSV into a pharma knowledge base, comparing concept=true vs concept=false, and applying reasoning rules to the retrieved drugs.

​
Minimal Configuration

All connection and embedding options have sensible defaults provided by the Prometheux platform configuration. This means the simplest possible @bind for a Qdrant write with concept-aware embeddings is:
And for a semantic search read:
You only need to specify options explicitly when you want to override the defaults.

​
@bind options for Qdrant

The Qdrant connector supports the following configuration options: Connection:
  • host: Qdrant server hostname (default: platform-configured)
  • port: gRPC port (default: 6334)
  • useTls: Enable TLS for the connection (default: false)
  • password: API key for Qdrant authentication (also available as apiKey)
Read options:
  • query: Natural language search query — the text is embedded via Azure OpenAI and used for KNN search
  • queryVector: Pre-computed vector for direct similarity search, bypassing text embedding (e.g., queryVector=[0.1 0.2 0.3])
  • limit: Maximum number of results to return (default: 10)
  • scoreThreshold: Minimum similarity score for results (optional)
  • includeVector: Whether to include the vector in the result (default: false)
  • filter: Payload filter expression (see Payload Filtering)
  • scrollPageSize: Page size for bulk scroll reads (default: 100)
  • concept: Enable concept-aware query enrichment (default: true; set to false to disable)
Write options:
  • vectorDimension: Dimension of the vectors in the collection. Automatically derived from the embedding model when not specified (e.g., 3072 for text-embedding-3-large, 1536 for text-embedding-3-small). Only required when using pre-computed embeddings with a custom dimension.
  • distance: Distance metric — Cosine, Euclid, Dot, or Manhattan (default: Cosine)
  • toEmbedField: Name of the column whose raw text value should be embedded. Used when concept=false and you want to embed a single column rather than generating a concept description from all fields (e.g., toEmbedField='description'). The platform embeds this column’s value via Azure OpenAI.
  • toEmbedFieldPos: 0-based positional index of the column to embed, as an alternative to toEmbedField when no @model annotation is present (e.g., toEmbedFieldPos=2 means the 3rd column is the text to embed)
  • embeddingField: Name of the DataFrame column containing a pre-computed vector (a list of floats). Use this when the embeddings are already generated externally — the platform skips embedding generation and writes the vector directly (e.g., embeddingField='my_vector').
  • embeddingFieldPos: 0-based positional index of the pre-computed vector column, as an alternative to embeddingField when no @model annotation is present (e.g., embeddingFieldPos=3 means the 4th column is the vector)
  • batchSize: Number of points per write batch (default: 64)
  • shardNumber: Number of shards for the collection (default: 1)
  • replicationFactor: Replication factor (default: 1)
  • embeddingModel: Azure OpenAI embedding model name (default: text-embedding-3-large)
Controlling what gets embeddedThere are three ways to control what gets embedded when writing to Qdrant:When none of these is specified and concept=true (default), the system auto-generates a concept description from all column names and values.Tip: @model with a description template already covers the toEmbedField use case and more. For example, @model("pred", "['name:string','desc:string']", "[desc]") embeds only the desc field — equivalent to toEmbedField='desc' — but with the added benefit that you can later enrich it (e.g., "[name] is described as [desc]") without changing the @bind.

​
Example 1: Writing Data with Concept-Aware Embeddings

This example writes city data to Qdrant. The @model annotation provides a description template that is resolved per row and used to generate embeddings. Each row produces a natural language sentence that is then embedded.
After execution, the cities collection contains 4 points, each with:
  • A 3072-dimensional embedding generated from the resolved description (e.g., “Paris is a city in France, located in Europe, with a population of 2161000”)
  • Payload fields: name, country, continent, population, plus a __qdrant_text__ field containing the description text

​
Example 2: Semantic Search with Natural Language Query

Read from Qdrant by providing a natural language query. The query text is embedded and used for KNN similarity search.
Project only the columns you needYou can also write the body as a SQL projection and skip the full positional binding:
The reader pushes the projection (and any LIMIT/WHERE/OFFSET) down to Qdrant, so only the requested payload fields are deserialized server-side.
System columnsQdrant results always include the following system columns, in a fixed canonical position:
  • __id__ (string) — the point ID (integer or UUID, returned as string)
  • __qdrant_text__ (string) — the text used for embedding (only present if the data was written with concept-aware embedding)
  • __vector__ (array of floats) — the raw vector. Returned only when includeVector=true is set on @bind, e.g. "qdrant includeVector=true, query='…'".
  • __score__ (double) — the similarity score (only present in search mode, not in scroll mode)

​
Example 3: Semantic Search in Rule Bodies (SELECT "..." / ASK)

The SELECT "..." FROM <pred> syntax (and its equivalent ASK "..." FROM <pred>) provides a concise way to write Qdrant semantic-search queries directly in rule bodies, without manually configuring @bind options. The rule is automatically rewritten into a synthetic @bind with the appropriate query parameters. The Qdrant reader always returns columns in a fixed canonical order:
__id__, payload fields (alphabetical), __qdrant_text__ if present, __vector__ if includeVector=true, __score__ (search mode only).
The head atom binds positionally to this layout — the i-th head variable receives whatever Qdrant put at column position i. The variable names are not significant. If the head has fewer variables than Qdrant returns, only the first N columns are projected.
ASK is accepted as a fully equivalent alias — both syntaxes go through the same rewriter, so pick whichever reads better in context:
If the head has fewer variables, you get the leading columns. For example, to capture just __id__ and __score__ of the top match, take the first column plus skip ahead positionally — typically you would still take the full canonical projection and ignore what you don’t need in the head atom of the consuming rule.
@model on a Qdrant read predicate@model does not declare the schema of the read result and does not reorder columns. On a Qdrant read predicate, @model is used only to enrich the embedding (concept description and context). The column layout returned by the reader is always the canonical Qdrant ordering.If you don’t need concept enrichment, you can omit @model entirely on a Qdrant read.

​
Returning the embedding vector

To fetch the raw vector along with the search result, set includeVector=true on the @bind. The vector is inserted in the canonical layout right before __score__ (for searches) or as the trailing column (for scroll reads):
Vector column type for writesWhen writing pre-computed embeddings, declare the vector column in @model as __vector__:list (an array of floats) — the writer uses @model to name the payload fields on the write side. See Example 10 and the embeddingField option.

​
Optional WHERE / LIMIT / SCORE_THRESHOLD clauses

Both SELECT "..." and ASK accept optional payload filters and limits, which are pushed down to Qdrant alongside the search:

​
Example 4: Concept-Aware Query Enrichment

When concept=true (the default), the query text is automatically enriched with semantic context derived from the @model description (or, in its absence, from the column names declared by @model). This biases the embedding toward the predicate’s structure, improving search relevance.
To disable concept enrichment:
When your data uses pre-computed embeddings (e.g., molecular fingerprints, image features, or custom domain embeddings), you can bypass the text-to-embedding step and search directly with a raw vector.
Vector value separatorInside queryVector=[...], use spaces to separate values, not commas. Commas are reserved as the @bind option delimiter.Correct: queryVector=[0.1 0.2 0.3 0.4] Incorrect: queryVector=[0.1,0.2,0.3,0.4] — commas will break the option parsing

​
Example 6: Bulk Scroll Read

When no query or queryVector is provided, Qdrant reads all points from the collection using the scroll API. This is useful for bulk data extraction.

​
Example 7: Payload Filtering

Payload filters can be applied to both search and scroll operations. Filters restrict results based on payload field values without affecting the vector similarity scoring. Equality and inequality:
Multiple conditions with AND:
IN operator (match any value from a list): When using IN with multiple values, wrap the filter in single quotes to protect the commas from the @bind option parser:
NOT IN operator:
Range filters (numeric comparisons):
Filter with scroll (no query):
Supported filter syntax

​
Example 8: Using the ask() Function

The ask() function is a Prometheux SQL function that performs Qdrant vector search per-row, returning results as a JSON array. This is useful for enriching existing data with similarity search results, or for using runtime vectors from other data sources (e.g., a parquet file). Syntax:
The second argument is a single options string containing comma-separated key-value pairs. Available options:
Connection defaultsThe ask() function resolves connection properties using a three-level fallback: inline option → pmtx.properties → hardcoded default. If your platform is already configured with Qdrant settings in pmtx.properties, the only option you typically need is collection:
To target a different Qdrant instance, override the connection inline:
Text mode — search by natural language query:
The Answer variable contains a JSON array:
Using ${Variable} interpolation in the prompt: When the prompt contains ${Variable} placeholders, the system automatically binds them to the corresponding Vadalog variables at runtime:
For each row, ${Description} is replaced with the actual value of the Description variable before embedding the prompt. Vector mode — search by pre-computed vector:

​
Example 9: Decomposing ask() Results with as_list, as_struct, struct:get

​
JSON structure returned by ask()

The ask() function returns a JSON array string where each element is a JSON object representing a Qdrant scored point. The objects always contain:
  • __id__ (string) — the point ID
  • All payload fields from the collection (e.g., name, formula, activity, __qdrant_text__ if concept-aware embeddings were used)
  • __score__ (double) — the similarity score
Example output:

​
as_list — parse a JSON array into a typed array of structs

Syntax: as_list(json_string, "field1:type1, field2:type2, ...") Parses a JSON array string into a typed array of structs. The schema is a comma-separated list of field_name:type pairs. Supported types: string, double, integer, long, boolean. Key rules:
  • Fields are matched by name, not by position. The order of fields in the schema does not need to match the order in the JSON; the parser locates each field by its key.
  • You only need to include the fields you want to extract. Omitted fields are simply ignored — you don’t need to list every payload field.
  • Include __score__:double if you need the similarity score.
  • Include __id__:string if you need the point ID.
  • __qdrant_text__ is present only when the collection was written with concept-aware or text embeddings. Include it in the schema only if you need it.
In this example, the schema "compound_id:string, name:string, __score__:double" extracts only 3 fields from the JSON — even if each object contains many more payload fields. The rest are discarded.

​
collections:get — access elements by index

Syntax: collections:get(array, index) Returns the element at the given position. Indexing is 1-based (1 = first element, 2 = second, etc.).

​
struct:get — extract a field from a struct

Syntax: struct:get("field_name", struct) Extracts a single named field from a struct. The field name must match one of the fields declared in the as_list or as_struct schema. Extracting multiple results:
Here the schema only declares name and __score__ — the country, __id__, and any other payload fields are ignored because they aren’t needed.

​
as_struct — parse a single JSON object

Syntax: as_struct(json_string, "field1:type1, field2:type2, ...") Same as as_list but for a single JSON object (not an array). Returns a struct directly, without wrapping it in an array.
Schema cheat sheet for Qdrant ask() resultsWhen writing the as_list schema for ask() results, you only need to declare the fields you actually use. Here are the available fields:Example — extracting only name and score from a molecules collection:
Example — extracting everything including the embedded text:
The field order in the schema does not matter — fields are matched by name.

​
Example 10: Writing Pre-Computed Embeddings

When your data already contains embedding vectors (e.g., from a custom model or external pipeline), you can write them directly to Qdrant without using the Azure OpenAI embedding service.

​
Example 11: End-to-End Pipeline — Write, Query, and Reason

This example demonstrates a complete workflow: writing data to Qdrant, querying it with semantic search, and applying Vadalog reasoning over the results.

​
The Role of @model on Reads

On a Qdrant read predicate, @model does not declare the schema and does not reorder columns. The read result is always shaped by the Qdrant reader itself, in the canonical column ordering documented in Deterministic Column Ordering:
__id__, payload fields (alphabetical), __qdrant_text__ if present, __vector__ if includeVector=true, __score__ (search mode only).
What @model does on a read predicate is purely embedding enrichment: it provides the concept description (and the field list used to auto-generate one) that the @bind query is enriched with when concept=true. See Example 4: Concept-Aware Query Enrichment.
If you only need a few fields, prefer a SQL projection in the rule body — the projection (and any LIMIT/OFFSET/WHERE) is pushed down to Qdrant so only the requested payload fields are deserialised:
On a Qdrant read, @model is entirely optional — you only need it if you want concept enrichment (and the description it contributes). Without it, the canonical column ordering is still produced and rule bodies can still bind positionally.@model is required on the write side: there it names the payload fields written to Qdrant and provides the description template used for concept-aware embedding.

​
Example 12: Ontology-grounded RAG with reasoning — a pharma knowledge base

This is the full ontology-grounded RAG pipeline: build a Qdrant knowledge base from three heterogeneous sources (a PDF, a text file, a CSV), retrieve over it with concept-aware semantic search, and then reason on the retrieved scored points with Vadalog rules. Three layers of the pipeline: The scenario: A pharmaceutical company maintains drug information across different formats. The pipeline below builds a unified knowledge base from those sources, retrieves drugs relevant to a cardiology query, and then applies clinical reasoning rules (e.g. “suggest cardiac drugs with high semantic relevance that are not contraindicated for the patient”).

​
Step 1: Extract data from a PDF (drug package inserts)

​
Step 2: Extract data from a text file (clinical notes)

​
Step 3: Read structured data from CSV (drug registry)

​
Step 4: Unify and write to Qdrant WITH concept enrichment

All three sources are unified into a single Qdrant collection. The @model description template tells Qdrant what each row means — this context is embedded alongside the data.
When this runs, a row like ("Aspirin", "Analgesic", "pain relief and anti-inflammatory", "drug_registry") is embedded as:
“Aspirin is a pharmaceutical drug in category Analgesic indicated for pain relief and anti-inflammatory”
This sentence is what gets converted into a 3072-dimensional vector — it carries the full semantic meaning.

​
Step 5a: Query WITH concept enrichment (default: concept=true)

Result with concept=true: All results are relevant cardiac drugs because the concept-enriched query embedding was grounded in the pharmaceutical domain.

​
Step 5b: Query WITHOUT concept enrichment

Result with concept=false: Without concept enrichment, the query “cardiac treatment options” is embedded as a generic phrase. It matches documents that happen to contain the words “cardiac” or “treatment” anywhere, rather than finding drugs specifically indicated for cardiac conditions. The top results are irrelevant PDF fragments and clinical notes that mention the words but are not actual cardiac drugs.
Why concept enrichment worksThe difference is in what gets embedded:
  • Without concept (concept=false): The query "cardiac treatment options" is embedded as-is — a generic 5-word phrase. The embedding model has no idea this is about pharmaceutical drugs.
  • With concept (concept=true): The query becomes "cardiac treatment options. Context: [name] is a pharmaceutical drug in category [category] indicated for [indication]". The embedding model now understands this query is looking for drugs, in categories, with specific indications. The resulting vector is pulled toward the same semantic region as the stored drug descriptions.
This is why concept enrichment is enabled by default — it bridges the gap between a user’s natural language question and the structured domain knowledge stored in Qdrant.

​
Step 6: Reason on top of the retrieved drugs

Up to this point we have done a high-quality semantic retrieval — but that is still only retrieval. The third layer of the pipeline is reasoning: the retrieved drugs are now a regular Vadalog predicate, so we can join them with structured clinical facts and derive new ones with logical rules. We add two small structured sources of clinical context — a patient allergy list and a recall list — and then declaratively express the policy “for this patient, suggest cardiac drugs that are highly relevant and safe to prescribe”:
For patient p001, who is allergic to ACE Inhibitors and Anticoagulants, and given the recall on Warfarin, only the beta-blocker survives all three filters — semantic relevance, allergy safety, recall safety: Lisinopril (ACE Inhibitor) is filtered out by the allergy rule; Warfarin (Anticoagulant and on the recall list) is filtered out by both rules.
What just happenedThis is what ontology-grounded RAG with reasoning looks like in practice:
  1. The @model description acted as a lightweight ontology, grounding every embedded row in domain semantics.
  2. concept=true reused that same ontology on the query side, so the semantic search returned drugs that are genuinely about cardiac treatment — not just documents containing the words cardiac or treatment.
  3. The retrieved drugs entered the reasoning engine as the predicate cardiac_drugs, where they were joined with structured allergy and recall facts and filtered by a declarative safety policy.
Each layer is auditable: you can inspect the description that was embedded, the enriched query that was sent to the embedding model, the scored points returned by Qdrant, and the chain of rules that produced the final recommendation. There is no opaque LLM in the loop deciding what is “safe” — the decision is a derivation.

​
Walkthrough: Molecular Database Demo

This walkthrough demonstrates a complete Qdrant workflow — writing molecules, querying them with semantic search, applying filters, and decomposing results — using a minimal self-contained example.
How embeddings work behind the scenesWhen your dataset does not contain a pre-computed embedding vector (i.e., no embeddingField or embeddingFieldPos is specified), Prometheux automatically generates embeddings using its default embedding system (Azure OpenAI text-embedding-3-large).On write, the system constructs a natural language description for each row and embeds it into a 3072-dimensional vector:
  • If @model is provided with a description template (e.g., "[name] is a drug indicated for [indication]"), the template is resolved per row and embedded.
  • If @model is provided without a description template (or no annotation at all), the system auto-generates a description from the column names (e.g., "mol_out: name is Aspirin, formula is C9H8O4, weight is 180.16, activity is anti-inflammatory").
  • The generated text is stored in the __qdrant_text__ payload field alongside the vector.
On read, the same embedding system is used to convert your search query into a vector:
  • A query='pain relief medication' is embedded via the same model, producing a 3072-dimensional vector that is compared against the stored vectors using cosine similarity.
  • When concept=true (the default), the query is enriched with the concept context provided by @model (description template or auto-generated from field names) before embedding, biasing the vector toward the same semantic space as the stored data. This is the only role @model plays on a Qdrant read predicate — it does not shape or reorder the read result.
This means you can go from raw tabular data to semantic similarity search without writing any embedding code — the platform handles the entire vector lifecycle transparently.

​
Step 1: Write Molecules with Concept-Aware Embeddings

Write three molecules to Qdrant. @model names the payload fields and declares their types. Since no description template is provided, the system auto-generates concept text from the column names for embedding.
The minimal @bind("mol_out", "qdrant", "", "molecules") uses all platform defaults: host, port, distance (Cosine), and vectorDimension (3072 for the default text-embedding-3-large model).

​
Step 2: Semantic Search with Natural Language

Search the collection with a natural language query. On read, the column layout is fixed by the canonical Qdrant ordering — @model is optional and only contributes a description used to enrich the query embedding.
Expected result: The anti-inflammatory drugs rank highest for “pain relief medication”, while Caffeine (a stimulant) scores lowest.

​
Step 3: Range Filter on Molecular Weight

Filter molecules by numeric payload fields using range operators. This uses bulk scroll (no query=) with a range filter.
Expected result: Ibuprofen (206.28) is excluded because it falls outside the 180–200 range.

​
Step 4: Combined Semantic Search with Equality Filter

Combine a semantic search with a payload filter to search for “pain relief” only among anti-inflammatory drugs.
Expected result: Caffeine is excluded by the equality filter despite being in the collection.

​
Step 5: Semantic Search in the Rule Body (SELECT "..." / ASK)

The SELECT "..." FROM <pred> syntax (and its ASK "..." alias) performs semantic search directly in the rule body. The head atom binds positionally to the canonical Qdrant column ordering — the first N columns are projected (where N is the head’s arity).
To also fetch the raw embedding, set includeVector=true on the @bind. The vector slides into the canonical layout just before __score__:
Expected result: Caffeine ranks first because “energy boosting substance” is semantically closest to a stimulant.

​
Step 6: ask() UDF with Variable Substitution

The ask() function searches Qdrant per-row using the ${Variable} pattern (same as llm:generate). It returns a JSON array of results that can be decomposed with as_list, collections:get, and struct:get.
For each molecule, ask() searches Qdrant with “find similar to [name]” and returns JSON results. The decomposition pipeline extracts the second match (collections:get(L, 2) — 1-based indexing) to find each molecule’s nearest neighbor. Expected result: The anti-inflammatory drugs (Aspirin, Ibuprofen) are each other’s nearest neighbors with higher similarity, while Caffeine’s nearest match has a lower score — confirming the semantic clustering works as expected.

​
Deterministic Column Ordering

Qdrant stores payload fields internally using a protobuf map, which does not guarantee a consistent iteration order across reads. To ensure predictable schemas, Prometheux enforces a fixed column layout on every Qdrant read: This ordering is deterministic and consistent across runs, regardless of how Qdrant’s protobuf layer iterates the payload map internally. The layout is fixed by the connector — @mapping / @model annotations on a Qdrant read predicate do not reorder, rename, or project this layout. Rule bodies bind to the result by position.
When the Prometheux data-manager generates Vadalog annotations for a Qdrant collection, the field list in the @model follows this same ordering convention. This means the schema you see in the data-manager UI matches exactly what the reader produces at runtime.

​
SQL in Rule Bodies

You can write SQL queries directly in rule bodies over Qdrant-bound predicates, just like you would with JDBC or Neo4j data sources. The system transparently converts the SQL into the appropriate Qdrant operations.

​
SELECT * with LIMIT and OFFSET

Read a page of results from a Qdrant collection:
The LIMIT and OFFSET clauses are extracted at optimization time and pushed to the Qdrant reader, which scrolls the collection and applies them efficiently. The result is shaped by the canonical Qdrant ordering — no @model needed.

​
Column projection with SELECT [columns]

You can project specific payload fields by listing them in the SELECT clause:
When the SQL body lists explicit columns (instead of SELECT *), the reader pushes a payload allowlist down to Qdrant: only the requested payload fields are fetched and deserialised. The LIMIT/OFFSET/WHERE clauses are also pushed down, exactly like with SELECT *. This works for any single-table SELECT over a Qdrant predicate (no JOINs, no aggregates) — column-projected and SELECT * bodies share the same fast path.

​
SELECT * with WHERE

Filter payload fields using SQL WHERE syntax:
The WHERE clause is converted into a Qdrant payload filter and applied server-side before results are returned.

​
Aggregations: COUNT, SUM, AVG, etc.

Aggregate queries are executed by Prometheux over the Qdrant collection:
Since Qdrant does not have a native SQL engine, aggregate queries (COUNT, SUM, AVG, DISTINCT, …) are handled by first loading the collection data into the Prometheux engine and then executing the SQL query in-memory.

​
Combining Both in a Single Program

You can mix SELECT * and aggregate queries over the same Qdrant predicate:
How SQL-in-body works for QdrantThe optimization pipeline handles Qdrant SQL rules in two ways:
  • Single-table SELECT (with or without explicit columns) with optional LIMIT/OFFSET/WHERE: The clauses are extracted and stored as parameters on the @bind annotation. The Qdrant reader applies LIMIT and OFFSET at scroll time, and WHERE is converted into a Qdrant payload filter applied server-side. If the SELECT lists explicit columns, the reader also pushes a payload allowlist down to Qdrant, so only the requested fields are fetched.
  • Aggregates / JOINs (COUNT, SUM, AVG, DISTINCT, multi-table joins): The Qdrant collection is first loaded into the Prometheux engine via the regular @bind mechanism, then the SQL query is executed in-memory. All standard SQL aggregation features are available.

​
Configuration Best Practices

  1. Use Platform Defaults: Connection settings (host, port, API key, Azure OpenAI credentials) are managed by the Prometheux platform configuration. You only need to specify options in @bind when overriding the defaults.
  2. Vector Dimensions: When using concept-aware or text embedding, vectorDimension is automatically derived from the embedding model (e.g., 3072 for text-embedding-3-large). You only need to set it explicitly when using pre-computed embeddings with a custom dimension.
  3. Concept Enrichment: Leave concept=true (the default) when querying collections written with @model descriptions. This ensures the query embedding is biased toward the same semantic space as the stored embeddings.
  4. Raw Vectors vs. Text Queries: Use queryVector= when working with domain-specific embeddings that were not generated from text (e.g., molecular fingerprints, audio features). Use query= for natural language semantic search.
  5. Filter Performance: Payload filters are applied server-side by Qdrant before results are returned. Use filters to reduce result sets rather than filtering in Vadalog rules for better performance.
  6. Pre-Computed Embeddings: Use embeddingField (column name) when you have an @model that names the vector column, or embeddingFieldPos (0-based positional index) when no schema annotation is present.

​
RDF Output (triple stores)

Prometheux can serialize a reasoning result to RDF files on disk or object storage, ready for bulk import into any RDF triple store. The connector is powered by Apache Jena, so IRI and literal escaping follow the RDF 1.1 grammar rather than a hand-rolled encoder. The rdf connector is write-only (an export target): you bind it on an @output concept exactly like the Parquet writer, using the same <schema>/<table> path convention. By default the output is a directory of RDF files plus a _SUCCESS marker — one file per Spark partition — so large results are written in parallel across the cluster. Set singleFile=true to instead emit one correctly named file (<table>.ttl / .rdf / .jsonld / .nt / .nq) that is trivial to hand off. The produced files are standard, W3C-conformant RDF (there is no Prometheux-specific framing), so any RDF platform can ingest them directly with its native bulk loader — see Loading into a triple store. The same row-to-triple mapping can also target a relational or warehouse connector (PostgreSQL, Databricks, Snowflake, …) instead of files, by adding outputFormat=rdf to that connector’s @bind — see Writing RDF triples to a database.

​
Serialization formats

Choose the serialization with the format option:
For bulk loading into a triple store, prefer the line-oriented N-Triples (or N-Quads) format: it is written fully in parallel and is the fastest to ingest. The document formats (turtle, rdfxml, jsonld) are more human/tooling-friendly; because each partition materializes an in-memory graph, increase the number of partitions for very large results.
In the default distributed mode each part file is a complete, standalone document over its slice of the data, so import the files individually (or point the loader at the directory). The line formats (ntriples/nquads) can also be safely concatenated into one file, but turtle, rdfxml and jsonld part files cannot be blindly cat-ed together (concatenating two <rdf:RDF> roots or two JSON documents is invalid). When you want a single valid document, use singleFile=true.

​
Row-to-RDF modes

How each row becomes triples is driven by the @model annotation and the mode option:
  • entity (default) — each row is one subject. The subject IRI is minted from the primary-key column; the writer emits an rdf:type triple to the concept’s class plus one triple per non-null column (<base><predicate>), typed with the appropriate xsd: datatype. Declared superclasses become rdfs:subClassOf and the @model description becomes an rdfs:comment.
  • triple — each row is one relationship triple <from> <predicate> <to>. The subject and object classes and the source/target id columns come from a triple @model (see below). This mode is selected automatically when the @model declares a triple; set mode='triple' explicitly only to override.
  • column-driven (generic) triple — each row is a full triple whose subject, predicate and object (and their types) are read from named columns, ideal for a generic (subject, subjectType, predicate, object, objectType) table where the predicate and types vary per row. Activated by setting predicateColumn (see Example 4).

​
@bind options for RDF

In addition to the shared writer options (saveMode, compression, repartition), the RDF connector supports:
  • format: ntriples (default), nquads, turtle, rdfxml, or jsonld (aliases accepted, see table above).
  • singleFile: when true, write one correctly named file <schema>/<table>.<ext> (extension derived from format: .nt / .nq / .ttl / .rdf / .jsonld) instead of a directory of Spark part files. For the document formats this rebuilds one valid document (instance triples + the @model ontology header) so the artifact is a single, self-contained .ttl / .rdf / .jsonld. This pulls the serialized document to the driver, so it is intended for moderate exports meant to be shared as one file; leave it false (default) for fully distributed output of very large results.
  • baseIri: the base IRI used to build subject, class and property IRIs. A trailing / or # is added if missing (default: http://prometheux.co.uk/).
  • mode: entity (default) or triple. Auto-derived from the @model when omitted.
  • subjectColumn: the column whose value identifies the subject entity in entity mode. Falls back to the @model primary key, then the first column.
  • graphIri: the named-graph IRI added as the fourth term of every data line in nquads (default: the default graph).
  • emitOntology: whether to also emit schema-level (TBox) triples derived from the @model — rdfs:Class/rdfs:subClassOf/rdfs:comment for entities, or rdf:Property with rdfs:domain/rdfs:range for triples (default: true).
Options for the column-driven (generic) triple mode (each accepts a column name or a 0-based index):
  • predicateColumn: the column holding the predicate of each triple. Setting this activates the mode.
  • objectColumn: the column holding the object of each triple.
  • subjectTypeColumn: the column holding the subject’s type/class. Used to mint the subject IRI (<base><type>/<id>) and, when emitOntology is on, an rdf:type triple. Optional.
  • objectTypeColumn: the column holding the object’s type. When the value is a datatype (string, int, long, double, float, decimal, boolean, date, dateTime; xsd:-prefixed forms accepted) the object becomes a typed literal; otherwise it is treated as a class and the object becomes an entity IRI (plus an rdf:type triple). Optional; a blank value defaults to a string literal.
  • literalTypes: an optional comma-separated list of extra type names to treat as (string) literals, for domain-specific datatype spellings.
In this mode the subject column reuses the shared subjectColumn option, and emitOntology controls whether the inline rdf:type triples are emitted. Each option can also be set in pmtx.properties with the rdf. prefix (e.g. rdf.baseIri=...).

​
Example 1: entity mode → N-Triples

Read people from a CSV and export them as N-Triples, one subject per person:
Produces (one file per partition, plus a class header):

​
Example 2: triple mode (relationships)

A triple @model has the shape (fromClass)predicate(toClass), and marks the source/target id columns with (sID)/(tID). Both endpoint classes must themselves be modeled. The rdf writer then auto-selects triple mode:
Produces one relationship triple per row (plus the rdf:Property / rdfs:domain / rdfs:range header):

​
Example 3: Turtle, RDF/XML, JSON-LD and N-Quads

The document formats (turtle, rdfxml, jsonld) produce tooling-friendly documents (one per partition); N-Quads places every data triple in a named graph via graphIri:
To share one clean file instead of a directory of part files, add singleFile=true. This is the usual choice for RDF/XML and JSON-LD, which are meant to be handed over as a single self-contained document:

​
Example 4: generic triple table (column-driven)

If your data is already a generic triple table — one relation whose columns are (subject, subjectType, predicate, object, objectType) and where the predicate and types vary from row to row — you don’t need a triple @model per predicate. Point the writer at the columns instead, and it emits one triple per row. Take this simple table:
The writer decides per row how to render the object from object_type: a class (Person, City) becomes an entity IRI (with an rdf:type triple); a datatype (string, int, …) becomes a typed literal:
The rdf:type lines can repeat (once per row that mentions an entity), because the writer stays fully parallel and never shuffles to deduplicate. RDF is a set of triples, so the triple store collapses the duplicates on load. Set emitOntology='false' to skip the rdf:type lines entirely.

​
From a flat table to a richer graph

New to RDF? The key idea is that everything is a triple subject → predicate → object, and an entity gets “properties” simply by being the subject of more triples. There are no separate files or records per entity — the triple store connects everything by matching IRIs. Here is how to grow a plain edge list into a graph with entity and relationship properties, using a tiny “who knows whom” example. Step 1 — a flat edge list. You start with just the relationships: This already produces a valid graph — two Person entities linked by knows:
Step 2 — add properties to the entities. To describe alice herself, add more rows with alice as the subject and a datatype object_type (so the object becomes a literal value, not a link):
Notice alice is the object of a knows triple and the subject of her own name/age triples — that is what ties the graph together. Step 3 — add properties to a relationship. A single triple can’t carry extra fields (it’s only subject–predicate–object). So when the link itself needs properties — say you want to record since when alice knows bob — you promote the relationship to its own entity (a “linking node”). Instead of one knows edge, emit a Friendship node that points at both people and carries the extra data:
All three steps use the same generic table and the same @bind — you only add rows. The rule of thumb:
  • Object is another thing → give the row an entity object_type (a class like Person, City, Batch) → you get a link.
  • Object is a value → give the row a datatype object_type (string, int, date, …) → you get a literal property.
  • The relationship needs its own fields → make it a subject (a linking node like Friendship) and attach from, to, and its properties as ordinary rows.

​
Writing RDF triples to a database

The RDF row-to-triple mapping is not tied to files. Adding outputFormat=rdf to any relational or warehouse @bind — PostgreSQL, MariaDB, SQL Server, SQLite, Teradata, ClickHouse, Databricks, Snowflake, MongoDB, and even the file connectors Parquet / CSV — reuses the exact same mapping (entity, triple and column-driven modes, plus the @model TBox), but instead of serializing files it persists a plain triple table (subject, predicate, object) in the target. This is the right choice when a downstream system consumes triples through SQL rather than by importing RDF files: you get a normalized subject/predicate/object table that any query engine or BI tool can read directly.
  • The target receives three string columns subject, predicate, object (a fourth graph column is added when format=nquads).
  • Every RDF mapping option (format, baseIri, mode, subjectColumn, predicateColumn, objectColumn, subjectTypeColumn, objectTypeColumn, graphIri, emitOntology, literalTypes) behaves exactly as for the file writer. These options are consumed by the RDF mapper and are never forwarded to the database driver.
  • Every other option on the bind (connection host/port, credentials, saveMode, compression, repartition, …) is handled by the target connector as usual.
  • Dedicated graph stores — Neo4j and native RDF triple stores — are intentionally out of scope for outputFormat=rdf. Use their native connectors, or the file rdf writer plus a bulk load (see below).
Example — write the generic triple table from Example 4 into a PostgreSQL table instead of files. Only the connector and the outputFormat=rdf flag change; the RDF options are identical:
The resulting facts_triples table holds one row per triple (IRIs are stored as their full string, literals as their bare lexical value): Because it is an ordinary table, you can read it back with the plain connector and continue reasoning over the triples:
Literals are persisted as their bare lexical value (e.g. Alice, 30) so the table stays easy to query with SQL. The datatype / language tag carried in a file serialization is dropped in this relational view — it is a “triples as a table” projection, not a lossless RDF store.

​
Loading into a triple store

The output contains standard, W3C-conformant RDF files, so any triple store can ingest them with its native bulk-load facility (e.g. Fuseki tdb2.tdbloader, GraphDB import, the rdf4j console, or a cloud bulk loader) or over the SPARQL 1.1 Graph Store HTTP Protocol. Point the loader at the whole directory, or upload files individually; in the default distributed mode each part file is a complete, standalone document, and with singleFile=true you get a single ready-to-load file. Set the request Content-Type to match the format you wrote: For example, uploading a single Turtle file into a named graph over the Graph Store Protocol:
Or, in the default distributed mode, upload one N-Triples part file (Spark names distributed part files part-*.txt; the content is valid RDF regardless of the .txt suffix — set the Content-Type explicitly, or use singleFile=true for a correctly named file):
By default the connector also emits lightweight ontology (TBox) triples derived from your @model (class/subclass/comment, or property/domain/range). Set emitOntology='false' if you want to load only the instance data (ABox) and manage the schema separately in your triple store.

​
S3 Storage

Amazon S3 (Simple Storage Service) is a widely used cloud storage service that allows for scalable object storage. In Vadalog, you can both read from and write to S3 buckets by treating the S3 storage as a file system. This example demonstrates how to interact with S3, specifically by writing a CSV file to S3 and then reading from it. In this example, we first read data from a CSV file stored locally and then write it to an S3 bucket.
In this example, we demonstrate how to read data from a CSV file stored in an S3 bucket.

​
Consuming Data via API

Vadalog supports consuming data from REST APIs with flexible authentication and SQL query capabilities. The API reader supports JSON (default), CSV, and XML response formats, with full SQL integration for querying nested data structures. Popular Use Cases:
  • Cryptocurrency Analytics: Bitcoin, Ethereum, real-time price monitoring
  • Sports Analytics: Football leagues, team statistics, match predictions
  • Geospatial Data: Country demographics, weather patterns, mapping
  • Monitoring Systems: Prometheus metrics, Kubernetes clusters, Alertmanager
  • Data Integration: Combine multiple APIs for comprehensive analytics

​
Quick Start Examples

Get Bitcoin Price (works immediately, no auth):
Get Football Team Data:
Query Prometheus Metrics:

​
API Configuration Options

The API datasource supports the following configuration options:
  • responseFormat: Response format - json (default), csv, or xml. If not specified, JSON is assumed.
  • authType: Authentication method - basic, bearer, or apikey
  • username: Username for basic authentication
  • password: Password for basic authentication
  • token: Token for bearer authentication
  • apikey: API key for API key authentication
  • headers: Custom headers in format header1:value1,header2:value2
  • delimiter: CSV delimiter character (default: ,)
Free Public APIsMany popular APIs don’t require authentication and can be used immediately:
  • CoinGecko - Cryptocurrency prices and market data (rate limits apply on free tier)
  • TheSportsDB - Football/sports statistics and results
  • REST Countries - Country demographics and geospatial data
  • JSONPlaceholder - Test data for prototyping
Perfect for getting started with Vadalog API integration!
API Rate LimitsFree-tier APIs often have rate limits (e.g., CoinGecko: ~10-30 requests/minute). For production use:
  • Space out requests or use caching
  • Consider upgrading to paid API tiers for higher limits
  • Handle HTTP 429 (Rate Limit Exceeded) errors gracefully

​
Example 1: Simple API Call (JSON - Default Format)

Since JSON is the default response format, you don’t need to specify responseFormat=json:

​
Example 2: Querying Nested JSON with SQL

The recommended approach for querying API data is to use SQL directly in rule bodies:
Accessing Nested Structures in API ResponsesThe following Prometheus examples demonstrate how to work with nested JSON structures using Vadalog collection functions:
  • struct:get("field", struct): Access a field within a struct
  • collections:explode(array): Convert an array into multiple rows (one row per element)
  • collections:get(array, index): Access a specific element in an array (0-based index)
  • SIZE(array): Get the number of elements in an array
These functions allow you to navigate complex API responses without SQL array indexing.

​
Example 3: Prometheus Metrics API

Prometheus is a popular monitoring system that exposes metrics via REST API. Here’s how to query Prometheus instant queries and extract metric values:

​
Example 4: Prometheus Targets Endpoint

Query Prometheus targets to get detailed information about scrape targets:

​
Example 5: Prometheus Goroutines Metric

Query and extract goroutine counts from Prometheus:

​
Example 6: Prometheus Alert Rules

Query Prometheus alert rule definitions (works even when no alerts are firing):

​
Example 7: Prometheus - List All Metric Names

Query Prometheus to get all available metric names and filter them:
You can also count the total number of metrics:

​
Example 8: Prometheus - Query Build Information

Query Prometheus build information to get version details:

​
Example 9: Prometheus - Range Query

Query metric values over a time range:
Using Parameters for Dynamic Time Ranges:
Dynamic TimestampsFor current data, you can use environment variables or update the parameter values to recent epoch timestamps. Prometheus typically retains metrics for 15 days by default, so use timestamps within your retention period.
Working Example Without Authentication: If your Prometheus instance doesn’t require authentication, you can omit the auth parameters:

​
Example 10: Prometheus Alertmanager API

Alertmanager is OptionalAlertmanager is a separate service from Prometheus that handles alert routing and notification. It typically runs on port 9093. If you only have Prometheus running (port 9090), you won’t have this endpoint available. This example is for advanced setups with Alertmanager deployed.
Query active alerts from Prometheus Alertmanager using the v2 API:
Working Example Without Authentication:

​
Example 11: Kubernetes API

Query Kubernetes resources via the API server:
Working with Nested ArraysWhen dealing with deeply nested arrays (like status.containerStatuses[0]), consider using the struct:get function in Vadalog rules instead of direct array indexing in SQL. This provides more reliable access to complex nested structures.

​
Example 12: CSV Format API (Weather Data)

For APIs that return CSV format, specify responseFormat=csv:

​
Example 13: Joining Multiple API Sources

You can join data from multiple API endpoints using SQL in rule bodies:

​
Example 14: Common Table Expressions (CTEs) with API Data

Use CTEs for complex multi-step transformations:

​
Example 15: Using Parameters in API Queries

Combine @param with API queries for dynamic filtering:

​
Example 16: Bitcoin & Cryptocurrency Price Analytics

Track and analyze cryptocurrency prices in real-time using the CoinGecko API (no authentication required): Basic Bitcoin Data:
Bitcoin Price with Nested Field Access:
Real-Time Bitcoin Price Monitoring with Multiple Currencies:
Advanced: Bitcoin Price Alerts with Market Signals:
Compare Multiple Cryptocurrencies:

​
Example 17: Football (Soccer) Analytics

Analyze football match data, team statistics, and player performance using TheSportsDB API (no authentication required):
TheSportsDB Response FormatTheSportsDB wraps response data in arrays (e.g., table, teams, results). Use SIZE() to count items or access top-level response structure.
Premier League Standings - Count Teams:
Search for Specific Team:
Get Recent Matches:
Get All Available Leagues:
Get All Countries with Sports:

​
Example 18: Working with Arrays Using Vadalog Collection Functions

Instead of using SQL queries, you can process arrays directly in Vadalog using collection built-in functions. This is particularly useful when you want to work with arrays in a more functional style.
When Do You Need collections:explode?API responses have two common structures:
  1. Root-level array - Returns [{item1}, {item2}, ...]
    • Each element becomes a separate row automatically
    • No explode needed
  2. Nested array field - Returns {"leagues": [{item1}, {item2}, ...]}
    • You get 1 row with the entire array as a single column
    • Use collections:explode to create multiple rows
Most public APIs (TheSportsDB, CoinGecko, etc.) return objects with nested array fields, so you’ll typically need collections:explode to process individual array elements.
Using collections:explode to Convert Arrays to Rows:
Using collections:transform with Lambda Expressions: Transform array elements using lambda expressions before exploding:
Cryptocurrency List with Collection Functions:
Collection Functions vs SQLWhen to use Collection Functions:
  • You prefer functional programming style
  • Working with simple array transformations
  • Need to combine struct:get with array processing
  • Want to leverage lambda expressions for transformations
When to use SQL:
  • Complex filtering and aggregations (WHERE, GROUP BY, HAVING)
  • Joining multiple data sources
  • Using SQL-specific functions (COUNT, SUM, AVG)
  • Need DISTINCT, ORDER BY, or LIMIT clauses
Both approaches are valid - choose based on your use case and coding style preference!

​
Example 19: Cross-Domain Analytics - Combining Multiple APIs

Combine data from multiple API sources for comprehensive analysis: Bitcoin Price + Multiple Cryptocurrencies:
Football Leagues + Teams Analysis:

​
Example 20: REST Countries API - Public Data

Query public APIs without authentication:

​
API Authentication Methods Summary

​
Best Practices

  1. Default Format: JSON is the default responseFormat, so you can omit it for JSON APIs
  2. Use Environment Variables: Store sensitive tokens in environment variables: token=${API_TOKEN}
  3. SQL in Rule Bodies: Preferred approach for querying nested API data
  4. Filter Early: Apply WHERE clauses to reduce data at the source
  5. Handle Nested Data: Use dot notation (market_data.current_price.usd) for nested JSON fields
  6. Array Access: Use SIZE() function for reliable array length checks
  7. Aggregations: Use GROUP BY for summarizing API data
  8. JOINs: When joining multiple API sources, use renamed column format (predicate_name_i)
  9. Real-Time Data: Combine CTEs with CASE statements for market signals and alerts
  10. Rate Limits: Be mindful of API rate limits; use aggregations to reduce query frequency
Financial & Crypto Analytics:
  • Track Bitcoin prices in real-time (tested with CoinGecko API)
  • Monitor price changes across multiple currencies (USD, EUR)
  • Set up price alerts using CASE statements and CTEs
  • Compare multiple cryptocurrencies by symbol
  • Access nested market data (prices, volumes)
Sports Analytics:
  • Count teams in league standings (tested with TheSportsDB)
  • Search for specific teams and get team counts
  • Track recent match history
  • Query available leagues and countries
  • Use SIZE() function for array-wrapped responses
Monitoring & DevOps:
  • Query Prometheus for system metrics
  • Access Prometheus response structure (status, resultType)
  • Monitor active targets with SIZE()
  • Use CTEs for multi-step metric processing
  • Default JSON format handling
Recommended Patterns:
  • Use SIZE(array_field) to count array elements
  • Access nested fields with dot notation: field.subfield.value
  • Use CTEs for complex multi-step transformations
  • Use CASE statements for conditional logic and alerts
  • Apply WHERE clauses with SIZE() to filter valid responses

​
Troubleshooting

Issue: API request failed with status code: 401 Solution: Verify authentication credentials and token validity Issue: responseFormat not recognized Solution: Ensure format is one of: json, csv, or xml Issue: Nested field not found Solution: Check the API response structure and use correct dot notation Issue: Column renamed to predicate_0, predicate_1 Solution: For JOINs between multiple sources, use renamed column references Issue: Error accessing deeply nested arrays (e.g., data.result[0].metric.instance) Solution: Array access within deeply nested JSON structures can be unreliable depending on the response format. Use one of these alternatives:
  1. Access top-level fields directly:
  2. Use struct:get in Vadalog rules:
  3. Use SIZE() for array lengths:
Issue: Prometheus metrics with complex nested structures Solution: For Prometheus API responses, focus on accessing:
  • Top-level response fields: status, data.resultType
  • Struct sizes: SIZE(data.activeTargets), SIZE(data.result)
  • Use CTEs and aggregations on response-level data rather than individual metric arrays
For accessing individual metric labels and values within Prometheus results, consider using the struct:get function in Vadalog rules after binding the data.

​
Text Files

Text files can be processed directly in Vadalog to extract concepts and relationships from textual content. The text datasource supports various text formats and can extract structured information from unstructured text.

​
Example: Reading from Text File

This example demonstrates how to read from a text file and extract structured information.

​
Binary Files

Binary files support various formats including PDF, JPG, PNG, and other formats. Vadalog can extract concepts and relationships from binary files, making it suitable for processing documents and images.

​
Example: Reading from PDF File

This example demonstrates how to read from a PDF file and extract structured information.

​
Example: Reading from Business Document (Invoice)

Binary files also support structured business documents such as ID documents, receipts, tax forms, mortgage documents, and other standardized business documents. The API supports various document types including:
  • Financial Documents: check.us, bankStatement.us, payStub.us, creditCard, invoice
  • ID Documents: idDocument.driverLicense, idDocument.passport, idDocument.nationalIdentityCard, idDocument.residencePermit, idDocument.usSocialSecurityCard
  • Receipts: receipt.retailMeal, receipt.creditCard, receipt.gas, receipt.parking, receipt.hotel
  • Tax Documents: tax.us.1040.2023, tax.us.w2, tax.us.w4, tax.us.1095A, tax.us.1098, tax.us.1099 (various forms)
  • Mortgage Documents: mortgage.us.1003 (URLA), mortgage.us.1004 (URAR), mortgage.us.closingDisclosure
  • Other Documents: contract, healthInsuranceCard.us, marriageCertificate.us
This example demonstrates how to read from an invoice PDF and extract structured business information.

​
Example: PDF Sections to Qdrant — Semantic Search over Documents

This example demonstrates a complete workflow for turning unstructured PDF documents into a searchable Qdrant knowledge base and querying it with natural language. The documentType='sections' mode extracts each document section as a row with its heading, content, role (e.g., title, section heading, body), and page number. These sections are then written to Qdrant with concept-aware embeddings, enabling semantic search that understands the document structure.

​
Step 1: Extract sections from PDF documents

Running this on a company annual report might produce rows like:

​
Step 2: Write sections to Qdrant with concept-aware embedding

Each section is embedded as a sentence like:
“Document section titled Financial Overview on page 3 of an annual report: Revenue grew 23% year-over-year to $4.2B…”
This contextual embedding captures not just the text, but its role and position within the document.

​
Step 3: Query with natural language

Result: The query “environmental sustainability initiatives” correctly returns the ESG and sustainability sections — not financial or risk sections that might coincidentally mention the word “environment”.

​
Step 4: Combine with structured reasoning

The power of Vadalog is that search results can feed into further reasoning rules:
This returns only sections about sustainability that contain percentage figures — combining semantic vector search with symbolic reasoning in a single Vadalog program.

​
HDFS File system

HDFS (Hadoop Distributed File System) is designed for distributed storage and large-scale data processing. Vadalog can integrate with HDFS by reading from and writing to files stored in HDFS clusters. This example shows how to read a CSV file from an HDFS location and process it within a Prometheux workflow.

​
Sybase Database

Sybase (now SAP ASE) is a relational database management system used for online transaction processing. This example shows how to read data from a Sybase database.

​
Teradata Database

Teradata is a highly scalable relational database often used in enterprise data warehousing. This example shows how to read data from a Teradata database.

​
Amazon Redshift

Amazon Redshift is a fully managed data warehouse service designed for large-scale data analytics. This example shows how to read data from a Redshift table.

​
Google BigQuery

Google BigQuery is a serverless, highly scalable, and cost-effective multi-cloud data warehouse. This example demonstrates configuring and querying data from a BigQuery dataset.

​
Setting up Google Cloud Access

This guide shows how to:
  1. Enable the required Google Cloud APIs
  2. Create a service account for Prometheux jobs
  3. Grant the minimum IAM roles (data access & Storage API)
  4. Allow a human user to impersonate the service account and obtain short‑lived access tokens
  5. Create a JSON key file
  6. Generate a one‑hour access token
Project ID example: project-example-358816 Service account name example: example-sa Human user example: example@gmail.com
Open a Google Cloud Shell within your Google Project and execute the following commands:

​
1 Enable required APIs

​
2 Create the service account

Resulting e‑mail: example-sa@project-example-358816.iam.gserviceaccount.com

​
3 Grant minimum BigQuery roles to the service account

​
4 Allow your user to impersonate the service account

Verify:

​
5 Create a JSON key file

If you prefer file‑based creds:
Read the content
Copy and paste it into a new file in your laptop or environment in your /path/to/gcp-credentials.json This authMode is the default one. Set the ENV var (in Docker or via EXPORT) GOOGLE_APPLICATION_CREDENTIALS=/path/to/gcp-credentials.json, or set the bigquery.credentialsFile=/path/to/gcp-credentials.json in the px.properties configuration file or declare the path via credentialsFile=/path/to/gcp-credentials.json as an option in the bind annotation.

​
6 Generate a one‑hour access token (impersonation)

If you prefer token‑based creds:
Enable token-based authMode by setting set the bigquery.authMode in the px.propertiesconfiguration file or declare it as an option in the bind annotation authMode=gcpAccessToken and set the ENV var (in Docker or via EXPORT) GCP_ACCESS_TOKEN=my-token, or set the bigquery.gcpAccessToken=my-token config property or pass it via gcpAccessToken=my-token as option in the bind annotation.

​
Example using credentials file

​
Example using token

​
Example via query

​
Snowflake

Snowflake is a cloud-based data warehousing service that allows for data storage, processing, and analytics.
Note on writing: When writing to Snowflake, the default saveMode is error, meaning the write fails if the target table already exists. To replace the existing table, add saveMode='overwrite' to the bind options (e.g. "snowflake saveMode='overwrite', url='...', ...").

​
How to Retrieve Your Snowflake Connection Info for reading or writing tables

To obtain the connection details for your Snowflake account:
  1. Click the user icon in the bottom-left corner of the Snowflake UI.
  2. Select “Connect a tool to Snowflake”.
  3. Go to the “Connectors / Drivers” section.
  4. Choose “JDBC” as the connection method.
  5. Select your warehouse, database, and set “Password” as the authentication method.
This will generate a JDBC connection string in the following format:
From this string, you can extract the following values for your bind configuration:
  • url = 'A778xxx-IVxxxx.snowflakecomputing.com'
  • username = 'PROMETHEUX'
  • password = 'my_password'
  • warehouse = 'COMPUTE_WH'
  • database = 'TEST' (note: database names are usually uppercase)

​
Using Programmatic Access Tokens (PAT)

Snowflake supports Programmatic Access Tokens (PAT) as an alternative to password authentication. This is particularly useful for:
  • Avoiding MFA prompts during automated workflows
  • Enhanced security with token rotation
  • Integration with CI/CD pipelines
To use PAT instead of password authentication, simply replace the password parameter with the PAT token value in your bind configuration.

​
Setting Up PAT in Snowflake

First, identify your Snowflake user:
Then execute the following script to set up PAT authentication:

​
PAT Token Management Commands

​
Example: Reading from Snowflake with Password

This example demonstrates reading data from a Snowflake table using password authentication.

​
Example: Reading from Snowflake with PAT

This example demonstrates reading data from a Snowflake table using Programmatic Access Token (PAT) authentication instead of password. This method avoids MFA prompts during execution.
Note: When using PAT, the syntax remains identical to password authentication - simply replace the password value with your PAT token. The token can be stored securely in px.properties configuration file or environment variables for better security practices.

​
Databricks

Databricks is a cloud-based platform for data engineering and data science.
Note on writing: When writing to Databricks, the default saveMode is error, meaning the write fails if the target table already exists. To replace the existing table, add saveMode='overwrite' to the bind options (e.g. "databricks saveMode='overwrite', ...").

​
Running inside a Databricks cluster (inCluster=true)

When Prometheux runs inside a Databricks cluster, add inCluster=true to the bind. The connector then uses the cluster’s ambient Spark session and native Unity Catalog access — spark.table(...) / saveAsTable(...) / spark.sql(...) — instead of opening a JDBC connection, so no host, token, OAuth2ClientId/OAuth2Secret or /sql/1.0/warehouses/... warehouse path is required. In this mode the third @bind argument is the catalog.schema and the fourth is the table; on write the schema is created if missing.
The same safe-default applies: if you omit saveMode on a write, the connector uses saveMode=errorIfExists (the write fails if the table already exists), so set saveMode='overwrite' or 'append' explicitly. Without inCluster=true, the connector uses the JDBC path shown below (host + OAuth/PAT credentials + a warehouse path as the third argument). This example demonstrates writing data to a Databricks table.
This example demonstrates reading data from a Databricks table.