Skip to main content

Writer Pipeline

Overview​

The Writer Pipeline writes content into a selected vector store index.
The content can be plain text, JSON, vectors, or mixed structured data, and is stored along with validated metadata.

This pipeline acts as the final persistence layer by ensuring:

  • Content is written to the correct index
  • Metadata follows the index schema
  • Constraints defined during index creation are enforced

What This Pipeline Does​

  • Writes content to a selected vector store index
  • Supports multiple content types:
    • Text
    • JSON
    • Vectorized data
    • Mixed structured payloads
  • Maps JSON fields to index metadata when applicable
  • Validates metadata against index constraints
  • Handles duplicate documents using configurable policies
  • Supports auto-generated metadata values such as timestamps and auto-increment IDs
info
  • The agent must be configured with a Vector Store dataset
  • A target index must be selected
  • All content is written to the selected index

Metadata-Aware Writing​

The Writer Pipeline validates and applies the metadata schema defined for the target index.

During writes, it:

  • Maps matching JSON keys to metadata fields
  • Enforces index-level metadata constraints

Supported Metadata Rules​

  • Required fields: write fails if missing
  • Default values: applied automatically when absent
  • Choice constraints: only predefined values are accepted
  • Auto-increment fields: values are generated automatically
  • Datetime fields: supports auto-timestamp or validated datetime input

All metadata validation occurs before data is written to the index.

How Content Is Handled​

  • JSON input
    Matching keys are written as metadata, and remaining fields are stored as document content.

  • Text or string input
    Stored directly as document content.

  • List input
    Each item is written as a separate document.

If a conflict occurs, metadata values take precedence over content fields.

Input and Output​

Input​

  • String
  • JSON object
  • List of strings or JSON objects
  • Parsed Document objects

Output​

  • Number of documents written
  • List of generated document IDs
  • Validation or runtime errors, if any

Typical Usage Flow​

  1. Create a Vector Store dataset
  2. Define metadata schema and constraints while creating the index
  3. Configure the agent with:
    • Vector Store dataset
    • Target index
  4. Add the Writer Pipeline to the agent workflow
  5. Send content (text, JSON, or vectors)
  6. The pipeline validates metadata and writes content to the index

Error Handling​

The pipeline handles errors for:

  • Missing required metadata
  • Invalid choice values
  • Metadata constraint violations
  • Duplicate document conflicts
  • Vector store runtime issues

Summary​

The Writer Pipeline provides schema-aware persistence of content into vector stores.
By validating metadata, enforcing constraints, and supporting flexible input formats, it enables reliable indexing for search, retrieval, and RAG-based workflows.