Prompt-Knowledge Validation

Use prompt validation to verify that AI prompts reference valid, current knowledge before execution.

Overview

This feature validates AI prompts against a versioned knowledge base. It detects missing or stale references and produces structured reports teams can use for release gates and runtime safeguards.

What Engineers Use It For

• Enforce reference checks before model execution
• Fail fast when prompts cite missing or outdated artifacts
• Run strict/tolerant modes by workflow risk profile
• Attach validation outcomes to decision traces

Features

• 3-Layer Validation Architecture
  • Reference Extraction - Find all knowledge references in prompts
  • Reference Resolution - Verify references exist in knowledge base
  • Semantic Validation - Check semantic consistency using LLM
• Multiple Validation Modes - Strict, tolerant, warn-only
• lakeFS Integration - Direct validation against versioned knowledge
• Detailed Reports - Understand validation failures and warnings

Quick Start

from briefcase_ai.validation import PromptValidationEngine

# Create validator with lakeFS connection
validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="strict"
)

# Validate a prompt
result = validator.validate(
    prompt="Based on policy DOC-2024-v3, is this claim eligible?"
)

if result.is_valid:
    print("Prompt is valid")
else:
    print(f"Validation failed: {result.errors}")
    for warning in result.warnings:
        print(f"  Warning: {warning}")

Constructor

PromptValidationEngine(
    versioned_client,                # Required - lakeFS client
    repository: str,              # Required - repository name
    branch: str = "main",         # Branch name
    mode: str = "strict",         # "strict", "tolerant", or "warn_only"
    enable_semantic: bool = False, # Enable LLM-based semantic checks
    llm_client=None               # Optional LLM client for semantic layer
)

Validation Layers

Layer 1: Reference Extraction

Identifies all knowledge references in the prompt:

# Automatically extracts references like:
# - "DOC-2024-v3" (document references)
# - "section 3.2" (section references)
# - "policy #1234" (policy number references)
# - "guideline X" (guideline references)

# Extraction happens automatically during validate()
result = validator.validate(
    prompt="Per DOC-2024-v3, section 3.2, policy #1234 states that..."
)

# References found are available in the result
for ref in result.extracted_references:
    print(f"{ref.type}: {ref.value}")

Layer 2: Reference Resolution

Verifies that references exist in the knowledge base (lakeFS):

# Resolution happens automatically during validate()
# The validator checks each extracted reference against
# the configured lakeFS repository and branch

result = validator.validate(prompt="Per DOC-2024-v3, section 3.2...")

# Check resolution results
for ref_result in result.resolution_results:
    if ref_result.found:
        print(f"Found: {ref_result.reference}")
    else:
        print(f"Missing: {ref_result.reference}")

Layer 3: Semantic Validation

Checks semantic consistency using LLM (when enabled):

# Enable semantic validation
validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="strict",
    enable_semantic=True,
    llm_client=my_llm_client
)

result = validator.validate(
    prompt="Using policy DOC-2024-v3, is the claim eligible?"
)

# Semantic scores available when semantic validation is enabled
if hasattr(result, 'semantic_scores'):
    for ref, score in result.semantic_scores.items():
        print(f"Confidence for {ref}: {score:.1%}")

Validation Modes

Strict Mode

Rejects any unresolved references or semantic issues:

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="strict"
)

result = validator.validate(prompt=prompt)

# Fails if:
# - Any reference unresolved
# - Semantic inconsistency detected
# - Confidence < 0.95
assert result.is_valid, f"Validation failed: {result.errors}"

Tolerant Mode

Allows minor issues but validates core references:

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="tolerant"
)

result = validator.validate(prompt=prompt)

# Passes if:
# - Core references resolved
# - Semantic confidence > 0.7
# - No critical inconsistencies

if result.warnings:
    log_warnings(result.warnings)

Warn-Only Mode

Logs issues but doesn't fail validation:

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="warn_only"
)

result = validator.validate(prompt=prompt)

# Always passes but provides:
result.errors       # All identified issues
result.warnings     # Non-critical warnings

ValidationReport

Returned from validate():

class ValidationReport:
    is_valid: bool                   # Validation passed
    mode: str                        # Validation mode used
    errors: List[ValidationError]    # Critical errors
    warnings: List[ValidationError]  # Non-critical warnings

    # Detailed breakdown
    extracted_references: List[Reference]      # Found references
    resolution_results: List[ResolutionResult] # Per-reference results

    # Metadata
    validation_timestamp: datetime
    execution_time_ms: float

Reference Types

The validator recognizes and extracts multiple reference types:

# Supported reference patterns:
# - Document IDs: "DOC-2024-v3", "policy.pdf"
# - Section references: "section 3.2"
# - Policy IDs: "policy #1234", "POL-2024-001"
# - File paths: "policies/claims.txt"
# - Versioned files: "policy@abc123" (file@commit)

Example:

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo"
)

result = validator.validate(
    prompt="""
    Per policy DOC-2024-v3, section 3.2:
    This claim requires verification per guideline X.
    Reference control CC6.1 for audit requirements.
    """
)

# Extracted references available in result
for ref in result.extracted_references:
    print(f"Type: {ref.type}, Value: {ref.value}")

Usage Patterns

Pre-Execution Validation

from briefcase_ai.validation import PromptValidationEngine

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="strict"
)

def execute_ai_decision(prompt):
    # Validate before execution
    result = validator.validate(prompt=prompt)

    if not result.is_valid:
        raise ValueError(f"Prompt validation failed: {result.errors}")

    # Proceed with AI execution
    return run_model(prompt)

Batch Validation with Reporting

validator = PromptValidationEngine(
    versioned_client=lakefs,
    repository="example-policy-repo",
    mode="tolerant"
)

prompts = [
    "Based on DOC-2024-v3, section 3.2...",
    "Per policy #1234, approve if...",
    "According to guideline X..."
]

results = [validator.validate(prompt=p) for p in prompts]

# Generate report
valid_count = sum(1 for r in results if r.is_valid)
warning_count = sum(len(r.warnings) for r in results)
error_count = sum(len(r.errors) for r in results)

print(f"Validation Summary")
print(f"  Valid: {valid_count}/{len(results)}")
print(f"  Warnings: {warning_count}")
print(f"  Errors: {error_count}")

Installation

Prompt validation is included in the core SDK:

pip install "briefcase-ai"

Optional: lakeFS Integration

For validation against lakeFS knowledge bases:

pip install "briefcase-ai[lakefs]"

Migration Note

briefcase remains available as a compatibility alias in 2.1.30. Use briefcase_ai imports for all new code. Alias removal is planned for 2.1.31.