This repository contains a comprehensive document management and intelligent knowledge retrieval system. It leverages large language models and vector search technologies to provide accurate, context-aware responses based on organizational documentation.
The following diagram illustrates the high-level architecture of the system, including multi-source ingestion pipelines, AI processing layers, and the core infrastructure.
The system supports three distinct pathways for data entry:
- Manual Document Upload: Users can upload PDF, PPTX, and DOCX files directly through the web interface. The system performs automated text extraction and image analysis using dedicated Python parsers.
- Automated Email Integration: Inbound communications (Subject, Body, and Attachments) are captured via n8n workflows and pushed to the system through a secure webhook endpoint.
- External Processed Data: Integration with external services, such as the Mantis Embedding & Conversion service, allows for the direct ingestion of pre-processed data and embeddings into the primary database.
- Embedding Generation: Utilizes the BGE-M3 model for high-dimensional vectorization, ensuring superior semantic understanding and cross-lingual support.
- Precision Reranking: Employs BGE-Reranker-v2-m3 to re-evaluate search results, prioritizing the most relevant information before it is presented to the generation model.
- Large Language Model (LLM): Integrates with Llama3 and Qwen models via self-hosted Ollama instances to generate factual, natural language responses.
The database is built on PostgreSQL 15 with the pgvector extension. It is designed to handle both structured relational data and high-dimensional vector embeddings.
erDiagram
USER {
int id PK
string name
string email UK
string username UK
string password
string department
string role
string avatar_url
timestamp created_at
}
DOCUMENTS {
int id PK
string document_name
string file_type
string project_name
string client_name
string source_id
timestamp created_at
}
DOCUMENT_CHUNKS {
bigint id PK
int doc_id FK
text content
jsonb metadata
tsvector fts
vector embedding
}
DOCUMENT_IMAGES {
int id PK
int doc_id FK
int page_number
int image_index
string image_path
bytea image_data
text description
vector embedding
jsonb metadata
tsvector fts
}
CONTENT_NODES {
bigint id PK
string node_type
string title
text content
string content_hash UK
jsonb metadata
vector embedding
tsvector fts
timestamp created_at
}
CONTENT_RELATIONSHIPS {
bigint id PK
bigint source_node_id FK
bigint target_node_id FK
string relationship_type
timestamp created_at
}
EMAIL_ATTRIBUTES {
bigint node_id PK,FK
string subject
string from_address
string_array to_addresses
timestamp email_date
string message_id UK
string thread_id
}
FILE_ATTRIBUTES {
bigint node_id PK,FK
string original_filename
string file_type
string storage_url
string processing_status
}
IMAGE_ATTRIBUTES {
bigint node_id PK,FK
string image_path
text description
text ocr_text
int page_number
}
DOCUMENTS ||--o{ DOCUMENT_CHUNKS : "contains"
DOCUMENTS ||--o{ DOCUMENT_IMAGES : "contains"
CONTENT_NODES ||--o{ CONTENT_RELATIONSHIPS : "source"
CONTENT_NODES ||--o{ CONTENT_RELATIONSHIPS : "target"
CONTENT_NODES ||--|| EMAIL_ATTRIBUTES : "has"
CONTENT_NODES ||--|| FILE_ATTRIBUTES : "has"
CONTENT_NODES ||--|| IMAGE_ATTRIBUTES : "has"
- User Table: Manages authentication and user profiles.
- Fields: id, name, email, username, password, department, role, avatar_url, created_at.
- Documents Table: Acts as the parent entity for all ingested files.
- Fields: id, document_name, file_type, project_name, client_name, source_id, created_at.
- Document Chunks Table: Stores partitioned text segments and their corresponding vector embeddings.
- Fields: id, doc_id (FK), content, metadata, fts (Full-text search vector), embedding (Vector 1024).
- Document Images Table: Stores extracted images, their descriptions generated by vision models, and associated embeddings.
- Fields: id, doc_id (FK), page_number, image_index, image_path, image_data (BLOB), description, embedding (Vector 1024), metadata, fts.
- Content Nodes (Graph Layer): Centralized node system for unified content management across different sources.
- Fields: id, node_type, title, content, content_hash, metadata, embedding, fts, created_at.
- Attributes Tables: Specialized tables (Email, File, Image attributes) that provide extended metadata for content nodes in the graph layer.
The system is designed with a cloud-native approach, ensuring scalability and robustness.
- Frontend: Built with React, Vite, and TypeScript for a responsive user experience.
- Backend: Developed using Node.js Express and TypeScript to manage business logic and orchestrate AI services.
- Database: PostgreSQL with pgvector for efficient hybrid search (Vector + Full-text).
- Orchestration: Fully compatible with Kubernetes (K8s) for production environments and Docker Compose for localized development.
Execute the initialization script to prepare the schema:
npm run server src/server/scripts/init_db.ts# Backend and Frontend dependencies
npm install
# Python-based document extractors
pip install pymupdf python-pptx mammoth sharpStart both the development server and the frontend client:
npm run dev:all- Hybrid Retrieval: Combines semantic vector search with lexical full-text search using Reciprocal Rank Fusion (RRF).
- Vision-enhanced RAG: Automatically analyzes and indexes visual content within documents to provide comprehensive answers.
- Enterprise Security: Includes structured authentication and multi-tenant isolation via project and client filtering.
Proprietary - AI Research Team