Microservices API
RiverGen uses a microservices architecture with separate services for specialized operations. These services are typically called internally by the main API but can also be accessed directly for advanced use cases.
Quick Navigation
Overview
Microservices provide specialized functionality that is separated from the main API for scalability, maintainability, and independent deployment. Each service has its own API endpoints and can be scaled independently.
Architecture
RiverGen's microservices architecture consists of:
- Data Orchestration Service - Query execution, catalog management, and data lineage
- Storage Service - File storage, quota management, and model registry
- Staging Service - Data staging for Model Studio consumption
These services communicate with the main API through HTTP requests and share the same database and infrastructure.
Authentication
Microservices use header-based authentication:
X-Org-ID: <organization_id>
X-User-ID: <user_id> (optional)
Note: Most microservices are called internally by the main API, which handles authentication and passes the organization context via headers.
Services
Data Orchestration Service
Base URL: http://data-orchestration:8000 (internal) or http://localhost:8000 (development)
Purpose: Handles query execution, data catalog management, schema discovery, and data lineage tracking.
Endpoints: 17 endpoints
- Health & Status (2)
- Execution (2)
- Connectors (3)
- Catalog (7)
- Lineage (3)
Documentation: Data Orchestration Service
Storage Service
Base URL: http://storage-service:8002 (internal) or http://localhost:8002 (development)
Purpose: Manages file storage, quota tracking, vector storage, caching, and model artifacts.
Endpoints: 22 endpoints
- Files - Public API (5)
- Quota - Public API (2)
- Vectors - Internal API (3)
- Files - Internal API (2)
- Cache - Internal API (3)
- Lifecycle - Internal API (3)
- Model Registry - Public API (4)
Documentation: Storage Service
Staging Service
Base URL: http://staging-service:8003 (internal) or http://localhost:8003 (development)
Purpose: Handles data staging operations for Model Studio, converting data sources to Parquet format for ML consumption.
Endpoints: 8 endpoints (Not Implemented)
- Job Management (4)
- Job Control (2)
- Chunk Management (2)
Status: All endpoints return 501 Not Implemented
Documentation: Staging Service
Base URLs
| Service | Internal URL | Development URL | Port |
|---|---|---|---|
| Data Orchestration | http://data-orchestration:8000 | http://localhost:8000 | 8000 |
| Storage Service | http://storage-service:8002 | http://localhost:8002 | 8002 |
| Staging Service | http://staging-service:8003 | http://localhost:8003 | 8003 |
Internal Notes
- Microservices are typically accessed internally by the main API
- Direct access to microservices requires proper network configuration
- All services share the same PostgreSQL database
- Services use header-based authentication (
X-Org-ID,X-User-ID) - Internal APIs (marked as "Internal") are for service-to-service communication only
- Public APIs can be accessed directly but require proper authentication headers