Skip to main content

Kafka Topics

Page Outline

Kafka Topics​

DIA uses Kafka as the primary event bus for asynchronous communication with other agents and services. This event-driven architecture enables real-time reactivity to system changes without blocking operations.

Event-Driven Architecture Benefits:

  • Decoupling: Agents can communicate without direct dependencies
  • Scalability: Events can be processed in parallel across multiple consumers
  • Resilience: Failed events can be retried without affecting the sender
  • Auditability: All events are logged and can be replayed for debugging

The following Kafka topics are used by DIA:

TopicDirectionMessage FormatPurposeConsumer Group
train.resultsMSA → DIAModel training completion eventsNotifies DIA when model training completes, enabling analysis of training patterns and outcomesdia-group
model.metricsMSA → DIAContinuous model performance metricsStreams real-time metrics (drift, AUC, latency) for continuous monitoring and anomaly detectiondia-group
decision.simulation.requestPSA → DIASimulation job requestsReceives simulation requests from Prompt Studio Agent for async processingdia-simulator-group
decision.simulation.resultDIA → PSA/UISimulation completion eventsPublishes simulation results for consumption by PSA and UI componentsN/A (multiple consumers)
governance.alertsGA → DIAPolicy violation and alert eventsReceives governance alerts to incorporate policy changes into decision-makingdia-group

Message Flow Example:

  1. Model Training Completion Flow:

    • MSA completes model training and publishes to train.results
    • DIA consumer receives the event
    • DIA updates its knowledge base with new model information
    • DIA may trigger analysis to understand training patterns

  2. Simulation Request Flow:

    • PSA publishes simulation request to decision.simulation.request
    • DIA consumer receives and validates the request
    • DIA creates simulation job and processes it
    • DIA publishes results to decision.simulation.result
    • PSA and UI components consume results for display

Consumer Group Strategy:

  • dia-group: Main consumer group for general events (train.results, model.metrics, governance.alerts)
  • dia-simulator-group: Dedicated consumer group for simulation requests to enable parallel processing
  • Multiple partitions per topic enable horizontal scaling of consumers