No library - no problem.
#735

@evrial:

• library allows customizing/overriding a number of metadata. Without it we'd be stuck with in-ZIM metadata.
• watching a directory would absolutely not solve this issue. An external program would not be able to know when kiwix-serve has detected and reloaded for a specific ZIM. For the cache example above, there would be no improvement.

HI @rgaudin @evrial @kelson42 ,

Added my thoughts in this comment. Requesting your response.

Proposal: Automating Cache Purge with Kafka in Kiwix

Overview

This proposal outlines a Kafka-based architecture to streamline the process of purging the cache when the Kiwix server reloads its library. Instead of invoking the purge API manually from library-maintain.py, this approach leverages Kafka for event-driven communication, enhancing scalability and reliability.

Architecture Design

Components

1. Kiwix Server

   • Reloads the library path upon updates.
   • Publishes a LibraryReloaded event to a Kafka topic.

2. Kafka Broker

   • Acts as a messaging layer for event-driven communication.
   • Stores the LibraryReloaded event in a dedicated topic.

3. Cache Purge Daemon

   • Subscribes to the LibraryReloaded topic.
   • Listens for updates and triggers the cache purge automatically.

Workflow

1. Library Update

   • The library-maintain.py script updates the library.
   • Kiwix server reloads the library and detects successful completion.

2. Kafka Event Emission

   • Kiwix server publishes a LibraryReloaded message to the Kafka topic kiwix.sever.events.

3. Daemon Processing

   • A background daemon subscribes to Kafka topic kiwix.server.events.
   • On receiving a LibraryReloaded message, it executes the cache purge process.

4. Automatic Cache Purge

   • The daemon clears outdated cache files or triggers a specific API internally.

Implementation Details

Kafka Setup

• Deploy a Kafka broker with a kiwix.sever.events topic.
• Configure retention policies to avoid excessive storage.

Kiwix Server Integration

• Modify the server to publish a message upon successful library reload:

  void produce_message(const std::string& broker, const std::string& topic, const std::string& message) {
      rd_kafka_conf_t* conf = rd_kafka_conf_new();
      rd_kafka_t* rk;
      char errstr[512];
  
      rk = rd_kafka_new(RD_KAFKA_PRODUCER, conf, errstr, sizeof(errstr));
      rd_kafka_topic_t* rkt = rd_kafka_topic_new(rk, topic.c_str(), NULL);
      rd_kafka_produce(rkt, RD_KAFKA_PARTITION_UA, RD_KAFKA_MSG_F_COPY,
                       const_cast<char*>(message.c_str()), message.size(),
                       NULL, 0, NULL);
      rd_kafka_flush(rk, 1000);
      rd_kafka_topic_destroy(rkt);
      rd_kafka_destroy(rk);
  }
  

Cache Purge Daemon

• Subscribe to the Kafka topic and trigger cache purge:

  consumer = KafkaConsumer('kiwix.server.events', bootstrap_servers='localhost:9092')
  for message in consumer:
      if b'LibraryReloaded' in message.value:
          purge_cache()  # Custom cache purge function
  

Benefits

• Decoupling: Eliminates direct dependencies between library-maintain.py and the purge process.
• Scalability: Kafka allows multiple services to listen for updates without modifying the core logic.
• Reliability: Ensures cache purge happens even if the initial process fails due to transient issues.

Conclusion

This Kafka-based approach improves the efficiency of cache purging by making it event-driven, reducing manual intervention, and enhancing scalability. By implementing this architecture, Kiwix can achieve a more resilient and automated workflow for managing library updates.

@Optimus-NP We can not rely on another architecture element (like Kafka) to do that. It should work only with Varnish and Kiwix Server.