rabbitmq - Message loss issues reported in production

Question

We have received multiple message loss issues in PROD recently.

Upon investigating such issues we have so far concluded that

1. Publishers are sending messages to RabbitMQ using messaging framework component

2. We have seen only some of the messages get published into RabbitMQ queues and have seen the same using RabbitMQ queue REST API by monitoring the "publish" and "deliver" fields in message_stats before sending the messages and after sending the messages.

in a recent use case if say 28 messages were sent the "publish" and and "deliver" that give the cumulative values got increased by only 20 and 8 messages were lost.

Also the logging implemented in messaging framework confirmed this behavior .The messaging framework recorded the logging of 28 incoming messages wheras the Post processor on the receiving side recorded only loggings for 20 messages.

3. Since we are using Spring AMQP for publishing the messages , so in the messaging framework we are not having control to further debug what is going wrong between Spring AMQP and RabbitMQ.

There was one more observation that we did the restart of the application and it solved the issue once in one particular use case and we could receive the logging of all messages on sender as well as receivng side.

Need some suggestion over here .Also recently we have rolled out the pause minority configuration in PROD without publish confirm. Can it be the cause of message loss. We will investigate the logs further though regarding this.

Any help would be beneficial over here

Regards,

Apoorv

question from:https://stackoverflow.com/questions/65841974/message-loss-issues-reported-in-production

Answer 1

Waitting for answers