backpressure - How to apply back pressure with AWS Lambda and MSK (Kafka) event stream

Question

Is it possible to do this?

When dealing with SQS event streams, if the Lambda function does not have adequate reserved concurrency, the function will be throttled, and the unprocessed events / messages can be retried via the SQS redrive policy. I've never liked this limitation as unprocessed messages will eventually end up on the DLQ after some arbitrary number of retries / message visibility timeout.

From my naive perspective, it would appear that the above solution would not be possible with MSK, as placing a message back on an MSK topic for some visibility timeout would effectively lose topic delivery order.

I've searched around but can't find any detail as to how back pressure can be implemented with MSK to Lambda. Does anybody have any insight into how the MSK topic consumer handles Lambda throttling?

Many thanks!

question from:https://stackoverflow.com/questions/65644736/how-to-apply-back-pressure-with-aws-lambda-and-msk-kafka-event-stream

Answer 1

Actually apart from support DLQ's this scenario is supported. But the understanding of how MSK works is a bit different from SQS. In MSK (Which is Apache Kafka) records are persistent and durable; and what indicates for the processors that a given record should be retried is a flag called committed-offset that consumers create. If the lambda function reads the record but doesn't finish its processing then it is just a matter of not committing its respective offset that in the next poll cycle the record will be picked up again.

Also, Kafka has a polling model instead of push. In this case, your lambda function performs a poll indicating how many records must be read on each poll. So you see; there is lots of controls in Kafka to implement backpressure -- just not exactly how it works in SQS.

The example below may give you an idea of how it works:

https://github.com/aws-samples/integration-sample-lambda-msk