We are receiving a lot of Kafka Confluent Control Center alerts related to topic being under replicated. We think these are not real issues because of the alerts bouncing in and off. This may be caused by having a tight value for replica.lag.time.max.ms. This setting controls when a replica is considered out of sync and thus removed from the In-Sync replicas list.

We could relax this value and received less alerts, but how do we guarantee this not becomes an issue of hiding real problems.

Is there an expected normal # of these alerts we can target to? Or are there any other metrics we can also use to assess the quality of our replicas after relaxing the setting?