1 year ago
#307043
PurpleGreen
How to use Amazon MSK in C# code to publish a message to Kafka
I am creating a c# application to publish a message to Kafka. In the current version of my application I set up cluster locally using docker (confluentinc-cp and confluentinc-zookeeper). However to run Kafka efficiently, we decided to use Managed Streaming Apache Kafka (MSK) to run a Cloud native kafka cluster. There's already a cluster created by our data-engineering team. There's already an endpoint for private network (bootstrap server) and plaintext url for Zookeeper.
How can I change my ProducerConfig to use MSK instead of localhost
My producer code looks like this:
public class ProducerHostedService : IHostedService
{
private readonly ILogger<ProducerHostedService> _logger;
private readonly IProducer<Null, string> _producer;
public ProducerHostedService(ILogger<ProducerHostedService> logger)
{
_logger = logger;
var config = new ProducerConfig
{
//SecurityProtocol = SecurityProtocol.Ssl,
BootstrapServers = "localhost:9092"
};
_producer = new ProducerBuilder<Null, string>(config).Build();
}
public async Task StartAsync(CancellationToken cancellationToken)
{
for (var i = 0; i < 100; ++i)
{
var order = new OrderRequest()
{
CustomerId = i,
ProductId = i,
OrderId = i,
Quantity = 1,
Status = "New"
};
string message = JsonSerializer.Serialize(order);
_logger.LogInformation(message);
// ProduceAsync creates a topic if not exists
await _producer.ProduceAsync("test", new Message<Null, string>()
{
Value = message
}, cancellationToken);
}
_producer.Flush(TimeSpan.FromSeconds(10));
}
public Task StopAsync(CancellationToken cancellationToken)
{
_producer?.Dispose();
return Task.CompletedTask;
}
}
c#
.net
apache-kafka
aws-msk
confluent-kafka-dotnet
0 Answers
Your Answer