1 year ago

#307043

test-img

PurpleGreen

How to use Amazon MSK in C# code to publish a message to Kafka

I am creating a c# application to publish a message to Kafka. In the current version of my application I set up cluster locally using docker (confluentinc-cp and confluentinc-zookeeper). However to run Kafka efficiently, we decided to use Managed Streaming Apache Kafka (MSK) to run a Cloud native kafka cluster. There's already a cluster created by our data-engineering team. There's already an endpoint for private network (bootstrap server) and plaintext url for Zookeeper.

How can I change my ProducerConfig to use MSK instead of localhost

My producer code looks like this:

public class ProducerHostedService : IHostedService
{
    private readonly ILogger<ProducerHostedService> _logger;
    private readonly IProducer<Null, string> _producer;

    public ProducerHostedService(ILogger<ProducerHostedService> logger)
    {
        _logger = logger;
        var config = new ProducerConfig
        {
            //SecurityProtocol = SecurityProtocol.Ssl,
            
            BootstrapServers = "localhost:9092"
        };
        _producer = new ProducerBuilder<Null, string>(config).Build();
    }
    
    public async Task StartAsync(CancellationToken cancellationToken)
    {
        for (var i = 0; i < 100; ++i)
        {
          var order = new OrderRequest()
                      {
                          CustomerId = i,
                          ProductId = i,
                          OrderId = i,
                          Quantity = 1,
                          Status = "New"
                      };
            string message = JsonSerializer.Serialize(order);
            _logger.LogInformation(message);
            // ProduceAsync creates a topic if not exists
            await _producer.ProduceAsync("test", new Message<Null, string>()
            {
                Value = message
            }, cancellationToken);
        }

        _producer.Flush(TimeSpan.FromSeconds(10));
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        _producer?.Dispose();
        return Task.CompletedTask;
    }
}

c#

.net

apache-kafka

aws-msk

confluent-kafka-dotnet

0 Answers

Your Answer

Accepted video resources