1 year ago

#341333

test-img

Anas Vakyathodi

Data Learn (AWS MSK with Google Dataflow connection issue)

My Google Dataflow Job is keep getting failed while creating "Kafka-Bigquery" dataflow with AWS MSK public access bootstrap server. Is there any way to solve this issue? enter image description here enter image description here

I tried accessing public access bootstrap server by disabling unauthenticated access and enabling IAM based access, and also updated security groups to access from anywhere. also created a topic using EC2 instance terminal by accessing through private connection string. In GCP, I created a dataflow using Kafka-Bigquery template and passed public-access-bootstrap-connection-string, and topic created in EC2 also passed bigquery table id. Expectation: Job creates successfully and table is populated based on kafka message sent to msk server.

Can I connect AWS MSK (Kafka) with Google Dataflow (Kafka-Bigquery) Job (as consumer)?

google-cloud-platform

apache-kafka

google-bigquery

dataflow

aws-msk

0 Answers

Your Answer

Accepted video resources