python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - spark-structured-streaming

Generate Kafka message with Headers using Apache Spark

I have an ETL (spark-scala). After writing in a table, a message with "header" must be sent to Kafka. I couldn't add the header in the message. I have a spark DataFrame with the "key&qu...
test-img

sherifdewichita

scala

apache-spark

apache-kafka

spark-structured-streaming

Votes: 0

Answers: 1

Latest Answer

The easiest way of add headers is using withColumn as you tried in your example. Headers must be an array<struct key:string,value:binary>, so make sure your column meet this format. You don't ne...
test-img

kuro

Spark spark-sql-kafka - java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArraySerializer

I am experimenting with spark reading from a kafka topic through "Structured Streaming + Kafka Integration Guide". Spark version: 3.2.1 Scala version: 2.12.15 Following their guide on the sp...
test-img

abaschkim

apache-spark

apache-kafka

apache-spark-sql

spark-structured-streaming

Votes: 0

Answers: 1

Latest Answer

The logs seem fine, but you can try to include kafka-clients dependency in --packages argument as well Otherwise, I'd suggest creating an uber jar instead of downloading libraries every time you submi...
test-img

OneCricketeer

Driver memory not getting cleaned up in Spark Structured Streaming

I am using my Spark Structured Streaming job to perform my ETL in AWS platform My Driver memory is not getting cleared-up. The job is reading the events from Kinesis and writing to S3 Below are the my...
test-img

Ankur Shrivastava

amazon-web-services

apache-spark

pyspark

spark-streaming

spark-structured-streaming

Votes: 0

Answers: 1

Latest Answer

We had seen this when Spark UI in yarn had lots of jobs being listed. We limited the no.of jobs (for ex. to 100 etc), tasks and stages shown in UI. This helped decrease driver memory usage. You can gi...
test-img

Vindhya G

changing parquet file column value without changing name of the parquet file so that _spark_metadata is not changed

So I have a spark structured streaming job that stores flattened json messages dataframes in a hdfs location in parquet format in date partitioned folders i.e /source/wireless/active_portal/activation...
test-img

Queen Anonymous

apache-spark

hive

spark-structured-streaming

Votes: 0

Answers: 1

Latest Answer

You can't modify a file on HDFS, so you have to use Hive. Assuming your table isn't a transactional table, you can't modify the data in place, so you have to wash it through another table. Here's a du...
test-img

Andrew

Posts

Questions

Blogs

Jobs