python (65.2k questions)

javascript (44.3k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (13k questions)

Questions - spark-bigquery-connector

Upload the PySpark dataframe to bigquery as a dataproc job

I'm trying to submit a PySpark job on Dataproc cluster. My Pyspark job is uploading a dataframe to bigquery. When I do it using submit job on the cluster, I face an error, the job fails. But, when I p...
test-img

Mano Bhargav

pyspark

google-bigquery

jar

google-cloud-dataproc

spark-bigquery-connector

Votes: 0

Answers: 1

Latest Answer

First of all, the mentioned is a must when reading and writing to BigQuery. If you don't want to add it to the job submission, you can add the BigQuery connector jar on cluster creation using the conn...
test-img

David Rabinowitz

Bigquery as metastore for Dataproc

We are trying to migrate pyspark script from on-premise which creates and drops tables in Hive with data transformations to GCP platform. Hive is replaced by BigQuery. In this case, the hive reads and...
test-img

Murli Krishnan

apache-spark

google-bigquery

google-cloud-dataproc

spark-bigquery-connector

Votes: 0

Answers: 1

Latest Answer

I don't think Spark SQL will support BigQuery as metastore, nor BQ connector will support BQ DDL. On Dataproc, Dataproc Metastore (DPMS) is the recommended solution for Hive and Spark SQL metastore. I...
test-img

Dagang

Posts

Questions

Blogs

Jobs