python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - parquet

parquet4s not returning all records

I have a simple Scala application that uses parquet4s with fs2 to read a set of partitioned records (spread across directories, generated by a Spark job). When I run the app, it only returns a fractio...
test-img

samirbajaj

scala

hadoop

parquet

cats-effect

fs2

Votes: 0

Answers: 1

Latest Answer

It looks like names of directories containing the partitioned data had characters that are not admissible, causing the reader to skip them, thereby returning fewer records.
test-img

samirbajaj

Python Statsmodel Logistic Regression iterate through Parquet file

I am trying to run a logistic regression model on a very large dataset with 2.3 billion observations in Python. I need a standard regression output. Statsmodels with parquet seemed promising: https://...
test-img

HLRA

python

logistic-regression

parquet

statsmodels

large-data

Votes: 0

Answers: 1

Latest Answer

You can read "subset" of your parquet file by using row groups: import random row_group = random.rand_range(self.parquet.num_row_groups) return self.parquet.read_row_group(row_group).to_pan...
test-img

0x26res

Kafka-connect file sink connector write in parquet file format

I am looking for kafka-connect connector that will write from kafka to local file system in parquet file format. I don't want to use the hdfs or s3 sink connector for the same.
test-img

Aman Jain

apache-kafka

parquet

apache-kafka-connect

Votes: 0

Answers: 1

Latest Answer

format.class=ParquetFormat only exists in the mentioned connectors You can use file:// prefix to write to local disk, or use a project like MinIO to reproduce self-hosted S3
test-img

OneCricketeer

Is it possible to reopen ParquetWriter after close() is called?

I'm currently using ParquetWriter to write Avro records to parquet files. I can use the write() and close() methods to write and close files as needed. Now I have a use case where I need to reopen a c...
test-img

WZH

parquet

parquet-mr

Votes: 0

Answers: 0

Posts

Questions

Blogs

Jobs