python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - large-data

Python Statsmodel Logistic Regression iterate through Parquet file

I am trying to run a logistic regression model on a very large dataset with 2.3 billion observations in Python. I need a standard regression output. Statsmodels with parquet seemed promising: https://...
test-img

HLRA

python

logistic-regression

parquet

statsmodels

large-data

Votes: 0

Answers: 1

Latest Answer

You can read "subset" of your parquet file by using row groups: import random row_group = random.rand_range(self.parquet.num_row_groups) return self.parquet.read_row_group(row_group).to_pan...
test-img

0x26res

How to feed in a list of numpy arrays into a TensorFlow model?

I have a large list of numpy arrays that I want to feed into a TensorFlow model. I can not concatenate the lists into one due to RAM memory issues. Below, I have recreated the dataset I have with the ...
test-img

Ossz

python

tensorflow

large-data

tensorflow-datasets

Votes: 0

Answers: 1

Latest Answer

It is easy using h5py, I use a bit of time correct and fixed all those error messages when the option os.environ['TF_GPU_ALLOCATOR'] = 'cuda_malloc_async' is help allocate memory issues of the GPU in ...
test-img

Jirayu Kaewprateep

DevExpress WPF GridContol Export Very Large Data to Excel File

I'm trying to export a DevExpress GridControl with more than 2 millions rows, then error 'OutOfMemory exception' appears . What is the best way to export and handle this? this my code retrieve from Da...
test-img

husham1414

c#

wpf

devexpress

export-to-excel

large-data

Votes: 0

Answers: 0

Pandas sum values between two dates in the most efficient way?

I have a dataset which shows production reported every week and another reporting the production every hours over some subproduction. I would now like to compare the sum of all this hourly subproducti...
test-img

fidu13

pandas

datetime

sum

large-data

between

Votes: 0

Answers: 1

Latest Answer

Store datetime as pd.Timestamp, then you can do all kinds of manipulation on the dates. For your problem, they is to group the hourly data by week (starting on Mondays), then merge it with the weekly ...
test-img

Code Different

Posts

Questions

Blogs

Jobs