python (65.1k questions)

javascript (44.2k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (12.9k questions)

Questions - distributed-computing

Create sparse matrix in parallel in Julia

I am trying to parallelize the creation of a sparse matrix in Julia. Inspired by this post this post I am trying this: using Distributed addprocs(4) @everywhere using DistributedArrays rows = [Int[] ...
test-img

Shep Bryan

parallel-processing

julia

sparse-matrix

distributed-computing

Votes: 0

Answers: 1

Latest Answer

distribute(rows) does not modify rows to be distributed; it returns a new distributed array filled with the input. You have to work with its result, something like rows = distribute([Int[] for _ in p...
test-img

phipsgabler

Apache Beam - use output of one aggregation in another aggregation

I am new to Apache Beam, coming over from PySpark's dataframe API. I'm having trouble using the output of one beam calculation in another beam calculation. Basically, I want to perform an aggregation ...
test-img

alexvas

python

apache-beam

distributed-computing

Votes: 0

Answers: 1

Latest Answer

What you need is a Side Input. Here's the programming guide about them. Working solution (note that I removed one of the prints so that there's an actual output there): import apache_beam as beam from...
test-img

Iñigo

How to monitor+report failures of async invoked concurrent lambdas?

I have this lambda in which the first instance invokes itself 30-40 times to process data concurrently. Invocation happens using the async fire and forget Event invocation type. The very first instanc...
test-img

Amol Borkar

amazon-web-services

aws-lambda

distributed-computing

Votes: 0

Answers: 1

Latest Answer

You can report on the number of instances triggered using the first Lambda invocation. For failures, it is not possible to use the same Lambda instance, as the error may happen much further down the l...
test-img

Register Sole

why equal partitioned data works faster?

repartition() redistribute the data among different nodes with new partitions and coalesce does the same thing but it keeps the some of original partitions without shuffling and add others partitions ...
test-img

Sophia

apache-spark

distributed-computing

Votes: 0

Answers: 1

Latest Answer

A 'partition' of data is processed by a 'task' as part of a 'stage'. A stage has many tasks that run in parallel. A Spark 'app' consists of multiple stages. The next stage can only start when the prio...
test-img

thebluephantom

Posts

Questions

Blogs

Jobs