python (65.2k questions)

javascript (44.3k questions)

reactjs (22.7k questions)

java (20.8k questions)

c# (17.4k questions)

html (16.3k questions)

r (13.7k questions)

android (13k questions)

Questions - spark-window-function

find the top n unique values of a column based on ranking of another column within groups in pyspark

I have a dataframe like below: df = pd.DataFrame({ 'region': [1,1,1,1,1,1,2,2,2,3], 'store': ['A', 'A', 'C', 'C', 'D', 'B', 'F', 'F', 'E', 'G'], 'call_date': ['2022-03-...
test-img

zesla

python

pyspark

pyspark-pandas

spark-window-function

Votes: 0

Answers: 4

Latest Answer

Try with groupby: >>> df.sort_values("call_date").drop_duplicates("store").groupby("region").apply(lambda x: x.nlargest(3, "call_date")).reset_index(dr...
test-img

not_speshal

Compare consecutive rows and extract words(excluding the subsets) in spark

I am working on a spark dataframe. Input dataframe looks like below (Table 1). I need to write a logic to get the keywords with maximum length for each session ids. There are multiple keywords that wo...
test-img

Abhi Sinha

scala

apache-spark

apache-spark-sql

spark-window-function

Votes: 0

Answers: 1

Latest Answer

You can solve it by using lead function: val windowSpec = Window.orderBy("session_id") dfM .withColumn("lead",lead("value",1).over(windowSpec)) .filter((functions.len...
test-img

Gabio

Find number of null records between two non-null records in scala dataframe

I have a dataframe as shown below. | ID | date | sig01_diff | +-----+------------+------------+ | 123 | 2019-11-04 | 93668 | | 123 | 2019-11-05 | 49350 | | 123 | 2019-11-07 | null ...
test-img

Antony

dataframe

scala

apache-spark

apache-spark-sql

spark-window-function

Votes: 0

Answers: 2

Latest Answer

It seems like there are no direct functions to find out number of null records between two non-null records. Below provided a working solution for this question. %scala import spark.implicits._ import...
test-img

Antony

Posts

Questions

Blogs

Jobs