AnswerBun.com

How to write to a csv within a pandas UDF in pyspark?

Stack Overflow Asked by codemaster2020 on December 26, 2020

This is the code that I have tried to write the CSV file.
Spark data frame would be written into a CSV file using pandas

from pyspark.sql.functions import pandas_udf,PandasUDFType
import os
import csv

df3 = spark.createDataFrame(
[("a", 1, 0), ("a", -1, 42), ("b", 3, -1), ("b", 10, -2)],
("key", "value1", "value2")
)

from pyspark.sql.types import *

schema = StructType([
    StructField("key", StringType()),
    StructField("avg_value1", DoubleType()),
    StructField("avg_value2", DoubleType()),
    StructField("sum_avg", DoubleType()),
    StructField("sub_avg", DoubleType()),
    StructField("result", StringType())
])

@pandas_udf(schema, PandasUDFType.GROUPED_MAP)
def g(df):
    gr = df['key'].iloc[0]
    x = df.value1.mean()
    y = df.value2.mean()
    w = df.value1.mean() + df.value2.mean()
    z = df.value1.mean() - df.value2.mean()
    
    fileName = '/mnt/test' + gr + '.csv'   
    df.to_csv(fileName, sep='t')  
    a = "Saved"
    return pd.DataFrame([[gr]+[x]+[y]+[w]+[z]+[a]])

df3.groupby("key").apply(g).show()

Output:

+---+----------+----------+-------+-------+------+
|key|avg_value1|avg_value2|sum_avg|sub_avg|result|
+---+----------+----------+-------+-------+------+
|  a|       0.0|      21.0|   21.0|  -21.0| Saved|
|  b|       6.5|      -1.5|    5.0|    8.0| Saved|
+---+----------+----------+-------+-------+------+

But the CSV files are not getting created.

Any suggestions would be appreciated.

Add your own answers!

Related Questions

How to fix dict to pandas dataframe

1  Asked on February 23, 2021 by spatial-digger

     

C Polynomial Addition exercise

3  Asked on February 23, 2021 by aaa

   

Fill a list with a for loop python

3  Asked on February 23, 2021 by dottorbooom

         

Sending ics file with MailKit and Stringbuilder

1  Asked on February 23, 2021 by nick-fleetwood

         

How to convert UINT16 into two signed chars?

1  Asked on February 22, 2021 by the-big-pescado

       

Understanding the increment in the array (v6++)[4] = v9;

2  Asked on February 22, 2021 by john-wick

 

Decrementing a number in c as an argument

1  Asked on February 22, 2021

 

$(ProjectName) and $(TargetName) MSBuild macros not working

2  Asked on February 22, 2021 by ivan-caravanio

     

Android: Using Microphone and Headphones Simultaneously

1  Asked on February 22, 2021 by poppy

     

How to print specific words from a sentence

1  Asked on February 22, 2021 by astronomical

 

getting parsing issue in GraphQL query

0  Asked on February 22, 2021 by sauribabu

       

Ask a Question

Get help from others!

© 2022 AnswerBun.com. All rights reserved. Sites we Love: PCI Database, MenuIva, UKBizDB, Menu Kuliner, Sharing RPP