I am trying to write a dataframe in parquet format to hdfs. The dataframe contains all values before writing but when i write it to the disk, it converts the boolean values (true,false) to null.

Here is the code:

    val finalDF = dataFrame.select(dataFrame.columns.map(c => col(c).cast(StringType)) : _*).select(dataFrame.columns.map(x => col(x).as(x.toLowerCase)): _*)
    print("final DF before writing")
    finalDF.show(4)
    finalDF.write.partitionBy(pushStreamInstance.getPartitionColsForHive.map(name => name): _*).option("compression","none").mode("append").parquet(pushStreamInstance.getHiveOutputPath)

When i show first 4 row of the dataframe then i get following data:

final DF before writing+-------+-----------+------+-----------+-----------+--------+---------------------------+----------+---------------------------+--------------------------+---------------------+------------+------------+---------+-------------+-------------+----------+------------+-----------------+--------------------+-------+----------------+------------+--------+-----------+---------+--------------+------------------+-------------+------------------+------------+------------+----------------+------------------------+----------------+----------------+------------------------+--------------------+-------------+--------------------+----------+--------------------------+-------------+---------------------------+--------------------+-----------------------+-----------+--------+---------------+-------------------------+-------------+-------+----------------+--------------------+----------+-------------+
|hd_conn|stb_profile|action|action_type|call_letter|category|channel_external_identifier|channel_id|client_software_version_app|client_software_version_mw|company_location_name|company_site| device_name|event_cat|event_counter|   event_time|event_type|eventcounter|hdmi_connectivity|household_identifier|hw_type|invocation_point|  ip_address| item_id| item_title|item_type|language_audio|language_interface|language_shop|language_subtitles|         mac| mac_address|oauth2_client_id|operation_system_version|program_category|program_duration|program_reference_number|program_sub_category|program_title|recommendations_flag|resolution|scheduled_trail_identifier|   start_time|stream_adaptivity_indicator|  stream_content_url|stream_control_protocol|stream_mode|sub_type|subscriber_type|targeted_advertising_flag|time_position|   type|viewing_duration|  viewing_identifier|event_date|logstash_date|
+-------+-----------+------+-----------+-----------+--------+---------------------------+----------+---------------------------+--------------------------+---------------------+------------+------------+---------+-------------+-------------+----------+------------+-----------------+--------------------+-------+----------------+------------+--------+-----------+---------+--------------+------------------+-------------+------------------+------------+------------+----------------+------------------------+----------------+----------------+------------------------+--------------------+-------------+--------------------+----------+--------------------------+-------------+---------------------------+--------------------+-----------------------+-----------+--------+---------------+-------------------------+-------------+-------+----------------+--------------------+----------+-------------+
|   true|      HD TV|  PLAY|      START|        BX1|   EVENT|                   UID50075|  UID50075|                     4.81.1|             PXM-SW-3.80.0|               02BRA0|     BEPBXL1|F83B1D937298|   LINEAR|        278.0|1638173907037|   VIEWING|       278.0|             true|             4727286|     v7|      CHANNEL-UP|192.168.1.64|43421543|Archiurbain|   SINGLE|            FR|                FR|           FR|               OFF|F83B1D937298|F83B1D937298|            NONE|               Android 9|        Magazine|       1080000.0|                43421543|        Architecture|  Archiurbain|                true|        SD|              202111203342|1638173520000|                      FIXED|rtp://239.255.1.1...|                   IGMP|  MULTICAST|  LINEAR|              R|                     true|     387031.0|VIEWING|             0.0|F83B1D937298_1638...|2021-11-29|   2021-11-29|
|   true|      HD TV|  PLAY|      START|        BX1|   EVENT|                   UID50075|  UID50075|                     4.81.1|             PXM-SW-3.80.0|               02BRA0|     BEPBXL1|F83B1D937298|   LINEAR|        278.0|1638173907037|   VIEWING|       278.0|             true|             4727286|     v7|      CHANNEL-UP|192.168.1.64|43421543|Archiurbain|   SINGLE|            FR|                FR|           FR|               OFF|F83B1D937298|F83B1D937298|            NONE|               Android 9|        Magazine|       1080000.0|                43421543|        Architecture|  Archiurbain|                true|        SD|              202111203342|1638173520000|                      FIXED|rtp://239.255.1.1...|                   IGMP|  MULTICAST|  LINEAR|              R|                     true|     387031.0|VIEWING|             0.0|F83B1D937298_1638...|2021-11-29|   2021-11-29|
|   true|      HD TV|  PLAY|      START|        BX1|   EVENT|                   UID50075|  UID50075|                     4.81.1|             PXM-SW-3.80.0|               02BRA0|     BEPBXL1|F83B1D937298|   LINEAR|        278.0|1638173907037|   VIEWING|       278.0|             true|             4727286|     v7|      CHANNEL-UP|192.168.1.64|43421543|Archiurbain|   SINGLE|            FR|                FR|           FR|               OFF|F83B1D937298|F83B1D937298|            NONE|               Android 9|        Magazine|       1080000.0|                43421543|        Architecture|  Archiurbain|                true|        SD|              202111203342|1638173520000|                      FIXED|rtp://239.255.1.1...|                   IGMP|  MULTICAST|  LINEAR|              R|                     true|     387031.0|VIEWING|             0.0|F83B1D937298_1638...|2021-11-29|   2021-11-29|
|   true|      HD TV|  PLAY|      START|        BX1|   EVENT|                   UID50075|  UID50075|                     4.81.1|             PXM-SW-3.80.0|               02BRA0|     BEPBXL1|F83B1D937298|   LINEAR|        278.0|1638173907037|   VIEWING|       278.0|             true|             4727286|     v7|      CHANNEL-UP|192.168.1.64|43421543|Archiurbain|   SINGLE|            FR|                FR|           FR|               OFF|F83B1D937298|F83B1D937298|            NONE|               Android 9|        Magazine|       1080000.0|                43421543|        Architecture|  Archiurbain|                true|        SD|              202111203342|1638173520000|                      FIXED|rtp://239.255.1.1...|                   IGMP|  MULTICAST|  LINEAR|              R|                     true|     387031.0|VIEWING|             0.0|F83B1D937298_1638...|2021-11-29|   2021-11-29|
+-------+-----------+------+-----------+-----------+--------+---------------------------+----------+---------------------------+--------------------------+---------------------+------------+------------+---------+-------------+-------------+----------+------------+-----------------+--------------------+-------+----------------+------------+--------+-----------+---------+--------------+------------------+-------------+------------------+------------+------------+----------------+------------------------+----------------+----------------+------------------------+--------------------+-------------+--------------------+----------+--------------------------+-------------+---------------------------+--------------------+-----------------------+-----------+--------+---------------+-------------------------+-------------+-------+----------------+--------------------+----------+-------------+
only showing top 4 rows

All columns(hd_conn) that have boolean values are converted to null values despite the fact that everything has StringType as type. In other words, the type of hd_conn is StringType.

After writing to the disk, I get only nulls for these columns. An example of the output is shown below:

{"hd_conn":"null","stb_profile":"HD TV","action":"EXITED","action_type":"STOP","advertisement_reference":"null","app_duration":"4.5691844E7","app_identifier":"F83B1D16ED51_1649207767570","boot_reason":"null","call_letter":"null","category":"EVENT","channel_external_identifier":"null","channel_id":"null","channel_inactivity_threshold":"null","client_software_version":"null","client_software_version_app":"4.92.2","client_software_version_mw":"PXM-SW-3.84.0","company_location_name":"03KAP0","company_site":"BEPVLA2","connected_deflect_state_action":"null","consecutive_playout_indicator":"null","currency_unit":"null","device_identifier":"null","device_name":"F83B1D16ED51","episode_number":"null","episode_title":"null","error_code":"null","error_key":"null","error_message":"null","error_type":"null","event_cat":"Proximus Pickx","event_counter":"50574.0","event_time":"1649253459414","eventcounter":"50574.0","exception_message":"null","exception_name":"null","external_id":"null","first_sign_of_life":"null","genre":"null","group_title":"null","hdmi_connectivity":"null","household_identifier":"1457100","hw_type":"v7","into_standby":"null","into_standby_elapsed_time":"null","invocation_point":"null","ip_address":"192.168.1.65","isolation_state_action":"null","item_id":"be.px.stbtvclient","item_title":"null","item_type":"null","language_audio":"NL","language_interface":"NL","language_shop":"NL","language_subtitles":"OFF","last_activity_timestamp":"null","last_known_heart_beat":"null","mac":"F83B1D16ED51","mac_address":"F83B1D16ED51","oauth2_client_id":"NONE","operation_system_version":"Android 9","operational_mode":"null","out_of_standby":"null","out_standby_elapsed_time":"null","period_of_inactivity":"null","playout_cat":"null","playout_url":"null","post_cycle_state":"null","power_cycle_type":"null","pre_cycle_state":"null","preview":"null","previous_state_uptime":"null","program_category":"null","program_duration":"null","program_reference_number":"null","program_sub_category":"null","program_title":"null","recommendations_flag":"true","remote_connected":"false","rental_cost":"null","resolution":"null"}

I think parquet is trying to do some optimizations that cause this issue.

Any help will be great.

I am using HDP version 2.6.5 & Spark 2.3.