0
$\begingroup$

My input is "20220212" and I should get output like "2022-02-12T00:00:00+00:00"

I have written the following code in PySpark:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, from_utc_timestamp, from_unixtime, unix_timestamp, date_format, coalesce
spark = SparkSession.builder.appName("TimestampConversion").getOrCreate()
data = [("2022-02-12",), ("2022/04/11",), ("2022.03.10",)]
columns = ["date_string"]
df = spark.createDataFrame(data, columns)
df_result = df.withColumn(
     "formatted_timestamp",
     date_format(
         coalesce(
             from_utc_timestamp(
                 from_unixtime(unix_timestamp(col("date_string"), "yyyy-MM-dd")), "UTC"
             ),
             from_utc_timestamp(
                 from_unixtime(unix_timestamp(col("date_string"), "yyyy/MM/dd")), "UTC"
             ),
            from_utc_timestamp(
                 from_unixtime(unix_timestamp(col("date_string"), "yyyy.MM.dd")), "UTC"
             )
         ),
         "yyyy-MM-dd'T'HH:mm:ssXXX"
     )
)
 df_result.show(truncate=False)

What i need is I'm using 3 built-in-function, but I don't want this much built-in functions. Is it possible to write the same code with 2 built-in functions and if yes, how?

$\endgroup$

1 Answer 1

2
$\begingroup$

Yes, you can do it with less built-ins:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_timestamp, date_format, coalesce

spark = SparkSession.builder.appName("TimestampConversion").getOrCreate()

data = [("2022-02-12",), ("2022/04/11",), ("2022.03.10",)]
columns = ["date_string"]
formats = ["yyyy-MM-dd", "yyyy/MM/dd", "yyyy.MM.dd", "yyyyMMdd"]

def to_timestamp_with_formats(col, formats):
    return coalesce(*[to_timestamp(col, fmt) for fmt in formats])

df = spark.createDataFrame(data, columns)
df_result = df.withColumn("formatted_timestamp", to_timestamp_with_formats(col("date_string"), formats))
df_formatted = df_timestamp.withColumn("formatted_timestamp", date_format(col("timestamp"), "yyyy-MM-dd'T'HH:mm:ssXXX"))
df_formatted.show(truncate=False)
$\endgroup$
2
  • $\begingroup$ its showing system local time 2022-02-12T00:00:00+05:30 but i want 2022-02-12T00:00:00+00:00. may i know why its taking default time and how to rectify this...?? $\endgroup$ Nov 21, 2023 at 6:50
  • $\begingroup$ Sure, you can specify a timezone, for example timezone = "America/New_York" and then adjust the return of the function to convert to timestamp like this return coalesce(*[to_timestamp(col, fmt).withTimeZone(timezone) for fmt in formats]) $\endgroup$
    – ABizz
    Nov 22, 2023 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.