#Pixelspark 4.2.8 driver
#Pixelspark 4.2.8 upgrade

: Correlated subqueries with equality predicates can return wrong results.: Nested column pruning fails to extract case-insensitive struct field from array.: Throw fetch failure exception when unable to deserialize broadcasted map statuses.: conv() does not convert negative inputs to unsigned correctly.: Non-nullable aggregates can return NULL in a correlated subquery.: SparkContext - avoid using local lazy val.: Catalyst error on on certain struct operation (Couldn’t find gen_alias).: Fix Interpreted evaluation by using getClass.getMethod instead of getDeclaredMethod.: NewInstance.resolved should not throw malformed class name error.: NewInstance.doGenCode should not throw malformed class name error.: Omit inlining passwords during build process.: lRegex should work with column names and qualifiers which contain newlines.: Do not reregister BlockManager when SparkContext is stopped.

: UnresolvedException when creating temp view twice.

: Avro should read decimal values with the file schema.

: For parquet table, after changing the precision and scale of decimal type in hive, spark reads incorrect value.

: Use available offset range obtained during polling when checking offset validation.

: BytesToBytesMap’s iterator hasNext method should be idempotent.

: PullOutNondeterministic should check and collect deterministic field.

: Duplicate field names causes wrong answers during aggregation.

: Vector reader got incorrect data with binary partition value.

: Partition spec in SHOW TABLE EXTENDED doesn’t respect.

: IllegalArgumentException when applying RemoveRedundantSorts before EnsureRequirements.

: Pyspark application will hang due to non Exception.

: GROUP BY using literal map should not fail.

: Make Literal ArrayBasedMapData string representation disambiguous.

: Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

: Fix bugs for casting data from/to PythonUserDefinedType.

: FileOutputWriter jobs have duplicate JobIDs if launched in same second.

: Bug in optimizer rule EliminateSorts.

: Handling nullability for complex types is broken during resolution of V2 write command.

: Fix grouping sets with having clause can not resolve qualified col name.

: LibSVM format does not propagate Hadoop config from DS options to underlying HDFS file system.

: ORC format does not propagate Hadoop config from DS options to underlying HDFS file system.

: TreeNode.nodeName should not throw malformed class name error.

: percentile_approx() returns incorrect results.

: UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling.

: UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.

: BytesToBytesMap at MAX_CAPACITY exceeds growth threshold.

: Fix DataStreamReaderWriterSuite to check writer options correctly.

: Fix LibSVM data source loading error on file paths with glob metacharacters.

: Rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 streaming sources.

: thread safe endpoints may hang due to fatal error.

: Broadcast block pieces may memory leak.

: Query optimization fails to reuse exchange with DataSourceV2.

: When () function is used with dataframe cache, it returns wrong result.

: HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT.

: spark with user provided hadoop doesn’t work on kubernetes.

#Pixelspark 4.2.8 driver

: Driver and executors use a different service account breaking pull secrets.: Spark overwrites existing value of .* instead of merging value.: CSV infer schema bug infers decimal(9,-1).: Creating parquet table with all the column null throws exception.

#Pixelspark 4.2.8 upgrade

We strongly recommend all 2.4 users to upgrade to this stable release. This release is based on the branch-2.4 maintenance branch of Spark. Spark 2.4.8 is a maintenance release containing stability, correctness, and security fixes.