38 Pull requests opened by 29 people

• [SPARK-49110][SQL] Fix reading metadata columns for tables with CHAR columns

  #47604 opened Aug 5, 2024

• [WIP][SPARK-48344][SQL] SQL Scripting execution (including Spark Connect)

  #47609 opened Aug 5, 2024

• [SPARK-49113] Dont assert on translation bugs - silently swallow the exception

  #47611 opened Aug 5, 2024

• [SPARK-48824] Add Identity Column SQL syntax

  #47614 opened Aug 5, 2024

• Fixed comma splice in cluster-overview.md

  #47615 opened Aug 5, 2024

• [SPARK-48778][SQL][TESTS] Improve collation support testing - add unit tests for string expressions

  #47620 opened Aug 5, 2024

• [SPARK-49119][SQL] Fix the inconsistency of syntax `show columns` between v1 and v2

  #47628 opened Aug 6, 2024

• [SPARK-49122][Connect][SQL] Add `addArtifact` API to the Spark SQL Core

  #47631 opened Aug 6, 2024

• [SPARK-49086][CONNECT] Move ML function registration to SparkSessionExtensions

  #47637 opened Aug 6, 2024

• [SPARK-49131][SS] TransformWithState should properly set implicit grouping keys even with lazy iterators

  #47641 opened Aug 6, 2024

• [SPARK-49136][SQL] Eagerly initialize DriverManager on Driver

  #47646 opened Aug 7, 2024

• [SPARK-49140][SQL] Widening type promotion from int/long to decimal and support custom decimal logic type for int in `AvroDeserializer`

  #47651 opened Aug 7, 2024

• [SPARK-48843][FOLLOWUP] Adding a PySpark test.

  #47653 opened Aug 7, 2024

• [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework

  #47656 opened Aug 7, 2024

• [SPARK-49152][SQL] V2SessionCatalog should use V2Command

  #47660 opened Aug 8, 2024

• [SPARK-49155][SQL][SS] Use more appropriate parameter type to construct `GenericArrayData`

  #47662 opened Aug 8, 2024

• [SPARK-49178][CORE][SQL][CONNECT][ML][R] Optimize performance of `Row#getSeq` to match the performance when using Spark 3.5 with Scala 2.12.

  #47664 opened Aug 8, 2024

• [SPARK-49162][SQL] Push down date_trunc function

  #47666 opened Aug 8, 2024

• [SPARK-49164][SQL] Fix not NullSafeEqual in predicate of SQL query in JDBC Relation

  #47669 opened Aug 8, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in StringReplace

  #47670 opened Aug 8, 2024

• [WIP][SPARK-48355][SQL] Support for CASE statement

  #47672 opened Aug 8, 2024

• [SPARK-49166][SQL] Support OFFSET in correlated subquery

  #47673 opened Aug 8, 2024

• [SPARK-49173][CONNECT] Change Spark Connect shell prompt from `@` to `scala>`

  #47679 opened Aug 9, 2024

• [SPARK-49175][PYTHON][MIRROR] Use UnsupportedType to check DataType validation for Arrow UDF

  #47680 opened Aug 9, 2024

• [SPARK-49179][SQL] Fix v2 multi bucketed inner joins throw AssertionError

  #47683 opened Aug 9, 2024

• [SPARK-49183][SQL] V2SessionCatalog.createTable should respect PROP_IS_MANAGED_LOCATION

  #47684 opened Aug 9, 2024

• [SPARK-49185][PS][PYTHON][CONNECT] Reimplement `kde` plot with Spark SQL

  #47685 opened Aug 9, 2024

• [SQL] Add more tests for toYearMonthIntervalString(…

  #47687 opened Aug 9, 2024

• [WIP][SPARK-49022] Use Column Node API in Column

  #47688 opened Aug 9, 2024

• [WIP] Implement Levenshtein distance for utf8_lcase collation

  #47689 opened Aug 9, 2024

• [SPARK-49114][SS][FOLLOWUP] Sub-categorize cannot load state store errors

  #47690 opened Aug 9, 2024

• [SPARK-49188][SQL] Internal error on concat_ws called on array of arrays of string

  #47691 opened Aug 9, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in SubstringIndex

  #47693 opened Aug 9, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in StringLocate

  #47694 opened Aug 9, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in StringInstr

  #47695 opened Aug 9, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in StringTrim

  #47696 opened Aug 9, 2024

• [WIP][SQL] Handle invalid UTF-8 strings in StringTranslate

  #47697 opened Aug 9, 2024

• [SQL][HotFix] Clear unused error methods in QueryCompilationErrors

  #47701 opened Aug 11, 2024

34 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-49050][SS] Integrate TransformWithState operator with Virtual Column Families

  #47524 commented on Aug 10, 2024 • 51 new comments

• [SPARK-47430][SQL] Rework group by map type to fix bind reference exception

  #47545 commented on Aug 10, 2024 • 13 new comments

• [SPARK-48347][SQL] Support for WHILE statement

  #47462 commented on Aug 8, 2024 • 11 new comments

• [SPARK-42307][SQL] Assign name for error _LEGACY_ERROR_TEMP_2232

  #47354 commented on Aug 11, 2024 • 10 new comments

• [SPARK-49021][SS] Add support for reading transformWithState value state variables with state data source reader

  #47574 commented on Aug 9, 2024 • 9 new comments

• [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements

  #47428 commented on Aug 7, 2024 • 9 new comments

• [SPARK-48781][SQL] Add Catalog APIs for loading stored procedures

  #47190 commented on Aug 9, 2024 • 8 new comments

• [SPARK-49043][SQL] Fix interpreted codepath group by on map containing collated strings

  #47521 commented on Aug 5, 2024 • 8 new comments

• [SPARK-49017][SQL] Insert statement fails when multiple parameters are being used

  #47501 commented on Aug 9, 2024 • 7 new comments

• [SPARK-48700] [SQL] Mode expression for complex types (all collations)

  #47154 commented on Aug 10, 2024 • 6 new comments

• [SPARK-48821][SQL] Support Update in DataFrameWriterV2

  #47233 commented on Aug 8, 2024 • 6 new comments

• [SPARK-48755][SS] State V2 base implementation and ValueState support

  #47133 commented on Aug 11, 2024 • 5 new comments

• [SPARK-49073][SQL][DOCS] Improve the doc `sql-ref-syntax-qry-select.md`

  #47552 commented on Aug 8, 2024 • 3 new comments

• [WIP][SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting

  #47423 commented on Aug 9, 2024 • 3 new comments

• [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled`

  #47500 commented on Aug 7, 2024 • 2 new comments

• [SPARK-49016][SQL] Queries from raw CSV files are disallowed when the referenced columns only include the internal corrupt record column

  #47506 commented on Aug 9, 2024 • 1 new comment

• [SPARK-48769][SQL] Support constant folding for ScalaUDF

  #47164 commented on Aug 6, 2024 • 1 new comment

• Enhance the metrics in SparkUI with logical plan stats

  #47534 commented on Aug 6, 2024 • 0 new comments

• [SPARK-49062][SQL] Migrate XML to File Data Source V2

  #47539 commented on Aug 6, 2024 • 0 new comments

• [MINOR] Use LinkedHashSet for ResolveLateralColumnAliasReference to generate stable hash for the plan

  #47571 commented on Aug 7, 2024 • 0 new comments

• [SPARK-49038][SQL] Fix regression in Spark UI SQL operator metrics calculation to filter out invalid accumulator values correctly

  #47516 commented on Aug 7, 2024 • 0 new comments

• [SPARK-49095][SQL] Update `DecimalType`and `decimal fields` compatible logic of `Avro` data source to avoid loss of decimal precision

  #47584 commented on Aug 8, 2024 • 0 new comments

• [DO-NOT-MERGE] Verifying Run documentation build

  #47587 commented on Aug 5, 2024 • 0 new comments

• [SPARK-49006] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files

  #47490 commented on Aug 9, 2024 • 0 new comments

• [SPARK-48960][CONNECT] Makes spark-submit works with Spark connect

  #47434 commented on Aug 11, 2024 • 0 new comments

• [SPARK-48939][AVRO] Support reading Avro with recursive schema reference

  #47425 commented on Aug 9, 2024 • 0 new comments

• [SPARK-48881][SQL] Some dynamic partitions can be compensated to specific partition values

  #47418 commented on Aug 7, 2024 • 0 new comments

• [SPARK-48906][SQL] Introduce `SHOW COLLATIONS LIKE ...` syntax to show all collations

  #47364 commented on Aug 5, 2024 • 0 new comments

• [SPARK-48882][SS] Assign names to streaming output mode related error classes

  #47327 commented on Aug 9, 2024 • 0 new comments

• [SPARK-44167][SQL] Support loading stored procedures in catalogs

  #47183 commented on Aug 5, 2024 • 0 new comments

• [SPARK-48835] Introduce versoning to jdbc connectors

  #47181 commented on Aug 7, 2024 • 0 new comments

• [SPARK-48590][BUILD] Upgrade netty to `4.1.112.Final`

  #46945 commented on Aug 5, 2024 • 0 new comments

• [SPARK-47257][SQL] Assign names to error classes _LEGACY_ERROR_TEMP_105[3-4] and _LEGACY_ERROR_TEMP_1113

  #46731 commented on Aug 9, 2024 • 0 new comments

• [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes

  #46368 commented on Aug 8, 2024 • 0 new comments