[CH-191] Support generate exec

[taiyang-li]

What changes were proposed in this pull request?

Support generate exec. close Kyligence/ClickHouse#191

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

test by CH[[194]]

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[Gluten-${ISSUES_ID}] ${detailed message}

See also:

• Other pull requests

[taiyang-li]

Spark plan:

0: jdbc:hive2://localhost:10000/> explain formatted  select explode(array(id, id+1)) as elem, cast(id as string) as spark  from range(3);   


+----------------------------------------------------+
|                        plan                        |
+----------------------------------------------------+
| == Physical Plan ==
CHNativeColumnarToRow (6)
+- * ProjectExecTransformer (4)
   +- * GenerateExecTransformer (3)
      +- RowToCHNativeColumnar (2)
         +- * Range (1)


(1) Range [codegen id : 1]
Output [1]: [id#121L]
Arguments: Range (0, 3, step=1, splits=None)

(2) RowToCHNativeColumnar
Input [1]: [id#121L]

(3) GenerateExecTransformer
Input [1]: [id#121L]
Arguments: explode(array(id#121L, (id#121L + 1))), [id#121L], false, [elem#122L]

(4) ProjectExecTransformer
Input [2]: [id#121L, elem#122L]
Arguments: [elem#122L, cast(id#121L as string) AS spark#115]

(5) WholeStageCodegenTransformer (8)
Input [2]: [elem#122L, spark#115]

(6) CHNativeColumnarToRow
Input [2]: [elem#122L, spark#115]

Return result:

0: jdbc:hive2://localhost:10000/> select explode(array(id, id+1)) as elem, cast(id as string) as spark  from range(3);   
+-------+--------+
| elem  | spark  |
+-------+--------+
| 0     | 0      |
| 1     | 0      |
| 1     | 1      |
| 2     | 1      |
| 2     | 2      |
| 3     | 2      |
+-------+--------+

[taiyang-li]

Native arrayJoin in clickhouse backend is almost 3x faster than Spark explode.

echo "SELECT * FROM generateRandom('a Array(String), b String', 1, 20, 10) limit 100000 FORMAT Parquet"  | clickhouse-client > test_generate_exec.parquet  

> CREATE TEMPORARY VIEW test_generate_exec
USING org.apache.spark.sql.parquet
OPTIONS (
  path  "/data1/liyang/cppproject/gluten/test_generate_exec.parquet"
) ;

> set spark.gluten.sql.columnar.generate=true;
> select count(*) from (select explode(a), b from test_generate_exec);
+-----------+
| count(1)  |
+-----------+
| 500257    |
+-----------+
1 row selected (0.205 seconds)



> set spark.gluten.sql.columnar.generate=false;
> select count(*) from (select explode(a), b from test_generate_exec);  
+-----------+
| count(1)  |
+-----------+
| 500257    |
+-----------+
1 row selected (0.594 seconds)
 

[taiyang-li]

@rui-mo Can you help review this pr, thanks!

[taiyang-li]

@rui-mo It it possible running gluten CI/CD with clickhouse version binding to Kyligence/ClickHouse#194 ?

[rui-mo]

@rui-mo It it possible running gluten CI/CD with clickhouse version binding to Kyligence/ClickHouse#194 ?

Sure. I think it is OK.

[zzcclp]

why remove?

[lviiii]

LGTM

[zzcclp]

LGTM

[EpsilonPrime]

Should this relation be removed in favor of the official Substrait repo's ExpandRel?

[taiyang-li]

GenerateRel and ExpandRel is different. The former is used to represent spark lateral view explode/posexpode, and the latter is used to represent spark grouping sets or with cube