diff --git a/docs/StardustDocs/d.tree b/docs/StardustDocs/d.tree index d0b561def0..e22a7c83f2 100644 --- a/docs/StardustDocs/d.tree +++ b/docs/StardustDocs/d.tree @@ -45,6 +45,7 @@ + diff --git a/docs/StardustDocs/topics/extensionPropertiesApi.md b/docs/StardustDocs/topics/extensionPropertiesApi.md index ece739a575..654b869286 100644 --- a/docs/StardustDocs/topics/extensionPropertiesApi.md +++ b/docs/StardustDocs/topics/extensionPropertiesApi.md @@ -291,3 +291,15 @@ However, you can work around this by casting back to the original schema: df.add("name") { "branchName" } .filter { it.cast().profit > 0 } ``` + +## Troubleshooting + +Sometimes you can get an exception with a message containing + +```plain text +..exception in generated DataFrame extension property.. +``` + +This may be caused by incompatible schema usage or incorrectly defined column types. + +See [](Data-Schemas-And-Extension-Properties-Troubleshooting.md) for more information. diff --git a/docs/StardustDocs/topics/schemas/Data-Schemas-And-Extension-Properties-Troubleshooting.md b/docs/StardustDocs/topics/schemas/Data-Schemas-And-Extension-Properties-Troubleshooting.md new file mode 100644 index 0000000000..7dcf3ff10d --- /dev/null +++ b/docs/StardustDocs/topics/schemas/Data-Schemas-And-Extension-Properties-Troubleshooting.md @@ -0,0 +1,83 @@ +# Data Schemas and Extension Properties Troubleshooting + +Sometimes you can get an exception with a message containing + +```plain text +..exception in generated DataFrame extension property.. +``` + +This means there is a runtime error while accessing a [`DataFrame` extension property](extensionPropertiesApi.md), +generated by the [Compiler Plugin](Compiler-Plugin.md) or in the [Kotlin Notebook](SetupKotlinNotebook.md). + +Such errors are caused by generating extension properties for data schemas that are not compatible with the +[`DataFrame`](DataFrame.md), [`DataRow`](DataRow.md), etc. +In most cases, the schema contains columns of the wrong names or types. + +```kotlin +@DataSchema +interface Schema { + val age: String +} +``` + +```kotlin +val df = dataFrameOf("age" to columnOf(17, 32, 26)).cast() + +// Compiles correctly but fails on runtime +df.filter { age > 20 } +``` + +## Possible reasons + +### Incompatible manually defined data schema + +If you define initad data schema manually, +make sure your data schema is compatible with the [`DataFrame`](DataFrame.md). + +* Use [special methods](DataSchemaGenerationMethods.md) for generating a data schema code +instead of defining data schema manually. +* Use [`.cast()`](cast.md) with `verify=true` for verifying the `Schema` compatibility. + +### Incorrect column types after `DataFrame` creation + +Sometimes the runtime schema is wrong itself, +because column types differ from the actual column value types. +This can happen when reading a DataFrame from files or databases. + +> Such cases are most probably bugs! Please report them on [GitHub Issues](https://github.com/Kotlin/dataframe/issues). +{style="warning"} + +Possible workarounds: + +* Specify the correct type using [`.replace {}`](replace.md) and `ValueColumn.changeType()`: + +```kotlin +df.replace { wrongTypeCol }.with { it.asValueColumn().changeType(typeOf) } +``` + +* Use [`.inferType { columns }`](inferType.md) to infer the correct types +for the selected columns from the actual values. +**It can take a long time and use up a lot of resources for large dataframes!** + +#### Problems with type affinity in SQLite + +Because of [SQLite type affinity](https://sqlite.org/datatype3.html), +the column typed defined by JDBC may differ from the actual values in the column. +This problem often occurs when reading data from an SQLite database with column of custom types. + +You can provide types for such columns manually: + +```Kotlin +import org.jetbrains.kotlinx.dataframe.io.db.Sqlite +import kotlin.reflect.typeOf + +val sqliteCustom = Sqlite.withCustomTypes( + mapOf( + "LONGVARCHAR" to typeOf(), + "LONGINT" to typeOf() + ) +) +val df = DataFrame.readSqlTable( + connection, "table_name", dbType = sqliteCustom +) +```