Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/StardustDocs/d.tree
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
<toc-element topic="schemasInheritance.md"/>
<toc-element topic="Data-Schemas-In-Kotlin-Notebook.md"/>
<toc-element topic="schemasImportOpenApiJupyter.md"/>
<toc-element topic="Data-Schemas-And-Extension-Properties-Troubleshooting.md"/>
<toc-element topic="Gradle-Plugin.md">
<toc-element topic="schemasGradle.md"/>
<toc-element topic="schemasImportSqlGradle.md"/>
Expand Down
12 changes: 12 additions & 0 deletions docs/StardustDocs/topics/extensionPropertiesApi.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,3 +291,15 @@ However, you can work around this by casting back to the original schema:
df.add("name") { "branchName" }
.filter { it.cast<BranchData>().profit > 0 }
```

## Troubleshooting

Sometimes you can get an exception with a message containing

```plain text
..exception in generated DataFrame extension property..
```

This may be caused by incompatible schema usage or incorrectly defined column types.

See [](Data-Schemas-And-Extension-Properties-Troubleshooting.md) for more information.
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Data Schemas and Extension Properties Troubleshooting

Sometimes you can get an exception with a message containing

```plain text
..exception in generated DataFrame extension property..
```

This means there is a runtime error while accessing a [`DataFrame` extension property](extensionPropertiesApi.md),
generated by the [Compiler Plugin](Compiler-Plugin.md) or in the [Kotlin Notebook](SetupKotlinNotebook.md).

Such errors are caused by generating extension properties for data schemas that are not compatible with the
[`DataFrame`](DataFrame.md), [`DataRow`](DataRow.md), etc.
In most cases, the schema contains columns of the wrong names or types.

```kotlin
@DataSchema
interface Schema {
val age: String
}
```

```kotlin
val df = dataFrameOf("age" to columnOf(17, 32, 26)).cast<Schema>()

// Compiles correctly but fails on runtime
df.filter { age > 20 }
```

## Possible reasons

### Incompatible manually defined data schema

If you define initad data schema manually,
make sure your data schema is compatible with the [`DataFrame`](DataFrame.md).

* Use [special methods](DataSchemaGenerationMethods.md) for generating a data schema code
instead of defining data schema manually.
* Use [`.cast<Schema>()`](cast.md) with `verify=true` for verifying the `Schema` compatibility.

### Incorrect column types after `DataFrame` creation

Sometimes the runtime schema is wrong itself,
because column types differ from the actual column value types.
This can happen when reading a DataFrame from files or databases.

> Such cases are most probably bugs! Please report them on [GitHub Issues](https://github.com/Kotlin/dataframe/issues).
{style="warning"}

Possible workarounds:

* Specify the correct type using [`.replace {}`](replace.md) and `ValueColumn.changeType()`:

```kotlin
df.replace { wrongTypeCol }.with { it.asValueColumn().changeType(typeOf<ActualType>) }
```

* Use [`.inferType { columns }`](inferType.md) to infer the correct types
for the selected columns from the actual values.
**It can take a long time and use up a lot of resources for large dataframes!**

#### Problems with type affinity in SQLite

Because of [SQLite type affinity](https://sqlite.org/datatype3.html),
the column typed defined by JDBC may differ from the actual values in the column.
This problem often occurs when reading data from an SQLite database with column of custom types.

You can provide types for such columns manually:

```Kotlin
import org.jetbrains.kotlinx.dataframe.io.db.Sqlite
import kotlin.reflect.typeOf

val sqliteCustom = Sqlite.withCustomTypes(
mapOf(
"LONGVARCHAR" to typeOf<String>(),
"LONGINT" to typeOf<Long>()
)
)
val df = DataFrame.readSqlTable(
connection, "table_name", dbType = sqliteCustom
)
```
Loading