Skip to content

Issue with the Beam example documentation #5

@akramarev

Description

@akramarev

Hi aws-samples maintainers & community, thank you for the examples, they help a lot.

I found a small issue in one of the examples, particularly amazon-kinesis-data-analytics-java-examples/Beam. Actually, the code is fine, but correspondent documentation is not.

Page https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-beam.html#examples-beam-resources recommends to:

  • use Apache Flink 1.11 cluster version
  • and build fat jar using this command: mvn package -Dflink.version=1.11.1 -Dflink.version.minor=1.8

If you build the example, upload .jar file and try to run your Kinesis Data Analytics app you will get this error:

Type 'org/apache/flink/streaming/api/transformations/TwoInputTransformation' (current frame, stack[4]) is not assignable to 'org/apache/flink/streaming/api/transformations/StreamTransformation'

Flink documentation (https://beam.apache.org/documentation/runners/flink/#flink-version-compatibility) states though:

The Flink cluster version has to match the minor version used by the FlinkRunner. The minor version is the first two numbers in the version string, e.g. in 1.8.0 the minor version is 1.8.

Thus flink.version.minor must be 1.11, and not 1.8. Full set of POM properties that work for me with the latest version in master branch:

<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.11</java.version>
        <scala.binary.version>2.12</scala.binary.version>
        <kda-runtime.version>1.2.0</kda-runtime.version>
        <beam.version>2.28.0</beam.version>
        <jackson.version>2.10.2</jackson.version>
        <flink.version>1.11.3</flink.version>
        <flink.version.minor>1.11</flink.version.minor>
    </properties>

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions