Generalize Classloader isolation for non-task / model builder plugins

[sschuberth]

Describe your use-case which is not covered by existing documentation.

Currently, the docs about Classloader isolation very much focus on plugins that provide tasks. However, in some cases plugins don't provide any tasks but only a ToolingModelBuilder implementation that's supposed to be used programmatically via the Gradle Tooling API. While I guess that Classloader isolation should work quite the same in that case, it would be good to generalize the docs to cover that case as well.

Reference any relevant documentation, other materials or issues/pull requests that can be used for inspiration.

No response

Are you interested in contributing to the documentation?

No response

[martinbonnin]

Hi 👋 Thanks for opening this!
I'm not overly familiar with ToolingModelBuilder but my high level overview is that it lies in "Gradle-land". By that I mean that it shouldn't do too much computation like codegeneration or other things that require dependencies?

In that case, I would most likely expect the ToolingModelBuilder to be executed in the main build script classpath. Do you have an example where the ToolingModelBuilder requires a separate classloader?

[sschuberth]

By that I mean that it shouldn't do too much computation like codegeneration or other things that require dependencies?

That's probably correct, though if you write the actual logic in Kotlin, like I do, larger dependencies like kotlin-stdlib get pulled in.

Do you have an example where the ToolingModelBuilder requires a separate classloader?

You might want to have a look at this code. The plugin also pulls in some Maven dependencies which should not leak into the Gradle build.

I originally wanted to solve this via https://github.com/GradleUp/gr8, but I struggled to get it running (you probably recall my Slack messages; this branch is where I got stuck).

[martinbonnin]

larger dependencies like kotlin-stdlib get pulled in.

Right 👍 That's something I am tempted to give up on and just write my Gradle logic using whatever the embedded Kotlin version is.

I originally wanted to solve this via https://github.com/GradleUp/gr8, but I struggled to get it running (you probably recall my Slack messages; this branch is where I got stuck).

Can definitely relate. Gratatouille is actually born from a desire to remove R8 in the process. It's slow to build, the process is fragile to reflection and other things and makes the stacktraces harder to read/debug... It has served us well for many years but Gratatouille should be a lot more robust.

The plugin also pulls in some Maven dependencies which should not leak into the Gradle build.

Thanks for the link! Indeed, might be worth adding a generic annotation that "just" runs a given function in a separate classloader for those usages. I'll look into it.

[sschuberth]

That's something I am tempted to give up on and just write my Gradle logic using whatever the embedded Kotlin version is.

Does that mean Gratatouille's Classloader isolation cannot be used to decouple the Kotlin version used in the plugin from the Kotlin version that ships with Gradle (if any)?

[martinbonnin]

Does that mean Gratatouille's Classloader isolation cannot be used to decouple the Kotlin version using in the plugin from the Kotlin version that ships with Gradle

Depends what you mean by "in the plugin". A plugin typically has 2 parts:

• The wiring is still required to use a compaible version of Kotlin.
• The tasks can use any version of Kotlin.

Something that could be explored is a transformation that completely strips all the Kotlin metadata from the plugin jar. That would make the compilation work. Then would remain the runtime. For this it's either shadowing or not using any newer stdlib symbol (or gradle/gradle#16345 but doesn't look like this is happening).

[sschuberth]

The wiring is still required to use a compaible version of Kotlin.

Coming back to this, which "wiring" do you refer to? Is it "the glue code and Gradle API that calls the tasks project through reflection" as described here?

Anyway, as described, in my case I have a plugin that does not provide a task, but a ToolingModelBuilder. Any my goal is to be able to use any third-party dependencies and / or Kotlin version / stdlib in the implementation of that builder, and isolate those dependencies from the place there the plugin is applied / from the "host" that "runs" the plugin via the Gradle TAPI.

Is that something you see gratatouille to support, @martinbonnin? Maybe by not only having a @GTask annotation but also a @GModelBuilder one?

[martinbonnin]

Coming back to this, which "wiring" do you refer to? Is it "the glue code and Gradle API that calls the tasks project through reflection" as described here?

Yes, this. Everything that can be called from build.gradle[.kts] files.

Is that something you see gratatouille to support, @martinbonnin? Maybe by not only having a @Gtask annotation but also a @GModelBuilder one?

Could be. Ideally, the feature is developped alongside a real life use case so we can dogfood it along the way. Is your use case OSS by any chance? We could take a look together

[sschuberth]

Is your use case OSS by any chance?

Yes, I had linked it above already, but take a look again at

https://github.com/oss-review-toolkit/ort/blob/main/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt

In essence, it's a plugin that determines Gradle project dependencies and writes them to some custom data model. It's a bit similar to https://github.com/gradle/github-dependency-graph-gradle-plugin.

[martinbonnin]

Thanks! Sorry I missed that link.

From what I undertand, ModelBuilders take a Project instance and generate Serializable models from them. There's no input/outputs like for tasks. No task implementations to write and/or workers.

Maybe what we need is a @GIsolated that acts on "isolated functions". Those functions can accept instances of classes loaded by the platform classloader (that is String, List, File, etc...) and also returns instance of those same type of classes.

Given that isolated function (pseudo code):

@GIsolated
fun computeModel(componentIds: List<String>): Any {
    // use maven resolver here
    val poms = resolvePoms(componentIds)
    // more things
    // ...
    return OrtDependencyTreeModelImpl(TODO())
}

It would generate pseudo code like this to be called from the Gradle plugin:

fun computeModel(project: Project, componentIds: List<String>): Any {
    val configuration = project.configurations.detachedConfiguration()
    // gradle-model contains computeModel above and OrtDependencyTreeModelImpl as well as a transitive dependency to maven-model
    configuration.dependencies.add(project.dependencies.create("com.github.oss-review-toolkit.ort:gradle-model:22.7.0"))
    val classpath = configuration.files

    val classloader = URLClassLoader(classpath.map { it.toURI().toURL() }.toTypedArray())

    val clazz = classloader.loadClass("com.github.oss-review-toolkit.ort.gradle.model.ComputeModelKt")
    val entryPoint = clazz.methods.single {
        it.name == "computeModel"
    }

    return entryPoint.invoke(null, componentIds)
}

It's still some amount of boilerplate but compared to the Task use case, it's considerably less. Compared to doing things manually, you'd have:

1. generation of the bit of code that configures the detached configuration.
2. generation of the reflection code.
3. the whole zip/unzip mechanism so you can generate in gradle-model classes that are going to be compiled in gradle-plugin.

Would that be of any interest?

[sschuberth]

Would that be of any interest?

I definitely think so! It still sounds much nicer than having to deal with Proguard via Gr8 😄