Projects clone should happen before other components are started

[l0rd]

What problem do we want to address?

The ones described here

How are we going to address it?

• When a DevWorkspace CR has one or more projects, those should be cloned before any other component is deployed
• Kubernetes secrets containing Git credentials and SSH key pairs should be used to clone password protected repositories

[l0rd]

We may want to use https://github.com/kubernetes/git-sync

[sleshchenko]

We should git clone all projects before a DevWorkspace is started.

Depending on how big the repository[ies] are, it may influences workspace start-up time essentially. Some examples I got locally:

Eclipse Che: **1m13s** with 1.76MiB/s at the last moment

Che Plugin Broker: **6s** with 900KiB/s at the last moment

Kubernetes Go Client: **18s** with 1.41MiB/s at the last moment

ODO: **1m36s** with 1.48MiB/s at the last moment

ODO master branch only: **1m25s** with 1.66MiB/s at the last moment

On RHPDS OpenShift 4.6 cluster it's better but still seems to move us back from faster workspace start-up goal:

ODO: **20s** with 21.27MiB/s at the last moment

[sleshchenko]

It seems to be related to

Start an editor only workspace and asynchronously update the CR and "recreate" the workspace pod

item from #209

[amisevsk]

I'm not sure this is a DevWorkspace Operator issue. Git cloning before the workspace is started can be implemented by whoever is creating a DevWorkspace by either applying an appropriate init container, or by using something like https://github.com/amisevsk/workspace-bootstrap. In general, I don't think it's on the DevWorkspace operator to think about whether the PVC for a workspace contains the appropriate files, as this is potentially more error prone due to factors outside of our control.

[l0rd]

I'm not sure this is a DevWorkspace Operator issue. Git cloning before the workspace is started can be implemented by whoever is creating a DevWorkspace by either applying an appropriate init container, or by using something like amisevsk/workspace-bootstrap. In general, I don't think it's on the DevWorkspace operator to think about whether the PVC for a workspace contains the appropriate files, as this is potentially more error prone due to factors outside of our control.

That means that the operator would only manage/reconcile the components of a DevWorkspace. If projects, commands or events in the CR are updated something else should take care of "reconciliation". That doesn't sound right.

[amisevsk]

That means that the operator would only manage/reconcile the components of a DevWorkspace. If projects, commands or events in the CR are updated something else should take care of "reconciliation". That doesn't sound right.

What do you mean? The DevWorkspaceOperator will reconcile whenever you put in your DevWorkspace, but reconciling doesn't extend to populating PVCs with data -- that's out of scope. If you apply a new preStart container to your DevWorkspace, the operator will already handle it.

[l0rd]

Yes sorry, events are not ignored, but commands or projects are right? I am providing more details in #273.

[amisevsk]

Currently, we do process projects and commands, in the sense that we construct the runtime-annotations configmap and mount it into the che REST API server emulator. In general, we've planned to move away from that, and having consumers of commands/projects directly process that info by reading the DevWorkspace (whether from the cluster or directly mounted into the container).

In general, the barrier I draw mentally for whether something should be explicitly handled by the DevWorkspace Operator is basically Kubernetes objects vs. interactive elements. As a result:

• The DevWorkspaceOperator processes commands as necessary, in order to correctly apply initContainers (prestart event + apply commands)
• The DevWorkspaceOperator will automatically make any changes to k8s objects where required by an updated DevWorkspace (you can add containers, endpoints, etc. and the involved k8s objects will be created)

The second point means we do handle updated commands, etc., since we currently propagate those to a configmap, and I expect we will continue to do some work in propagating that information downwards. I also had a PR (#268) open to automatically apply postStart events to a devworkspace, but Serhii and I decided the concept needed more discussion.

For projects, I think our implementation is well-designed enough that it can be enabled or disabled based on how a devfile gets turned into a DevWorkspace. For example, the "plugin" below would be sufficient to automatically clone projects on-demand:

kind: DevWorkspaceTemplate
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: project-cloner
spec:
  components:
    - name: project-cloner
      container:
        image: '<an-appropriate-image>'
        mountSources: true
        args:
          - '/bin/bash'
          - '-c'
          - 'read devfile projects, clone them if necessary'
  commands:
    - id: clone-projects
      apply:
        component: project-cloner
  events:
    preStart:
      - clone-projects

This could be added to any devworkspace by including

spec:
  components:
    - name: cloner
      plugin:
        kubernetes:
          name: project-cloner

in a devworkspace.

The DWO could apply such a component automatically, but that blurs the line between what you write in a devfile and what you get ultimately. We would have to, in the operator, address the edge cases of devfiles without projects, or devfiles that don't want PVCs mounted at all (e.g. web terminal)

[amisevsk]

After disucssing this with Mario briefly, we've come up with two potential approaches to managing projects in a devworkspace. I think both have pros and cons, to the extent that I'm pretty 50-50 between the two.

There are of course other options possible, these are just the two that came to me readily

Option 1: Projects are cloned via a plugin

With this approach, we define a plugin DWT (similar to the one above) that can be added to a DevWorkspace by either the Dashboard, Theia, etc. This plugin would define an init container that reads the DevWorkspace and ensures all projects in the Devfile are cloned.

Pros

• Simple, standalone definition; could be added/disabled directly and potentially handle edge cases more easily (dirty branches, etc.)
• More statically defined; could be updated/modified by simply changing the plugin's definition

Cons

• Harder to manage certain cases; if the projects in a devworkspace CR are updated, the change won't be reflected until the DW restarts. This means e.g. Theia still has to clone repos sometimes

Option 2: DevWorkspace Operator adds init container that syncs projects

Here, the DevWorkspace Operator would automatically add an init container to the workspace deployment that clones the projects from the DevWorkspace. Functionally similar to Option 1 (an init container that clones projects), but the definition lives in the operator -- effectively, this makes devfile projects resolve to initContainers in the pod spec. A devfile attribute could disable automatic cloning for a DevWorkspace if necessary.

Pros

• Moves more devfile support into the DevWorkspace Operator, rather than external components. With Option 1, to fully "resolve" a Devfile you have to include a plugin.
• Would support automatically restarting a workspace to clone projects added to the DW: if e.g. the entrypoint to a container is git clone <myproject>, then updating projects would change the pod spec and cause a restart

Cons

• Somewhat more complex, as definition lives in the operator. Debugging/improving this functionality requires debugging/improving the operator itself.
• No way to customize functionality (the init container you get is defined in the version of DWO)
• It's not strictly necessary to restart a workspace to add a project, but this approach forces it.

[amisevsk]

Another detail for consideration here is how this interplays with async storage. When async storage is used, volumes mounted to the workspace are all ephemeral, and are filled after startup by the async sidecar. This means that git cloning does unnecessary work (except for the very first time its run) for async storage.