Continuous integration and deployment (CI/CD) update

[davidheryanto]

This pull request addresses #180

In order to improve transparency, simplicity and speed of the CI process, I feel it's easier if we switch to a CI tool that integrates natively with Git repository as opposed to generic workflow manager like Argo. CI build process can then be triggered based on Git events such as push, pull request, and tag creation (vs manually listening to these if we use generic workflow tools).

We can still use prow to handle issue and pull request management

There are many CI tools that integrate well with Git and Github like Travis and CircleCI, but for this proposal I choose Drone because of its simplicity and flexibility.

The file that defines the CI pipeline looks like this: https://github.com/davidheryanto/feast/blob/44e3d6dd827a26799a698c8558e6a32acb3b78be/.drone.yml

The build output can be seen from Drone Dashboard:
(the link below will be visible automatically in Github pull request page / build status badge)
http://drone.dragonin.me/davidheryanto/feast/378

How this improves transparency

• The build status badge that appears in the repo page is clickable and links to the actual build log page
• The build status is also immediately visible in pull request page for contributors to easily figure out why the build is failing. Clicking on the "Details" link will open up the build log page.
  
• The actual commands that are executed in the CI pipeline is easily visible in .drone.yml file at the root of the repository. And contributors can suggest additional build steps.

How this improves simplicity

• The proposed CI pipeline does not include the details of infrastructure provisioning (such as creating Kubernetes cluster, setting up BigQuery or Kafka servers). It focuses on testing Feast components and deployment using a Helm chart.
• Compared to previous workflow in Argo, the Drone pipeline is more linear and makes less references to other steps so it's easier to understand the CI process.
• Drone supports job run condition based on Git events, which makes it simple e.g. to make a Feast release when a tag is created, and just run tests otherwise.

How this improves speed

• The proposed pipeline should generally take about 8 minutes to complete unit tests and integrations tests (vs more than 20 minutes previously)
• Supports running multiple CI build jobs, where the systems under test are independent of each other, i.e. Feast being tested in build no. 1 will not affect Feast being tested in build no. 2. So contributors making a pull request can push new commits without waiting for previous tests to complete.

EXTRA NOTES

• I'm open to using other CI tools like Circle CI but I think Drone does the job quite well and it's simple to use. And the steps defined in .drone.yml are generic enough that it's easy to move between different CI tools (apart from the build Docker and restore cache steps), i.e. we're not tied in to Drone.
• This pull requests remove $FEAST_REPO/testing/ folder which includes prow config. So we may need to add prow config back later.
• This pull requests also update the following parts in the Helm chart (I follow the some behaviour of other Helm charts in deciding what should be / not user configurable in helm values.yaml file):
  • Allow users to define custom service annotations (we do not want to restrict users to create Feast service in a certain manner)
  • Update default service type to ClusterIP to prevent accidental exposing of Feast service to external
  • Allow users to define custom liveness and readiness probes properties
• We can allow pull requests from anyone to initiate the build process (to encourage more contributions). However, we should protect the repository so only project owners can approve changes to CI process. This is to guard against accidental exposure of secrets and credentials. Alternatively, we can avoid running steps that may expose secrets when the build is triggered from external pull request.

[feast-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: davidheryanto

If they are not already assigned, you can assign the PR to them by writing /assign @davidheryanto in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment