| Proposal | Metadata |
|---|---|
| Author | Kenneth Daily |
| Status | Accepted |
| Created | 28-March-2022 |
Community contributions for the AWS CLI are not frequently reviewed, resulting in stale and incomplete contributions. This document proposes changes to the contribution process to ensure that community contributed feature requests are successfully added to the AWS CLI. The new process aims to regain the trust of contributors who have not had successful contributions and demonstrates our customer obsession through a renewed focus on the community.
The following terms are used throughout this document:
- The community comprises the individuals who use the AWS CLI.
- A user is a member of the community who uses the AWS CLI. They may also interact with the AWS CLI GitHub repository by opening or commenting on issues.
- A contribution is a proposed change to the AWS CLI code base. This can be adding new functionality, fixing a bug, or improving documentation.
- A contributor is a user who makes a contribution to the AWS CLI via a pull request. Contributors are not able to merge pull requests.
- A maintainer is a user who maintains the AWS CLI GitHub repository. They are also a contributor and write core functionality. They triage and moderate GitHub issues, review pull requests, and accept contributions by merging pull requests.
The AWS CLI GitHub repository is available for any individual to report bugs, make feature requests, and ask guidance questions through GitHub issues, as well as open pull requests to propose code changes. There are currently roughly three stages to manage contributions that are common to open source projects: intake, implementation, and review.
GitHub issues are triaged regularly to determine that they are correctly categorized and express a real and relevant problem or request. An answer is provided as soon as possible to acknowledge or resolve the issue. Feature requests are reviewed for general suitability and uniqueness. Users can vote for features via "reactions" on the issue.
Users can open pull requests to propose changes that address bugs or feature requests. There is no requirement to work on an existing feature request or to discuss a contribution prior to opening a pull request. Occassionally guidance is provided to the contributor to improve the proposed change.
Pull requests are selected for review opportunistically when the maintainers have decided that a change or feature should be incorporated. When a feature is selected for review it is added to an internal queue. This queue manages the prioritization of these features but is not visible to the community.
Pull requests are subjected to automated tests and checks which provide preliminary feedback. Once a pull request passes all automated tests and checks, it requires review from a maintainer to proceed toward being merged.
Maintainers may also push changes directly to a contributor's pull request in order to expedite completion or if the contributor is no longer engaged. In some cases, the maintainers will open their own pull request if the proposed change is not acceptable as is or they fail to find an existing pull request.
When the pull request is completed to the maintainers satisfaction, it is merged in for the next release.
The GitHub repository has a contribution guide, standard to many open source projects, to provide guidance on contributing code to the project. The guide describes how to report an issue or feature request and states the minimum requirements and suggestions for a code contribution. It describes how to perform the basic tasks using Git, including retrieving and building the development version.
The current contribution process is applied unevenly and inconsistently. The lack of commitment to the process has led to an accumulation of community feature requests and bug fixes. In turn, the AWS CLI community is dissatisfied and frustrated because their voices, votes, and efforts do not have an impact. Here, we describe some examples of deficiencies in the current process.
86 community pull requests were opened in 2021. Of these, 43 (50%) have been closed: 24 by merging and 19 by closing without merging. The pull requests that were reviewed and merged were chosen opportunistically - they corresponded to work the team needed to complete, like bumping versions, or were low risk, like documentation changes. There is a willingness from the community to make improvements to the AWS CLI through contributions, but their contributions are not reviewed. This inconsistent approach frustrates contributors and results in missed opportunities to support community efforts to improve AWS CLI.
34 pull requests opened in 2021 do not have any response from a maintainer. Of these, 27 pull requests (79%) are still open. The project lacks documentation on the guidelines and requirements for making a contribution as well as what types of contributions would be accepted. Furthermore, the maintainers do not consistently provide acknowledgement, feedback, next steps, or indications of when a review would occur. The result is contributors are unaware if they are making a contribution correctly, what the next steps are, and when it will be reviewed. Contributors are frustrated that their efforts are not being acknowledged and do not know what to do to move forward.
There are 14 pull requests opened in 2021 that had a comment from a maintainer but remain open. These comments include feedback that the contributor responded to and was not followed up on by a maintainer. In some cases, approval was given for a contribution but never merged. The maintainers do not consistently check on the status of existing pull requests and the contribution process stalls. The loss of momentum of the contribution process erodes trust with individual contributors, who feel ignored, and the community at large, who see stale contributions.
Based on the problems identified in the current process, the proposed improvements for customer contributions should satisfy the following goals:
- Maintainers should review community contributions regularly.
- Contributors and maintainers should always be able to determine the current state of a contribution and what steps remain to drive the contribution to a resolution.
- Maintainers should resolve in-flight contribution reviews before committing to any new reviews.
Figure 1: Flowchart of contribution guide review process.
We propose to formalize the existing intake, implementation, and review stages to assure they are applied systematically and transparently. In addition, we will augment the current process with new stages to clarify what issues are available for contribution and what stage they are in. Figure 1 demonstrates how a request and contribution will proceed through the improved process.
We will also publish an improved contributing guide to document the review process, acceptance criteria, and expectations for communication. We will use existing issue labeling in conjunction with a GitHub Project Board to provide transparent status to the community to indicate what is available for contribution and to track the progress of a contribution.
The following sections detail the proposed improvements to the existing process.
A contribution is initiated by opening a GitHub issue. The GitHub issue captures information necessary to make the contribution, such as:
- A description of the requested change.
- The number of users in the community that request the change, measured through GitHub 👍 reactions.
- Design discussions, including use cases, edge cases, and alternative solutions.
Maintainers will regularly review open GitHub issues and label the ones that are ready for community contribution. The decision to label an issue as ready for community contribution is ultimately up to the discretion of the maintainers. Depending on the issue type (bug or feature request) and area (code or documentation), the maintainers will use a set of criteria to decide if the issue is ready for contribution. This may include:
- The change is feasible for a community member to contribute. See the appendix for details.
- Feature requests should have at least 10 👍 reactions. See the rationale for further discussion.
- The maintainers agree with the implementation plan, including any design discussions (e.g., public interface design, edge cases, backwards compatibility concerns).
If the criteria are not satisfied, the maintainers should explicitly comment what criteria are not satsified and how the community can satisfy them. All discussions must be resolved prior to making a contribution so that feedback during the review process focuses on the implementation of the agreed upon design.
Issues that do not receive enough 👍 reactions and do not have any activty for a year are considered stale. Without further interaction, they will be closed.
A GitHub issue enters the contribution ready stage when a maintainer publicly labels the issue ready for contribution. This stage indicates any user is welcome to initiate a contribution via a GitHub pull request. All contribution ready issues are listed publicly for the community to discover.
A GitHub issue enters the implementation stage once a user submits a GitHub pull request implementing the change. This stage indicates that a contribution has been submitted for the issue, but the maintainers have not committed to a full review of the contribution. The maintainers will respond to let the contributor know to expect a preliminary review to confirm:
- It is implemented in the manner described in the issue.
- It meets the criteria specified in the contribution guide. See the appendix for details.
If the criteria are not satisfied, the maintainers will comment on the issue specifically indicating what work is still required. If the contributor has questions, they should refer to the contributing guide documentation or ask the maintainers to clarify. Once the contributor makes the required changes, the maintainers will follow up.
If the contributor is no longer actively engaged by responding to comments from maintainers or making code changes for more than 30 business days, the contribution is stale. If not, the pull request will be closed and the corresponding issue will be marked as ready for contribution.
Contributions made without a GitHub issue will not be prioritized to move to the next stage. Pull requests that are not linked to an issue and are more than six months old will be considered stale and closed. See the rationale for further discussion.
Once the preliminary review criteria from the implementation stage have been met, the tracking GitHub issue is labeled as ready for review and added to a queue for maintainer review. The maintainers will select issues for full review from this queue. The issues will be prioritized for review publicly based on the following criteria in descending order of importance:
- Bugs, documentation changes, and feature requests will be prioritized in that order. See the rationale for further discussion.
- Older contributions will be selected before more recent ones.
A maintainer must select the issue from the queue with the current highest priority. The prioritization of issues in the queue is reviewed on a regular cadence and is ultimately decided based on the maintainers' discretion. See the rationale for further discussion.
Selecting a pull request for review will move the tracking GitHub issue to the under review stage. In this stage, the maintainers will work with the contributor to perform a full review of the contribution with the goal of merging the pull request. Having a pull request selected for review does not guarantee how long it will be until it is merged. The maintainers will review no more than four pull requests at a time. The criteria for merging a pull request may include:
- The change is made in the right place and fits in the existing architecture.
- The change does not increase the maintainability in an unreasonable fashion.
- There is sufficient documentation, both in the code and for end users.
- The code is readable and follows best practices and conventions.
If a pull request is not ready to merge, the maintainers will comment on the pull request with their feedback and questions for the contributor to address. If changes are requested, the contributor should make the changes within 15 business days. If the contributor does not respond within 15 days, the pull request will be marked as stale and the contributor will have 5 days to respond. If there is no response, the maintainers will complete the pull request.
Once a pull request is completed to the satisfaction of the maintainers, it will be approved and merged. The corresponding tracking issue will be closed closed and labeled to acknowledge it as a community contribution.
Figure 2: Example GitHub project for tracking contribution status.
The contribution process will be implemented with a publicly visible GitHub project Kanban board. This project board will track GitHub issues from the contribution-ready stage through completion by merging a pull request. Each lane of the project board maps to a stage of the contribution process. New and existing GitHub labels and GitHub issue events will be used to drive the movement of the issue between the lanes of the GitHub project, which map to the stages of the contribution process. GitHub issue labels will be used to move an issue to the next lane.
The contribution-ready lane is the entry point for users interested in making a contribution. The ready for review lane is the prioritization queue, and issues will be ordered in decreasing priority from top to bottom.
Figure 2 demonstrates an example project with contributions in various stages of completion.
As of 2021-12-15, there are 188 pull requests and 436 issues in the AWS CLI GitHub repository. 43 pull requests and 87 issues have not been updated in more than a year. 95 issues have ten or more upvotes. Many of these issues are candidates for the ready to contribute or ready for review stages. We can use them to backfill the proposed contribution process, providing the team with pull requests to review and issues for the community to contribute.
We propose to manage existing pull requests in two phases. First, we will determine if there is a corresponding tracking issue for each pull request and link them together. If no tracking issue exists, we will create one and link it to the pull request. Then, we will apply the new specification to all open issues and identify the current stage. We will prioritize the review of issues with a linked pull request. After review, all pull requests will follow the process and timelines defined in the specification.
We do not want to discourage users from contributing or proposing changes, but we need to acknowledge that the maintainers have limited bandwidth to give feedback on implementation or design and review pull requests. The maintainers have work that comes from outside of the GitHub repository that needs to be prioritized along with user contributions. Controlling the volume of pull request reviews and prioritizing completing the in flight reviews over new reviews will result in a shorter time to merge and release a contributor's change for a feature requested by the community. It will also reduce the amount of context switching for the maintainers, allowing them to focus on giving high quality feedback to users. As we use this framework to process existing feature requests, we can revisit what the in flight limit should be.
Having a defined process for intake removes any ambiguity on how to initiate a contribution by ensuring that all potential contributions start out as a GitHub issue. We feel that having a more general discussion about the problem and solutions will help reduce the overall effort into code review and provide contributors with enough detail to make a sucessful contribution.
This requirement also solidifies the position that all pull requests that are currently open are something we'd like to pull in. It removes any ambiguity for a contributor about the status of the request.
Pull requests are often made for problems that only affect the contributor or a small portion of the user base. An issue provides a mechanism to gather quantitative feedback in the form of "upvotes" through GitHub reactions to estimate the impact of the issue on the community.
We intend for issues to be a way to improve contributor confidence in both their contributions and the overall process. Draft pull requests can be used to demonstrate a potential implementation and get community feedback or interest and may be converted to pull requests for review at a later time. If a user does open a pull request without an issue, we will use the opportunity to educate them about our process, and when necessary assist them in opening an issue.
We add issues to the queue in a way so that those with higher priority can be resolved quickly and not blocked by larger proposed changes. However, there are scenarios where a new issue may be added to the queue ahead of issues already there. One example considers if the issues currently prioritized for review are all large and complex. In this case, they risk blocking smaller and easier changes. We may pull up the smaller issues to keep the queue moving. In another case, a new feature may increase in urgency due to other API changes. Since the main responsibility of the AWS CLI is to facilitate access to the AWS API, we would re-prioritize a feature request to address that change.
Feature requests carry a risk of adding new technical debt and increased maintenance cost and should have a higher bar to be included.
The request should have user impact beyond the individual who proposed it. This is difficult to measure objectively, but one proxy for impact can be the number of upvote reactions on the issue. This is intended to prevent narrowly scoped changes that only affect a single user. It also helps to limit the number of in flight reviews that the maintainers need to be engaged on, as there is not an unlimited amount of bandwidth to devote to this task.
At present, there are more than 95 open issues with at least 10 upvotes. We will use the current upvote counts to review the backlog and identify issues that we feel would be suited for community contributions. This process can then inform us how to handle new issues that might be candidates for community contribution.
As part of the contribution guide, we will provide specific guidance on how to upvote a feature so we can more accurately estimate user impact.
No. We do not want to require that users who want to contribute pre-register their interest. Multiple contributions can result in a better implementation. However, we should encourage users to comment on the issue that they are working on it. This can be an opportunity for community collaboration instead of duplicated effort.
The AWS CLI is a tool that is widely used in critical, production environments. Reliability is essential and as such the priority is to address known bugs. Feature requests need to be carefully assessed before taking on new potential technical debt. Adding new features before known bugs are addressed would reduce the reliability of the AWS CLI. Once a bug has been triaged and confirmed, it should be made available for community contribution immediately without any upvote requirements. High priority or high impact bugs will be handled by the maintainers immediately without waiting for community contributions.
Most documentation changes are low risk and should move quickly through the review process. Documentation changes also have a high reward potential, measured by the number of users who would read the documentation to solve their problem. In addition, there are also more maintainers available for validation and review of documentation changes.
The contribution process requires significant effort from both contributors and maintainers. The process encourages high quality, desirable changes make it to the review phase, and much of the work to vet the idea and an accompanying pull request has already been completed. At this point, there is enough evidence and effort that a proposed change should be accepted. In addition, the changes are fresh in the maintainer's minds and will be easier to push to completion than waiting for another review cycle. If the original contributor is unable to finish the work, the maintainers should finish it so that the whole community benefits.
Currently, all open issues in the AWS CLI GitHub repository are visible to the community on the issues tab. We use GitHub issue labels to group related issues together, The labels can be used to search and filter issues. However, there is no way to for users to see the all of the stages of the contribution process or how issues transition through it. A user would need to use predefined searches to see which issues are in each stage and could only look at one stage at a time. A Kanban board is a natural tool to display this information, and a GitHub project board implements this using GitHub issues. Lanes in the Kanban board map naturally to the stages of the contribution process.
There are parts of the codebase that are not suitable for community contributions. They may pose a security risk, be part of the code that is dynamically generated, may require interaction with internal teams to implement, or may require knowledge of the codebase that is not well documented. These include (but are not limited to):
- Changes to build processes. Some of our build processes are available through GitHub, but some are not. We need to carefully test any potential changes to be sure that they are compatible with all of our systems.
- GitHub Actions. Changes to our actions need to be made and reviewed with extra scrutiny for security purposes.
- Changes to models, including waiters, paginators, and resources. These requests are cross-SDK and managed by service teams so that all SDKs can use them with uniform behavior.
- Changes to configuration or credential files or processes. These changes can also affect the behavior in other SDKs and must thus be made in coordination with internal teams.
The maintainers will use a set of criteria to move a pull request from the implementation to the ready for review stage, which may include (but are not limited to):
- The pull request must include passing tests.
- The pull request must pass all existing tests.
- The code must work on all supported operating systems.
- The code must pass all linting and style checks.
- The code must be documented, which may include inline documentation and user documentation.
There are times when a change warrants an even more intentional and structured process before introducing a change. For example, we publicly posted a proposal for a source distribution for the AWS CLI v2. The proposal was open for comments and feedback that the approach would solve the problems that exist. The proposed change was a major deviation from how users currently install the AWS CLI, and we felt that including users in the conversation for this change would result in better decisions.
In this situation, we would use the same format as this document for a proposal that would be opened for public comment and review. The document would be incorporated into the codebase as an accepted proposal as a matter of record.