Skip to content

feat: adds script for git migration. #10882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
atulep merged 12 commits into from
Apr 20, 2023
Merged

feat: adds script for git migration. #10882

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
61a3bb6
feat: adds script.
Apr 3, 2023
2c1e1e4
feat: fix some bugs.
Apr 7, 2023
fcb0211
feat: add update script.
Apr 7, 2023
39432b1
chore: address pr feedback.
Apr 13, 2023
2096df7
chore: address feedback.
Apr 14, 2023
e336cd3
chore: address feedback and change args.
Apr 18, 2023
a3694e4
chore: address pr feedback.
Apr 19, 2023
57e96dc
chore: add gh.
Apr 19, 2023
c7c0d48
fix: fix details for post-processor.
Apr 19, 2023
a303e0f
chore: conventional commit.
Apr 19, 2023
5c57e45
chore: fix.
Apr 19, 2023
997909f
Merge branch 'main' into monorepo_migration_git_migration_script
Apr 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 28 additions & 18 deletions scripts/split_repo_migration/git-migrate-history.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,35 +13,45 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Sample usage:
#
# git-migrate-history.sh \
# "googleapis/python-speech" \
# "~/temp/google-cloud-python" \
# "" \
# "" \
# ".github/.OwlBot.yaml"
#
# This invocation runs migration script on python-speech, where the source path is root directory,
# nothing is ignored, and .OwlBot.yaml is being kept.

set -ex

[[ "${BASH_SOURCE[0]}" != "${0}" ]] && EXIT=return || EXIT=exit

if [ $# -lt 4 ]
if [ $# -lt 3 ]
then
echo "Usage: $0 <source-repo> <target-repo> <source-path> <target-path> [folders,to,skip] [files,to,keep] [branch-name]"
echo "Usage: $0 <source-repo> <target-repo> <source-path> [folders,to,skip] [files,to,keep] [branch-name]"
$EXIT 1
fi

# source GitHub repository. format: <owner>/<repo>
SOURCE_REPO=$1

# destination GitHub repository. format: <owner>/<repo>
TARGET_REPO=$2
TARGET_REPO=$2 # Path to google-cloud-python
Comment thread
atulep marked this conversation as resolved.

# path in the source repo to copy code from. Defaults to the root directory
SOURCE_PATH=$3

# path in the target repo to put the copied code
TARGET_PATH=$4

# comma-separated list of files/folders to skip
IGNORE_FOLDERS=$5
IGNORE_FOLDERS=$4

# keep these specific files that would otherwise be deleted by IGNORE_FOLDERS
KEEP_FILES=$6
KEEP_FILES=$5

# override the HEAD branch name for the migration PR
BRANCH=$7
BRANCH=$6


Comment thread
atulep marked this conversation as resolved.
if [[ -z "${BRANCH}" ]]
Expand Down Expand Up @@ -102,6 +112,8 @@ then
--tree-filter "${FILTER}"
fi

TARGET_PATH=$(jq -r '.distribution_name' .repo-metadata.json) # -r removes quotes around the name.
Comment thread
atulep marked this conversation as resolved.
Outdated

# reorganize the filtered code into the desired target locations
if [[ ! -z "${TARGET_PATH}" ]]
Comment thread
atulep marked this conversation as resolved.
Outdated
then
Expand All @@ -117,12 +129,10 @@ fi
popd

# merge histories
echo "Cloning target repository: ${SOURCE_REPO}"
git clone "git@github.com:${TARGET_REPO}.git" target-repo
pushd target-repo
pushd $TARGET_REPO

git remote add --fetch migration ../source-repo
git checkout -b "${BRANCH}"
git remote add --fetch migration ${WORKDIR}/source-repo
Comment thread
atulep marked this conversation as resolved.
Outdated
git checkout -B "${BRANCH}"
git merge --allow-unrelated-histories migration/main --no-edit

echo "Success"
Expand All @@ -133,15 +143,15 @@ popd # back to workdir
git clone "git@github.com:${SOURCE_REPO}.git" source-repo-validation # Not ideal to clone again.
rm -rf source-repo-validation/.git # That folder is not needed for validation.

if diff -r target-repo/"${TARGET_PATH}" source-repo-validation; then
DIFF_FILE="${WORKDIR}/diff.txt"
if diff -r "${TARGET_REPO}/${TARGET_PATH}" source-repo-validation > "${DIFF_FILE}" ; then
echo "No diff"
else
diff -r target-repo/"${TARGET_PATH}" source-repo-validation > "diff.txt"
echo "Diff non-empty. See ${WORKDIR}/diff.txt"
echo "Diff non-empty. See ${DIFF_FILE}"
$EXIT 1
fi

pushd target-repo # To target repo
pushd "${TARGET_REPO}" # To target repo
Copy link
Copy Markdown
Contributor

@vchudnov-g vchudnov-g Apr 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per offline chat, sounds like we also need to mkdir -p owl-bot-staging/${DISTRIBUTION_NAME} after validation and before creating the PR.

Which means maybe we do the jq extract into ${DISTRIBUTION_NAME} and then set TARGET_PATH="packages/${DISTRIBUTION_NAME}" above?

Copy link
Copy Markdown
Contributor

@vchudnov-g vchudnov-g Apr 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per correction in chat, we'll need to

mkdir -p owl-bot-staging/${DISTRIBUTION_NAME}/${DISTRIBUTION_NAME}
touch owl-bot-staging/${DISTRIBUTION_NAME}/${DISTRIBUTION_NAME}/${DISTRIBUTION_NAME}.txt
git add owl-bot-staging
git commit -m "Trigger owlbot post-processor"

although it's not clear this is sufficient to make the migrated repo pass pre-submits.

Copy link
Copy Markdown
Contributor Author

@atulep atulep Apr 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, added owl-bot-staging. But, we should add any necessary fixes, if needed, in follow-up PRs.


git push -u origin "${BRANCH}" --force

Expand Down
56 changes: 0 additions & 56 deletions scripts/split_repo_migration/run.sh
View file
Open in desktop

This file was deleted.