diff --git a/README.md b/README.md index 0f28262a3..bba402db8 100644 --- a/README.md +++ b/README.md @@ -4,13 +4,16 @@ This repository contains: - - `ditto clone`: a script to crawl an instance of PokeAPI and download all objects - - `data/api`: a static copy of the JSON data generated with the above script - - `ditto analyze`: a script to generate a JSON schema of the above data - - `data/schema`: a static copy of the PokeAPI schema generated from the above data - - `ditto transform`: a script to apply a new base url to data in `data/api` - - `ditto serve`: a script to serve the data in the same form as PokeAPI - - with full support for dynamic pagination using GET args `offset` and `limit` + - Ditto script: + - `ditto clone`: a script to crawl an instance of PokeAPI and download all objects + - `ditto analyze`: a script to generate a JSON schema of the above data + - `ditto transform`: a script to apply a new base url to data in `data/api` + - `ditto serve`: a script to serve the data in the same form as PokeAPI + - with full support for dynamic pagination using GET args `offset` and `limit` + - Static data: + - [data/api](data/api): a static copy of the JSON data generated with the above script + - [data/schema](data/schema): a static copy of the PokeAPI schema generated from the above data + - [updater](updater): a bot that runs in docker and can update the data stored in this repo ## Docker @@ -21,7 +24,7 @@ just have to run one command. - Replace `http://localhost:8080` with the base url of your choice ``` bash -docker run -p 8080:80 -e DITTO_BASE_URL=http://localhost:8080 sargunv/pokeapi-ditto +docker run -p 8080:80 -e DITTO_BASE_URL=http://localhost:8080 sargunv/pokeapi-ditto:latest ``` ## Usage @@ -44,64 +47,4 @@ The general idea is the same. ## Advanced -You can manually update the data if necessary. If I abandon this -project, here’s how to update it. It's a bit of an involved process. - -Before starting, you’ll need to install [Docker and Docker -Compose](https://docs.docker.com/compose/install/). You'll -also need [Poetry](https://poetry.eustace.io/) in your PATH. - -First clone the PokeAPI and Ditto repositories: - -``` bash -cd ~ -git clone https://github.com/PokeAPI/ditto.git -git clone https://github.com/PokeAPI/pokeapi.git -``` - -Apply the patch to disable rate limiting on your local PokeAPI: - -``` bash -# Assuming you have the repos in ~ -cd ~/pokeapi -git apply ~/ditto/extra/disable-rate-limit.patch -``` - -Run PokeAPI using docker-compose: - -``` bash -docker volume create --name=redis_data -docker volume create --name=pg_data -docker-compose up -d -``` - -Build the PokeAPI database: - -``` bash -docker-compose exec app python manage.py migrate -docker-compose exec app python manage.py shell -``` - -``` python -from data.v2.build import build_all -build_all() -``` - -Once it’s done, you can update Ditto’s data: - -``` bash -cd ~/ditto -rm -r ./data -poetry install -poetry run ditto clone --src-url http://localhost/ --dest-dir ./data -poetry run ditto analyze --api-dir ./data/api --schema-dir ./data/schema -``` - -This will crawl your local instance of PokeAPI, copy all the data to -./data, and regenerate the schema. - -Once that's finished, you can serve the freshly updated data! - -``` bash -poetry run ditto serve --port 8080 --base-url http://localhost:8080 -``` \ No newline at end of file +You can manually update the data if necessary. See [the updater bot](updater). You can run the bot in docker, or read and adapt its update script yourself. diff --git a/updater/Dockerfile b/updater/Dockerfile new file mode 100644 index 000000000..b4fa26ee0 --- /dev/null +++ b/updater/Dockerfile @@ -0,0 +1,23 @@ +FROM docker:dind + +RUN apk update +RUN apk add curl python3 git bash dos2unix openssh build-base python3-dev + +RUN ln -sf $(ls /usr/bin/easy_install*) /usr/bin/easy_install +RUN easy_install pip +RUN pip install docker-compose +RUN pip install poetry + +RUN mkdir /updater +WORKDIR /updater +COPY . /updater/ +RUN dos2unix cmd.bash + +ENV COMMIT_NAME 'Updater Bot' +ENV COMMIT_EMAIL '' +ENV COMMIT_MESSAGE '[Updater Bot] Regenerate data' +ENV BRANCH_NAME 'updater-bot' +ENV REPO_POKEAPI 'git@github.com:PokeAPI/pokeapi' +ENV REPO_DITTO 'git@github.com:PokeAPI/ditto' + +CMD bash cmd.bash diff --git a/updater/README.md b/updater/README.md new file mode 100644 index 000000000..d77ebc1ef --- /dev/null +++ b/updater/README.md @@ -0,0 +1,29 @@ +# Updater Bot + +## Usage + +First, make sure you can read/write the target repository over SSH. +Launch the bot with a volume containing the SSH keys to `/root/.ssh` and an environment variable for email address. +Since this container runs Docker within itself, it needs to run in privileged mode. + +``` +docker run --privileged -v ~/.ssh:/root/.ssh -e COMMIT_EMAIL=example@example.com sargunv/pokeapi-ditto:updater +``` + +**Note:** Due to lack of support for file permissions, this does not work on Docker for Windows. + +## Environment Variables + +### Required + + - `COMMIT_EMAIL` + +### Optional + +See [the Dockerfile](updater/Dockerfile) for the defaults. + + - `COMMIT_NAME` + - `COMMIT_MESSAGE` + - `BRANCH_NAME` + - `REPO_POKEAPI` + - `REPO_DITTO` diff --git a/updater/cmd.bash b/updater/cmd.bash new file mode 100644 index 000000000..50e22e4cd --- /dev/null +++ b/updater/cmd.bash @@ -0,0 +1,48 @@ +#!/usr/bin/env bash + +[ -z "${COMMIT_NAME}" ] && { echo "Need to set COMMIT_NAME"; exit 1; } +[ -z "${COMMIT_EMAIL}" ] && { echo "Need to set COMMIT_EMAIL"; exit 1; } +[ -z "${COMMIT_MESSAGE}" ] && { echo "Need to set COMMIT_MESSAGE"; exit 1; } +[ -z "${REPO_POKEAPI}" ] && { echo "Need to set REPO_POKEAPI"; exit 1; } +[ -z "${REPO_DITTO}" ] && { echo "Need to set REPO_DITTO"; exit 1; } +[ -z "${BRANCH_NAME}" ] && { echo "Need to set BRANCH_NAME"; exit 1; } + +set -e +set -o pipefail +set -x + +export COMPOSE_INTERACTIVE_NO_CLI=1 + +dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 &> /dev/null & + +git clone --depth=1 "$REPO_POKEAPI" pokeapi +git clone --depth=1 "$REPO_DITTO" ditto + +# set up the pokeapi side +cd pokeapi +git apply ../disable-rate-limit.patch + +docker volume create --name=redis_data +docker volume create --name=pg_data +docker-compose up -d + +docker-compose exec -T app python manage.py migrate +docker-compose exec -T app sh -c 'echo "from data.v2.build import build_all; build_all()" | python manage.py shell' + +# set up the ditto side +cd ../ditto +git branch -D "$BRANCH_NAME" || true +git branch "$BRANCH_NAME" +git checkout "$BRANCH_NAME" + +poetry install +rm -r ./data +poetry run ditto clone --src-url http://localhost/ --dest-dir ./data +poetry run ditto analyze --api-dir ./data/api --schema-dir ./data/schema + +# commit and push +git add data +git config user.name "$COMMIT_NAME" +git config user.email "$COMMIT_EMAIL" +git commit -m "$COMMIT_MESSAGE" +git push -fu origin "$BRANCH_NAME" diff --git a/extra/disable-rate-limit.patch b/updater/disable-rate-limit.patch similarity index 100% rename from extra/disable-rate-limit.patch rename to updater/disable-rate-limit.patch