Pre-release app is broken with out of date database and can't be deployed

[nickcharlton]

Currently, if you go to https://administrate-demo-prerelease.herokuapp.com/admin, you get a 500 error.

We have this setup to deploy immediately from main, so it sometimes breaks when we make changes.

Looking in the logs on Heroku, the origin of that error is:

2024-12-02T14:14:49.220711+00:00 app[web.1]: I, [2024-12-02T14:14:49.220645 #5]  INFO -- : [ad08f3b6-48e9-4a0f-bf35-c717fb30dda9] Completed 500 Internal Server Error in 42ms (ActiveRecord: 15.1ms | Allocations: 3949)
2024-12-02T14:14:49.221283+00:00 app[web.1]: F, [2024-12-02T14:14:49.221240 #5] FATAL -- : [ad08f3b6-48e9-4a0f-bf35-c717fb30dda9]   
2024-12-02T14:14:49.221284+00:00 app[web.1]: [ad08f3b6-48e9-4a0f-bf35-c717fb30dda9] ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "customers" does not exist
2024-12-02T14:14:49.221285+00:00 app[web.1]: LINE 9:  WHERE a.attrelid = '"customers"'::regclass
2024-12-02T14:14:49.221286+00:00 app[web.1]:                             ^
2024-12-02T14:14:49.221286+00:00 app[web.1]: ):
2024-12-02T14:14:49.221287+00:00 app[web.1]: [ad08f3b6-48e9-4a0f-bf35-c717fb30dda9]   
2024-12-02T14:14:49.221287+00:00 app[web.1]: [ad08f3b6-48e9-4a0f-bf35-c717fb30dda9] app/controllers/admin/customers_controller.rb:16:in `scoped_resource'

But, if I deploy, we get:

remote: -----> Detecting rake tasks
remote:
remote:  !
remote:  !     Could not detect rake tasks
remote:  !     ensure you can run `$ bundle exec rake -P` against your app
remote:  !     and using the production group of your Gemfile.
remote:  !     /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/definition.rb:35:in `build': /tmp/build_8acb78df/spec/example_app/Gemfile not found (Bundler::GemfileNotFound)
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler.rb:208:in `definition'
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler.rb:156:in `setup'
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/setup.rb:23:in `block in <top (required)>'
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/ui/shell.rb:159:in `with_level'
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/ui/shell.rb:111:in `silence'
remote:  !     from /tmp/build_8acb78df/vendor/bundle/ruby/3.2.0/gems/bundler-2.4.22/lib/bundler/setup.rb:23:in `<top (required)>'
remote:  !     from <internal:/tmp/build_8acb78df/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
remote:  !     from <internal:/tmp/build_8acb78df/vendor/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
remote:  !     from /tmp/build_8acb78df/spec/example_app/config/boot.rb:3:in `<top (required)>'
remote:  !     from /tmp/build_8acb78df/bin/rake:3:in `require_relative'
remote:  !     from /tmp/build_8acb78df/bin/rake:3:in `<main>'

It's buried in there a bit, but it's because something (the buildpack?) is looking for the Gemfile in the wrong place.

[nickcharlton]

I've been spending time on this on and off all week, and finally made some sort of breakthrough.

I discovered that buildpacks, whilst tagged, aren't actually versioned anywhere visible on Heroku (via the Web or the CLI), which was frustrating. We last probably used v264, so I tried replacing that and redeploying but the current version of main failed:

$ heroku buildpacks:clear --app=administrate-demo-prerelease
Buildpacks cleared. Next release on administrate-demo-prerelease will detect buildpacks normally.
$ heroku buildpacks:set --app=administrate-demo-prerelease https://github.com/heroku/heroku-buildpack-ruby.git#v264
Buildpack set. Next release on administrate-demo-prerelease will use https://github.com/heroku/heroku-buildpack-ruby.git#v264.
Run git push heroku main to create a new release using this buildpack.

I then tried a branch based off of where we last successfully deployed (2024-01-17), which worked:

$ git checkout -b last-successful-deploy 6e03eb5a
Switched to a new branch 'last-successful-deploy'
$ git push -f prerelease last-successful-deploy:main
[ … ] 
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/administrate-demo-prerelease.git
 + 2d667c1c...6e03eb5a last-successful-deploy -> main (forced update)

So somewhere we made an application change which is causing problems. More experiments needed.

[nickcharlton]

Oh, and: it's 500ing because I guess we never ran the migrations earlier in the year. That was an easy fix, so that's the app up and running again at least.

[nickcharlton]

I just (finally) got to bisect the changes, and it was the addition of #2705 which caused this.