Proposal: Maintainable binary gem ecosystem

[kou]

I want to propose the "maintainable binary gem ecosystem" concept.

Background

Currently, binary gems are maintained by gem maintainers. For example, Nokogiri's binary gems are maintained by Nokogiri maintainers.

Target platforms for binary gem are increased. It also increases maintenance costs of gem maintainers. For example, Nokogiri 1.19.0 provides 9 targets:

https://rubygems.org/gems/nokogiri/versions

• aarch64-linux-gnu
• aarch64-linux-musl
• arm-linux-gnu
• arm-linux-musl
• arm64-darwin
• java
• x64-mingw-ucrt
• x86_64-darwin
• x86_64-linux-gnu
• x86_64-linux-musl

I want to reduce maintenance costs of gem maintainers as much as possible to help gem ecosystem thrive. If maintenance costs of gem maintainers is higher, new gem maintainers may not be increased and current gem maintainers may stop maintaining. It may stagnate gem ecosystem.

Proposed solution

The "maintainable binary gem ecosystem" concept shares maintenance costs of binary gems with community instead of maintaining only by gem maintainers. Binary gems are maintained by others (not gem maintainers) like other package repositories such as Linux distribution package repositories (e.g. Debian's APT repository, Fedora's Yum repository, ...) and Homebrew. These package repositories are maintained by these package repositories' maintainers not upstream maintainers. I want to use this approach for binary gem ecosystem for maintainability.

How to implement the proposed solution

To split maintainers of source gem and binary gems, we need separated "gem owners" (who can push new versions) groups. Currently, rubygems.org has one "gem owners" group per gem. So we can't use the current rubygems.org features as-is to implement the proposed solution.

Here are some implementation ideas to split maintainers of source gem and binary gems:

1. Use separated RubyGems repositories instead of using only rubygems.org
2. Improve rubygems.org

1. Use separated RubyGems repositories instead of using only rubygems.org

If we use separated RubyGems repositories for binary gems, we can split maintainers of source gem and binary gems.

Gem maintainers pushes source gem to rubygems.org. Binary gem maintainers build binary gems from the source gem and push these binary gems to a separated RubyGems repository. rubygems.org and a separated RubyGems repository can use different "gem owners" groups. (Gem maintainers can push to rubygems.org but can't push to a separated RubyGems repository. Binary gem maintainers can't push to rubygems.org but can push to a separated RubyGems repository.)

For example, we can use GitHub's RubyGems repository feature: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-rubygems-registry

If we want to create binary gems maintainers for Ruby 4.0 on Ubuntu 24.04, we can create the "rubygems-ruby-4.0-ubuntu-24.04" GitHub organization and add maintainers to the organization. Maintainers can push binary gems for Ruby 4.0 on Ubuntu 24.04 to https://rubygems.pkg.github.com/rubygems-ruby-4.0-ubuntu-24.04 RubyGems repository.

Ruby 4.0 on Ubuntu 24.04 users can use binary gems by the following Gemfile:

source "https://rubygems.org"

source "https://rubygems.pkg.github.com/rubygems-ruby-4.0-ubuntu-24.04" do
  gem "nokogiri"
end

gem "no-binary-gem"

There are some considerations for this solution:

• Users need to know which gem has a binary gem for the target (Ruby 4.0 on Ubuntu 24.04).
  • It may be inconvenient.
• Should we create binary gems maintainers per OS + OS version instead of Linux + libc variant (e.g. Linux + glibc, Linux + musl. This is "manylinux" like approach)?
  • In my opinion, yes. I want to split binary gems maintainers as much as possible to reduce maintenance costs of each binary gems maintainers.
• ...

2. Improve rubygems.org

npm has the "scopes" feature: https://docs.npmjs.com/about-scopes

It provides visibility and namespace. In this context, namespace is only important. If rubygems.org has the namespace feature, we don't need separated RubyGems repositories.

For example, we can create a @ruby-4.0-ubuntu-24.04 namespace for binary gems of Ruby 4.0 on Ubuntu 24.04. We can have separated "gem owners" for source gem (e.g. nokogiri) and binary gems for binary gems (e.g. @ruby-4.0-ubuntu-24.04/nokogiri) The "gem owners" of @ruby-4.0-ubuntu-24.04 namespace can push binary gems for Ruby 4.0 on Ubuntu 24.04.

The next step

I want to evaluate the proposed solution. I want to move some binary gem related features from rake-compiler to RubyGems for easy to evaluate. Here are features I want to move:

• Building a binary gem for the current platform
  • rake-compiler can build a binary gem for different platform but I don't need it in RubyGems
  • gem build --binary? It doesn't use fake.rb technique that mimics RbConfig. It just does gem install and package the installed one as a binary gem.
• ...(I may need more features)...

[tenderlove]

I think this is a great idea. In my opinion, we should try to work on the first solution. I think we could provide binary gems to the community without making any changes to RubyGems.org, so it would be easier to deploy this "new feature" to members of the community (since we don't have to coordinate with RubyGems.org).

I would like it if we could write Gemfiles like this:

source "https://rubygems.org"

# Deploy to Ubuntu
source "https://rubygems.pkg.github.com/rubygems-ruby-4.0-ubuntu-24.04"

# Develop on macOS
source "https://rubygems.pkg.github.com/rubygems-ruby-4.0-macos-1234"

gem "nokogiri"

gem "no-binary-gem"

Then, if https://rubygems.pkg.github.com/rubygems-ruby-4.0-ubuntu-24.04 provides a binary version of nokogiri, it will automatically be used and the user doesn't have to investigate which source has what binary gems. Bundler currently doesn't recommend people use multiple top level source calls, but I think we should investigate why, and then maybe we can fix it so that multiple source is OK.

[kou]

Good point. Your suggestion will solve the "Users need to know which gem has a binary gem for the target (Ruby 4.0 on Ubuntu 24.04)" consideration.

In my opinion, we should try to work on the first solution.

It makes sense. I've created a GitHub organization for this: https://github.com/rubygems-ruby-4-0-ubuntu-24-04
(I couldn't use . for organization name.)

I'll prepare some binary gems in the organization.

Bundler currently doesn't recommend people use multiple top level source calls, but I think we should investigate why, and then maybe we can fix it so that multiple source is OK.

It seems that CVE-2013-0334 is the reason why Bundler doesn't recommend multiple sources:
https://bundler.io/blog/2014/08/14/bundler-may-install-gems-from-a-different-source-than-expected-cve-2013-0334.html

https://github.com/rubygems/bundler-site/blob/68bb2de20437dd2d58a69f1dda7f554f334a5849/source/v1.7/whats_new.html.haml#L57-L66
This part may be the important part for the CVE:

The order that top-level sources are searched is documented as last added to first added, but prior to version 1.7.0, there was not an automated test for this ordering, and it was inadvertently reversed in versions 1.5.0 through 1.6.5. The documented ordering is restored in this version, and is now included in the test suite to prevent regression.

Bundler 1.7.0 or later was released on 2014-08-13: https://github.com/rubygems/bundler/blob/1-7-stable/CHANGELOG.md#170-2014-08-13
So the "last added to first added" order is kept at least 10+ years. There is no ambiguity here now.

So I think that we can remove that multiple top-level source isn't recommended.

[Edouard-chin]

Thank you for opening this proposal and help move towards offering binary gems broadly for the ruby ecosystem!

Having multiple top-level source make sense to me. But I believe we'd need to modify Bundler to prioritize the source that contains gem with precompiled binaries (e.g. if two gems of the same name exists on rubygems.org and rubygems-ruby-4-0-ubuntu-24-04, Bundler will prioritize the first source).

---

Just a few questions that I'm wondering:

1. Do you think each source should be distribution/version specific ? e.g. ubuntu-24-04. It seems to be very narrow and I wonder if there is such a big maintenance burden of compiling for multiple platforms that would justify scoping each binary to a specific platform.
2. By multiplying the number of sources, should we be worried about malicious people that would create new sources with compromised binaries and promoting it to ruby developers? Maybe Rubygems could have a "trusted" source kind of feature 🤔
3. Would "binary gem maintainers" need consent from the gem maintainer to publish gem with precompiled binary on a different gem server ? Some issues with precompiled binaries may arise and ultimately developers will open issues on the gem issue tracker, so it would be good to always keep the gem maintainer in the loop. Similarly the gem maintainer may want to compile in a specific way (e.g. some may want to strip debugging symbols for smaller gem size, other would prefer to leave them as its easier to investigate crashes).

[kou]

1. Do you think each source should be distribution/version specific ? e.g. ubuntu-24-04. It seems to be very narrow and I wonder if there is such a big maintenance burden of compiling for multiple platforms that would justify scoping each binary to a specific platform.

Yes. I want to recommend it. In general, "one binary for multiple Linux distributions/versions" (like wheel does) is difficult to build "one binary only for one Linux distribution + version". The "one binary for multiple Linux distributions/versions" approach can reduce the number of artifacts but it has large maintenance costs. For example, we need to build all dependencies from source. We can't use existing packages.

The "one binary only for one Linux distribution + version" approach increases the number of artifacts but it's easier to maintain than the "one binary for multiple Linux distributions/versions". I think that total maintenance costs of the "one binary only for one Linux distribution + version" approach is lower than the "one binary only for one Linux distribution + version" approach. And it will reduce maintenance costs of each binary gem maintainer. Binary gem maintainers just need to know about only the target Linux distribution + version. They don't need to know all about all target Linux distributions/versions.

2. By multiplying the number of sources, should we be worried about malicious people that would create new sources with compromised binaries and promoting it to ruby developers? Maybe Rubygems could have a "trusted" source kind of feature 🤔

Good point. It may happen but it can happen event without my proposal. Because the current RubyGems and Bundler can already accept multiple sources.

We may need to recommend users to add additional sources carefully. We may not be able to mark a source as "trusted"... It seems that we don't have resources to check a source continually...

3. Would "binary gem maintainers" need consent from the gem maintainer to publish gem with precompiled binary on a different gem server ? Some issues with precompiled binaries may arise and ultimately developers will open issues on the gem issue tracker, so it would be good to always keep the gem maintainer in the loop. Similarly the gem maintainer may want to compile in a specific way (e.g. some may want to strip debugging symbols for smaller gem size, other would prefer to leave them as its easier to investigate crashes).

No. Because other package repositories don't do it.

"Binary gem maintainers" need to announce "users should report binary gem related problems to binary gem maintainers not the original gem maintainers" to users like other package repositories do. Binary gem maintainers will break down the problem into binary gem related problem and binary gem unrelated problem. If there is binary gem unrelated problem, binary gem maintainers or users will report the problem to the original gem maintainers.

(e.g. some may want to strip debugging symbols for smaller gem size, other would prefer to leave them as its easier to investigate crashes)

In this case, I suggest to create separated gem servers: rubygems-ruby-4-0-ubuntu-24-04 (that is optimized and debug symbols are stripped) and rubygems-ruby-4-0-ubuntu-24-04-debug (that has debug symbols)
Users switch to -debug gem server when there are some problems.

[kou]

I've created a RubyGems plugin that adds gem build_binary sub command: https://rubygems.org/gems/rubygems-build-binary