Feature Request: add --cells-to-watch flag to VTOrc

[MargaretMorehead]

Feature Description

VTOrc currently supports scoping its monitoring to specific keyspaces/shards via --clusters-to-watch, but there is no way to scope it by cell. In multi-cell deployments, operators may want to run separate VTOrc instances that each monitor only a subset of cells.

This proposes adding a --cells-to-watch flag that accepts a comma-separated list of cells. When set, VTOrc will only discover and monitor tablets in the specified cells. When unset, the existing behavior of watching all cells is preserved. The flag works in combination with --clusters-to-watch — both filters are applied when both are set.

Use Case(s)

• Running different Vitess versions in different cells and needing version-matched VTOrc instances to prevent cross-version recovery operations. This is our use case at HubSpot.
• Reducing expensive topo reads by limiting VTOrc to only fetch tablets from specified cells instead of all known cells (see RFC: VTOrc to support cell locality #18982)
• Running a dedicated VTOrc instance per cell/region to reduce blast radius and improve isolation

[timvaillancourt]

@MargaretMorehead I'm curious how a setup like this works in practice (today without the improvements mentioned in #18982)

Mainly, I'm curious: how primaries in another cell jive today. Does it get confused if the VTOrc is not watching the cell with the primary? Or any ill effects?

[corbantek]

@MargaretMorehead I'm curious how a setup like this works in practice (today without the improvements mentioned in #18982)

Mainly, I'm curious: how primaries in another cell jive today. Does it get confused if the VTOrc is not watching the cell with the primary? Or any ill effects?

We're currently upgrading from Vitess 14 to Vitess 22 at HubSpot, running two cells simultaneously:

• Vitess 14 (current production)
• Vitess 22 (upgrade target)

Each cell runs its own VTOrc instance matching the Vitess version. Without --cells_to_watch, each VTOrc discovers tablets from all cells, which creates cross-version visibility issues since Vitess 14 VTOrc is not fully compatible with Vitess 22+ tablets.

During the upgrade, keyspaces temporarily exist in both cells as we migrate traffic. From our observations so far:

The VTOrc instance in the same cell as the primary appears to take precedence for recovery operations

• We haven't seen significant ill effects, though we're still early in the process
• The main concern is ensuring the version-matched VTOrc handles failovers for its tablets
• This feature gives us explicit control rather than relying on implicit behavior. It's been working well enough to unblock our Vitess 14 → 22 migration path.

FWIW, we originally tried using --clusters_to_watch to dynamically update VTOrc's keyspace list as keyspaces migrated between cells. This didn't scale—as we have 2000+ keyspaces in some cells, and the constant config updates triggered VTOrc restarts. Since VTOrc uses emptyDir for raft consensus state, restarts caused leader election state loss and degraded our failover reliability.

[timvaillancourt]

@corbantek thanks! If this is already functioning maybe there is less work than I expected to achieve #18982. We've never approached a VTOrc deployment like this, so my mental picture has been theory until now

Let's say you have 3 x cells and the PRIMARY fails in "cell A". Let me know if I got anything wrong here:

1. Assuming only 1 x tablet in "cell A", the VTOrc in "cell A" would notice the PRIMARY is broken, but it wouldn't do anything, because it has no knowledge of other replicas to form a quorum
2. The VTOrcs in "cellB" and "cellC" would each see that non-PRIMARY tablets cannot connect to their PRIMARY
3. Because each VTOrc only knows it's local-cell tablets, it would see this as "100% replicas broken" and trigger an EmergencyReparentShard
4. This reparent updates the Shard record in the global topo
5. All isolated VTOrcs see the new Shard record in the global topo

If I got this correct above, that's pretty interesting! VTOrc wasn't intentionally designed for that setup, but I can now see how it would function, assuming I got this right

What worries me about this approach (if it were long term) is point #3, where we aren't really using the replicas as a quorum. I think in your approach you'll only use the replicas you can "see" to form a quorum of PRIMARY down, and this can have some gotchas

[corbantek]

@corbantek thanks! If this is already functioning maybe there is less work than I expected to achieve #18982. We've never approached a VTOrc deployment like this, so my mental picture has been theory until now

Let's say you have 3 x cells and the PRIMARY fails in "cell A". Let me know if I got anything wrong here:

1. Assuming only 1 x tablet in "cell A", the VTOrc in "cell A" would notice the PRIMARY is broken, but it wouldn't do anything, because it has no knowledge of other replicas to form a quorum
2. The VTOrcs in "cellB" and "cellC" would each see that non-PRIMARY tablets cannot connect to their PRIMARY
3. Because each VTOrc only knows it's local-cell tablets, it would see this as "100% replicas broken" and trigger an EmergencyReparentShard
4. This reparent updates the Shard record in the global topo
5. All isolated VTOrcs see the new Shard record in the global topo

If I got this correct above, that's pretty interesting! VTOrc wasn't intentionally designed for that setup, but I can now see how it would function, assuming I got this right

What worries me about this approach (if it were long term) is point #3, where we aren't really using the replicas as a quorum. I think in your approach you'll only use the replicas you can "see" to form a quorum of PRIMARY down, and this can have some gotchas

@timvaillancourt

I can do more scenario testing with this next week, but for our use case its slightly different than your example. We have only two cells and each cell has full working replica set (IE at least 3 tablets).

So in our case if the primary exists in "Cell A" then vtorc A will perform operations in Cell A; the same is true if the primary were to be in Cell B. We have other automation that handles reparenting between A and B (using vtctldclient)

Honestly we rarely have tablets existing in both cells for a given keyspace, its only during the transitional period when upgrading the keyspace (in our case from Vitess 14 to Vitess 22). This allows us to spin up Vitess 22 for the keyspace, have it run for a bit, then we can reparent over from Vitess 14 to 22 and let it bake (a few hours to days depending on criticality). If anything goes nuts in 22 we can instantly move back to 14 without issue. Otherwise we delete the tablets in 14 and the keyspace is back to only existing in a single cell...