08. Sampler Overrides

Overview

Clients can send sampler parameters that users are unaware of, or even not send any parameters at all!

Sampler overrides is tabbyAPI's flexible proxy for adding or forcing sampler values before generation.

Setting up

All supported samplers are located in sampler_overrides/sample_preset.yml.

Note

Sampler overrides can also be switched at runtime via the /v1/sampling/override endpoint set. Please read the API docs for more information.

Steps:

1. Create a new YML in the sampler_overrides folder or duplicate sample_preset.yml
2. Adjust the samplers you want to override using the examples below
3. Save and rename the file (if you haven't already)
4. Open config.yml and inside the sampling block, set the sampler_override key to the preset name from step 3

Examples

Let's say a client doesn't send a value for top_p:

top_p:
	override: 0.7
	force: False

This override will change the default fallback of top_p from a neutral value of 1.0 to 0.7.

Now, let's say a client forces a value that you don't like (ex. top-k: 30):

top_k:
	override: 0
	force: True

This override forces top_k to be 0 no matter what the client sends.

Finally, let's say a client sends a list, but you want to add something else:

stop:
	override: ["a"]
	force: False
	additive: True

Now this override appends the given stop string to whatever the client sends.