API reference · Schema
EngineOptions
context_lengthintegerContext length in tokens each request may use (default: the model's context length, else the engine's default)
128–1048576
parallelintegerRequests one replica runs at once
1–256
Example
{
"context_length": 128,
"parallel": 1
}