Skip to content

ModelLoadOptions

ModelLoadOptions = Pick<ChatRequest, "num_ctx" | "num_gpu" | "num_batch" | "num_thread" | "keep_alive">

Defined in: src/lib/providers/types.ts:175

The request fields that decide how a server loads a model, as opposed to how it samples. A warmup must send the same values the chat request will, or the server loads the model once for the warmup and again for the first chat (#315).