ONNX Runtime generate() C API
Note: this API is in preview and is subject to change.
- Overview
- Model API
- Tokenizer API
- Generator Params API
- Generator API
- Enums and structs
- Utility functions
Overview
Model API
Create model
Creates a model from the given directory. The directory should contain a file called genai_config.json
, which corresponds to the configuration specification.
Parameters
- Input: config_path The path to the model configuration directory. The path is expected to be encoded in UTF-8.
- Output: out The created model.
Returns
OgaResult
containing the error message if the model creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateModel(const char* config_path, OgaModel** out);
Destroy model
Destroys the given model.
Parameters
- Input: model The model to be destroyed.
Returns
void
OGA_EXPORT void OGA_API_CALL OgaDestroyModel(OgaModel* model);
Generate
Generates an array of token arrays from the model execution based on the given generator params.
Parameters
- Input: model The model to use for generation.
- Input: generator_params The parameters to use for generation.
- Output: out The generated sequences of tokens. The caller is responsible for freeing the sequences using OgaDestroySequences after it is done using the sequences.
Returns
OgaResult containing the error message if the generation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerate(const OgaModel* model, const OgaGeneratorParams* generator_params, OgaSequences** out);
Tokenizer API
Create Tokenizer
Parameters
- Input: model. The model for which the tokenizer should be created
Returns
OgaResult
containing the error message if the tokenizer creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateTokenizer(const OgaModel* model, OgaTokenizer** out);
Destroy Tokenizer
OGA_EXPORT void OGA_API_CALL OgaDestroyTokenizer(OgaTokenizer*);
Encode
Encodes a single string and adds the encoded sequence of tokens to the OgaSequences. The OgaSequences must be freed with OgaDestroySequences when it is no longer needed.
Parameters
Returns
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerEncode(const OgaTokenizer*, const char* str, OgaSequences* sequences);
Decode
Decode a single token sequence and returns a null terminated utf8 string. out_string must be freed with OgaDestroyString
Parameters
Returns
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerDecode(const OgaTokenizer*, const int32_t* tokens, size_t token_count, const char** out_string);
Encode batch
Parameters
-
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerEncodeBatch(const OgaTokenizer*, const char** strings, size_t count, TokenSequences** out);
Decode batch
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerDecodeBatch(const OgaTokenizer*, const OgaSequences* tokens, const char*** out_strings);
Destroy tokenizer strings
OGA_EXPORT void OGA_API_CALL OgaTokenizerDestroyStrings(const char** strings, size_t count);
Create tokenizer stream
OgaTokenizerStream is used to decoded token strings incrementally, one token at a time.
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateTokenizerStream(const OgaTokenizer*, OgaTokenizerStream** out);
Destroy tokenizer stream
Parameters
OGA_EXPORT void OGA_API_CALL OgaDestroyTokenizerStream(OgaTokenizerStream*);
Decode stream
Decode a single token in the stream. If this results in a word being generated, it will be returned in ‘out’. The caller is responsible for concatenating each chunk together to generate the complete result. ‘out’ is valid until the next call to OgaTokenizerStreamDecode or when the OgaTokenizerStream is destroyed
OGA_EXPORT OgaResult* OGA_API_CALL OgaTokenizerStreamDecode(OgaTokenizerStream*, int32_t token, const char** out);
Generator Params API
Create Generator Params
Creates a OgaGeneratorParams from the given model.
Parameters
- Input: model The model to use for generation.
- Output: out The created generator params.
Returns
OgaResult
containing the error message if the generator params creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateGeneratorParams(const OgaModel* model, OgaGeneratorParams** out);
Destroy Generator Params
Destroys the given generator params.
Parameters
- Input: generator_params The generator params to be destroyed.
Returns
void
OGA_EXPORT void OGA_API_CALL OgaDestroyGeneratorParams(OgaGeneratorParams* generator_params);
Set search option (number)
Set a search option where the option is a number
Parameters
- generator_params: The generator params object to set the parameter on
- name: the name of the parameter
- value: the value to set
Returns
OgaResult
containing the error message if the generator params creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetSearchNumber(OgaGeneratorParams* generator_params, const char* name, double value);
Set search option (bool)
Set a search option where the option is a bool.
Parameters
- generator_params: The generator params object to set the parameter on
- name: the name of the parameter
- value: the value to set
Returns
OgaResult
containing the error message if the generator params creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetSearchBool(OgaGeneratorParams* generator_params, const char* name, bool value);
Try graph capture with max batch size
Graph capture fixes the dynamic elements of the computation graph to constant values. It can provide more efficient execution in some environments. To execute in graph capture mode, the maximum batch size needs to be known ahead of time. This function can fail if there is not enough memory to allocate the specified maximum batch size.
Parameters
- generator_params: The generator params object to set the parameter on
- max_batch_size: The maximum batch size to allocate
Returns
OgaResult
containing the error message if graph capture mode could not be configured with the specified batch size
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsTryGraphCaptureWithMaxBatchSize(OgaGeneratorParams* generator_params, int32_t max_batch_size);
Set inputs
Sets the input ids for the generator params. The input ids are used to seed the generation.
Parameters
- Input: generator_params The generator params to set the input ids on.
- Input: input_ids The input ids array of size input_ids_count = batch_size * sequence_length.
- Input: input_ids_count The total number of input ids.
- Input: sequence_length The sequence length of the input ids.
- Input: batch_size The batch size of the input ids.
Returns
OgaResult
containing the error message if the setting of the input ids failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetInputIDs(OgaGeneratorParams* generator_params, const int32_t* input_ids, size_t input_ids_count, size_t sequence_length, size_t batch_size);
Set input sequence
Sets the input id sequences for the generator params. The input id sequences are used to seed the generation.
Parameters
- Input: generator_params The generator params to set the input ids on.
- Input: sequences The input id sequences.
Returns
OgaResult containing the error message if the setting of the input id sequences failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetInputSequences(OgaGeneratorParams* generator_params, const OgaSequences* sequences);
Set model input
Set an additional model input, aside from the input_ids. For example additional inputs for LoRA adapters.
Parameters
- generator_params: The generator params to set the input on
- name: the name of the parameter to set
- tensor: the value of the parameter
Returns
OgaResult containing the error message if the setting of the input failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGeneratorParamsSetWhisperInputFeatures(OgaGeneratorParams*, OgaTensor* tensor);
Generator API
Create Generator
Creates a generator from the given model and generator params.
Parameters
- Input: model The model to use for generation.
- Input: params The parameters to use for generation.
- Output: out The created generator.
Returns
OgaResult
containing the error message if the generator creation failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateGenerator(const OgaModel* model, const OgaGeneratorParams* params, OgaGenerator** out);
Destroy generator
Destroys the given generator.
Parameters
- Input: generator The generator to be destroyed.
Returns
void
OGA_EXPORT void OGA_API_CALL OgaDestroyGenerator(OgaGenerator* generator);
Check if generation has completed
Returns true if the generator has finished generating all the sequences.
Parameters
- Input: generator The generator to check if it is done with generating all sequences.
Returns
True if the generator has finished generating all the sequences, false otherwise.
OGA_EXPORT bool OGA_API_CALL OgaGenerator_IsDone(const OgaGenerator* generator);
Run one iteration of the model
Computes the logits from the model based on the input ids and the past state. The computed logits are stored in the generator.
Parameters
- Input: generator The generator to compute the logits for.
Returns
OgaResult containing the error message if the computation of the logits failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_ComputeLogits(OgaGenerator* generator);
Generate next token
Generates the next token based on the computed logits using the configured generation parameters.
Parameters
- Input: generator The generator to generate the next token for.
Returns
OgaResult containing the error message if the generation of the next token failed.
OGA_EXPORT OgaResult* OGA_API_CALL OgaGenerator_GenerateNextToken(OgaGenerator* generator);
Get number of tokens
Returns the number of tokens in the sequence at the given index.
Parameters
- Input: generator The generator to get the count of the tokens for the sequence at the given index.
- Input: index. The index at which to return the tokens
Returns
The number tokens in the sequence at the given index.
OGA_EXPORT size_t OGA_API_CALL OgaGenerator_GetSequenceCount(const OgaGenerator* generator, size_t index);
Get sequence
Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by OgaGenerator_GetSequenceCount
.
Parameters
- Input: generator The generator to get the sequence data for the sequence at the given index. The pointer to the sequence data at the given index. The sequence data is owned by the OgaGenerator and will be freed when the OgaGenerator is destroyed. The caller must copy the data if it needs to be used after the OgaGenerator is destroyed.
- Input: index. The index at which to get the sequence.
Returns
A pointer to the token sequence
OGA_EXPORT const int32_t* OGA_API_CALL OgaGenerator_GetSequenceData(const OgaGenerator* generator, size_t index);
Enums and structs
typedef enum OgaDataType {
OgaDataType_int32,
OgaDataType_float32,
OgaDataType_string, // UTF8 string
} OgaDataType;
typedef struct OgaResult OgaResult;
typedef struct OgaGeneratorParams OgaGeneratorParams;
typedef struct OgaGenerator OgaGenerator;
typedef struct OgaModel OgaModel;
typedef struct OgaBuffer OgaBuffer;
Utility functions
Set the GPU device ID
OGA_EXPORT OgaResult* OGA_API_CALL OgaSetCurrentGpuDeviceId(int device_id);
Get the GPU device ID
OGA_EXPORT OgaResult* OGA_API_CALL OgaGetCurrentGpuDeviceId(int* device_id);
Get error message
Parameters
- Input: result OgaResult that contains the error message.
Returns
Error message contained in the OgaResult. The const char* is owned by the OgaResult and can will be freed when the OgaResult is destroyed.
OGA_EXPORT const char* OGA_API_CALL OgaResultGetError(OgaResult* result);
Destroy result
Parameters
- Input: result OgaResult to be destroyed.
Returns
void
OGA_EXPORT void OGA_API_CALL OgaDestroyResult(OgaResult*);
Destroy string
Parameters
- Input: string to be destroyed
Returns
OGA_EXPORT void OGA_API_CALL OgaDestroyString(const char*);
Destroy buffer
Parameters
- Input: buffer to be destroyed
Returns
void
OGA_EXPORT void OGA_API_CALL OgaDestroyBuffer(OgaBuffer*);
Get buffer type
Parameters
- Input: the buffer
Returns
The type of the buffer
OGA_EXPORT OgaDataType OGA_API_CALL OgaBufferGetType(const OgaBuffer*);
Get the number of dimensions of a buffer
Parameters
- Input: the buffer
Returns
The number of dimensions in the buffer
OGA_EXPORT size_t OGA_API_CALL OgaBufferGetDimCount(const OgaBuffer*);
Get buffer dimensions
Get the dimensions of a buffer
Parameters
- Input: the buffer
- Output: a dimension array
Returns
OgaResult
OGA_EXPORT OgaResult* OGA_API_CALL OgaBufferGetDims(const OgaBuffer*, size_t* dims, size_t dim_count);
Get buffer data
Get the data from a buffer
Parameters
Returns
void
OGA_EXPORT const void* OGA_API_CALL OgaBufferGetData(const OgaBuffer*);
Create sequences
OGA_EXPORT OgaResult* OGA_API_CALL OgaCreateSequences(OgaSequences** out);
Destroy sequences
Parameters
- Input: sequences OgaSequences to be destroyed.
Returns
void
Returns
OGA_EXPORT void OGA_API_CALL OgaDestroySequences(OgaSequences* sequences);
Get number of sequences
Returns the number of sequences in the OgaSequences
Parameters
- Input: sequences
Returns
The number of sequences in the OgaSequences
OGA_EXPORT size_t OGA_API_CALL OgaSequencesCount(const OgaSequences* sequences);
Get the number of tokens in a sequence
Returns the number of tokens in the sequence at the given index
Parameters
- Input: sequences
Returns
The number of tokens in the sequence at the given index
OGA_EXPORT size_t OGA_API_CALL OgaSequencesGetSequenceCount(const OgaSequences* sequences, size_t sequence_index);
Get sequence data
Returns a pointer to the sequence data at the given index. The number of tokens in the sequence is given by OgaSequencesGetSequenceCount
Parameters
- Input: sequences
Returns
The pointer to the sequence data at the given index. The pointer is valid until the OgaSequences is destroyed.
OGA_EXPORT const int32_t* OGA_API_CALL OgaSequencesGetSequenceData(const OgaSequences* sequences, size_t sequence_index);