solar pro4 tokenizer
upstage/solar-pro4solar pro4 tokenizer (upstage/solar-pro4) is an AI model from Upstage with a 524,288-token context window and 131,072 max output tokens, priced at $0.09/1M input and $0.36/1M output tokens. Available via the haimaker.ai OpenAI-compatible API.
Overview
Tokenizer for Upstage Solar Pro 4, with a vocabulary of 196,608 tokens and support for English, Korean, Japanese, and more.
Model Card
Solar Pro 4 Tokenizer
Tokenizer for Upstage Solar Pro 4, with a vocabulary of 196,608 tokens and support for English, Korean, Japanese, and more.
This repository contains tokenizer files for encoding text and estimating text token counts. It does not contain model weights or a chat template.
Usage
Install the Transformers library:
pip install transformersfrom transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("upstage/solar-pro4-tokenizer")
text = "Hi, how are you?"
token_ids = tokenizer.encode(text, add_special_tokens=False)
print("Token IDs:", token_ids)
print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Number of tokens:", len(token_ids))
print("Decoded text:", tokenizer.decode(token_ids, skip_special_tokens=False))
To load a downloaded copy without network access:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"./solar-pro4-tokenizer",
local_files_only=True,
trust_remote_code=False,
)
These examples count tokens in the supplied text. A full API request can also include message formatting, system instructions, and tool definitions. Use the API response's usage fields for the actual request token usage.
See the token counting guide for more information.
License
Features & Capabilities
| Mode | chat |
| Context Window | 524,288 tokens |
| Max Output | 131,072 tokens |
| Function Calling | Supported |
| Vision | Not supported |
| Reasoning | Supported |
| Web Search | Not supported |
| Url Context | Not supported |
Technical Details
| Languages | en, ko, ja |
API Usage
from openai import OpenAI
client = OpenAI(
base_url="https://api.haimaker.ai/v1",
api_key="YOUR_API_KEY",
)
response = client.chat.completions.create(
model="upstage/solar-pro4",
messages=[
{"role": "user", "content": "Hello, how are you?"}
],
)
print(response.choices[0].message.content)Frequently Asked Questions
What is the context window of solar pro4 tokenizer?
solar pro4 tokenizer (upstage/solar-pro4) has a 524,288-token context window and supports up to 131,072 output tokens per request.
How much does solar pro4 tokenizer cost?
solar pro4 tokenizer is priced at $0.09 per 1M input tokens and $0.36 per 1M output tokens when accessed via the haimaker.ai OpenAI-compatible API.
What features does solar pro4 tokenizer support?
solar pro4 tokenizer supports function calling, reasoning.
How do I use solar pro4 tokenizer via API?
Send requests to https://api.haimaker.ai/v1/chat/completions with model "upstage/solar-pro4" using any OpenAI-compatible SDK. Authentication uses a Bearer API key from https://app.haimaker.ai.
Use solar pro4 tokenizer with the haimaker API
OpenAI-compatible endpoint. Start building in minutes.