Efficiency: Require significantly less computational resources to train and deploy
效率：训练和部署所需的计算资源显著减少
Customization: Easier to fine-tune and adapt to specific domains
定制：更容易进行微调并适应特定领域
Control: Better understanding and control of model behavior
控制：更好地理解和控制模型行为
Cost: Lower operational costs for training and inference
成本：更低的训练和推理运营成本
Privacy: Can be run locally without sending data to external APIs
隐私：可以在本地运行，而无需将数据发送到外部 API (Application Programming Interface)
Green Technology: Advocates efficient usage of resources with reduced carbon footprint
绿色技术：倡导高效利用资源，减少碳足迹
Easier Academic Research Development: Provides an easy starter for academic research with cutting-edge LLMs with less logistical constraints
更容易的学术研究开发：为学术研究提供了一个简单的入门，使用最先进的 LLM (Large Language Models)，减少了后勤方面的限制

Basic understanding of machine learning and natural language processing.
对机器学习和自然语言处理的基本理解。
Familiarity with Python, PyTorch, and the transformers library.
熟悉 Python、PyTorch 和 transformers 库。
Access to a pre-trained language model and a labeled dataset.
访问预训练的语言模型和标记数据集。

Follow user instructions accurately
准确地遵循用户指令
Engage in natural conversations
参与自然的对话
Provide helpful, harmless, and honest responses
提供有帮助、无害和诚实的回答
Maintain context across multi-turn interactions
维持跨多轮交互的上下文
Use tools or MCP servers to perform tasks
使用工具或 MCP (Model Control Program) 服务器来执行任务

Fits in a single GPU at a reasonable cost
以合理的成本适配单个 GPU
Achieves competitive performance
达到具有竞争力的性能
Supports multilingual conversations
支持多语言对话
Supports extended context length up to 8k tokens (with some variants supporting longer contexts up to 128k tokens)
支持长达 8k tokens 的扩展上下文长度（某些变体支持长达 128k tokens 的更长上下文）
Features dual-mode reasoning with explicit thinking capabilities
具有带显式思考能力的双模式推理
Comes with complete training recipes so you understand exactly how it was built
附带完整的训练配方，以便您确切了解它是如何构建的

How SmolLM3's chat template works
SmolLM3 的聊天模板如何工作
Converting conversations to the proper format
将对话转换为正确的格式
Working with system prompts and multi-turn conversations
使用系统提示和多轮对话
Using the Transformers library's built-in template support
使用 Transformers 库的内置模板支持

The theory behind SFT and when to use it
SFT 背后的理论以及何时使用它
Working with the SmolTalk2 dataset
使用 SmolTalk2 数据集
Using TRL's SFTTrainer for efficient training
使用 TRL 的 SFTTrainer 进行高效训练
Best practices for data preparation and training configuration
数据准备和训练配置的最佳实践

Process datasets for instruction tuning
处理用于指令调优的数据集
Fine-tune SmolLM3 on different tasks
在不同的任务上微调 SmolLM3
Use both Python APIs and CLI tools
使用 Python API 和 CLI 工具
• Compare base model vs fine-tuned model performance

What You'll Build

Fine-tuned your own SmolLM3 model on a custom dataset
在自定义数据集上微调的您自己的 SmolLM3 模型
Understanding of chat template formatting and conversation structure
对聊天模板格式和对话结构的理解
Experience with both programmatic and CLI-based training workflows
使用基于程序和基于 CLI 的训练工作流程的经验
A model deployed to Hugging Face Hub that others can use
部署到 Hugging Face Hub 且其他人可以使用的模型
Foundation knowledge for more advanced fine-tuning techniques
用于更高级微调技术的基础知识

Chat template: A structured format for interactions between language models, users, and external tools.
Chat template (聊天模板): A structured format for interactions between language models, users, and external tools. (一种结构化的格式，用于构建语言模型、用户和外部工具之间的交互。)
Supervised fine-tuning: The technique used to train the model to generate appropriate responses.
Supervised fine-tuning (监督微调): The technique used to train the model to generate appropriate responses. (用于训练模型生成适当响应的技术。)

from transformers import pipeline

# Initialize the pipeline
pipe = pipeline("text-generation", "HuggingFaceTB/SmolLM3-3B", device_map="auto")

# Define your conversation
messages = [
    {"role": "system", "content": "You are a friendly chatbot who always responds in the style of a pirate"},
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]

# Generate response - pipeline handles chat templates automatically
response = pipe(messages, max_new_tokens=128, temperature=0.7)
print(response[0]['generated_text'][-1])  # Print the assistant's response

Output:

{
    'role': 'assistant', 
    'content': "Matey, I'm afraid I must inform ye that humans cannot eat helicopters. Helicopters are not food, they are flying machines. Food is meant to be eaten, like a hearty plate o' grog, a savory bowl o' stew, or a delicious loaf o' bread. But helicopters, they be for transportin' and movin' around, not for eatin'. So, I'd say none, me hearties. None at all."
}

Applies the correct chat template for the model based on the model's tokenizer configuration on the Hugging Face Hub repo.
Applies the correct chat template for the model based on the model's tokenizer configuration on the Hugging Face Hub repo. (根据 Hugging Face Hub 仓库上模型的 tokenizer (分词器) 配置，为模型应用正确的聊天模板。)
Handles tokenization and generation automatically based on the model's tokenizer configuration.
Handles tokenization (分词) and generation automatically based on the model's tokenizer configuration. (根据模型的 tokenizer 配置，自动处理 tokenization 和生成。)
Returns structured output with role information
Returns structured output with role information (返回带有角色信息的结构化输出)
Manages generation parameters and stopping criteria
Manages generation parameters and stopping criteria (管理生成参数和停止条件)

# Configure generation parameters
generation_config = {
    "max_new_tokens": 200,
    "temperature": 0.8,
    "do_sample": True,
    "top_p": 0.9,
    "repetition_penalty": 1.1
}

# Multi-turn conversation
conversation = [
    {"role": "system", "content": "You are a helpful math tutor."},
    {"role": "user", "content": "Can you help me with calculus?"},
]

# Generate first response
response = pipe(conversation, **generation_config)
conversation = response[0]['generated_text']

# Continue the conversation
conversation.append({"role": "user", "content": "What is a derivative?"})
response = pipe(conversation, **generation_config)

print("Final conversation:")
for message in response[0]['generated_text']:
    print(f"{message['role']}: {message['content']}")

<|im_start|>system
You are a helpful assistant focused on technical topics.<|im_end|>
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant

<|im_start|> and <|im_end|>: Special tokens that mark the beginning and end of each message
<|im_start|> and <|im_end|>: Special tokens that mark the beginning and end of each message (<|im_start|> 和 <|im_end|>：标记每条消息开始和结束的特殊 token)
Roles: system, user, assistant (and tool for function calling)
Roles: system, user, assistant (and tool for function calling) (角色：system（系统）, user（用户）, assistant（助手）（以及用于函数调用的 tool（工具））)
Content: The actual message text between the role declaration and <|im_end|>
Content: The actual message text between the role declaration and <|im_end|> (内容：角色声明和 <|im_end|> 之间的实际消息文本)

<|im_start|>user
What is 15 × 24?<|im_end|>
<|im_start|>assistant
15 × 24 = 360<|im_end|>

<|im_start|>user
What is 15 × 24?<|im_end|>
<|im_start|>assistant
<|thinking|>
I need to multiply 15 by 24. Let me break this down:
15 × 24 = 15 × (20 + 4) = (15 × 20) + (15 × 4) = 300 + 60 = 360
</|thinking|>

15 × 24 = 360<|im_end|>

from transformers import AutoTokenizer

# Load SmolLM3's tokenizer
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

# Structure your conversation as a list of message dictionaries
messages = [
    {"role": "system", "content": "You are a helpful assistant focused on technical topics."},
    {"role": "user", "content": "Can you explain what a chat template is?"},
    {"role": "assistant", "content": "A chat template structures conversations between users and AI models by providing a consistent format that helps the model understand different roles and maintain context."}
]

# Apply the chat template
formatted_chat = tokenizer.apply_chat_template(
    messages,
    tokenize=False,  # Return string instead of tokens
    add_generation_prompt=True  # Add prompt for next assistant response
)

print(formatted_chat)

Output:

<|im_start|>system
You are a helpful assistant focused on technical topics.<|im_end|>
<|im_start|>user
Can you explain what a chat template is?<|im_end|>
<|im_start|>assistant
A chat template structures conversations between users and AI models by providing a consistent format that helps the model understand different roles and maintain context.<|im_end|>
<|im_start|>assistant

role: Identifies who is speaking (system, user, assistant, or tool).
role: 标识谁在说话（系统 (system)、用户 (user)、助手 (assistant) 或工具 (tool)）。
content: The actual message content.
content: 实际的消息内容。

System Messages: Set behavior and context for the entire conversation
系统消息 (System Messages)：为整个对话设置行为和上下文。
User Messages: Questions, requests, or statements from the human user
用户消息 (User Messages)：来自人类用户的问题、请求或陈述。
Assistant Messages: Responses from the AI model
助手消息 (Assistant Messages)：来自 AI 模型的响应。
Tool Messages: Results from function calls (for advanced use cases)
工具消息 (Tool Messages)：来自函数调用的结果（用于高级用例）。

# Professional assistant
system_message = {
    "role": "system",
    "content": "You are a professional customer service agent. Always be polite, clear, and helpful."
}

# Technical expert
system_message = {
    "role": "system",
    "content": "You are a senior software engineer. Provide detailed technical explanations with code examples when appropriate."
}

# Creative assistant
system_message = {
    "role": "system",
    "content": "You are a creative writing assistant. Help users craft engaging stories and provide constructive feedback."
}

conversation = [
    {"role": "system", "content": "You are a helpful programming tutor."},
    {"role": "user", "content": "I'm learning Python. Can you explain functions?"},
    {"role": "assistant", "content": "Functions in Python are reusable blocks of code that perform specific tasks. They're defined using the 'def' keyword."},
    {"role": "user", "content": "Can you show me an example?"},
    {"role": "assistant", "content": "Sure! Here's a simple function:\n\n```python\ndef greet(name):\n    return f'Hello, {name}!'\n\nresult = greet('Alice')\nprint(result)  # Output: Hello, Alice!\n```"},
    {"role": "user", "content": "How do I make it return multiple values?"},
]

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

messages = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"},
    {"role": "user", "content": "Can I ask a question?"}
]

# Without generation prompt - for completed conversations
formatted_without = tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=False
)

print("Without generation prompt:")
print(formatted_without)
print("\n" + "="*50 + "\n")

# With generation prompt - for inference
formatted_with = tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

print("With generation prompt:")
print(formatted_with)

Output:

Without generation prompt:
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>

==================================================

With generation prompt:
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant

For inference: Use add_generation_prompt=True when you want the model to generate a response.
对于推理 (inference)：当您希望模型生成响应时，使用 add_generation_prompt=True。
For training: Use add_generation_prompt=False when preparing training data with complete conversations.
对于训练 (training)：当您准备具有完整对话的训练数据时，使用 add_generation_prompt=False。
For evaluation: Use add_generation_prompt=True to test model responses.
对于评估 (evaluation)：使用 add_generation_prompt=True 来测试模型响应。

# Prefill a JSON response
chat = [
    {"role": "user", "content": "Can you format the answer in JSON?"},
    {"role": "assistant", "content": '{"name": "'},
]

# Continue the final message
formatted_chat = tokenizer.apply_chat_template(
    chat, 
    tokenize=False, 
    continue_final_message=True
)

print("Continuing final message:")
print(formatted_chat)
print("\n" + "="*50 + "\n")

# Compare with starting a new message
formatted_new = tokenizer.apply_chat_template(
    chat, 
    tokenize=False,
    add_generation_prompt=True
)

print("Starting new message:")
print(formatted_new)

Continuing final message:
<|im_start|>user
Can you format the answer in JSON?<|im_end|>
<|im_start|>assistant
{"name": "

==================================================

Starting new message:
<|im_start|>user
Can you format the answer in JSON?<|im_end|>
<|im_start|>assistant
{"name": "<|im_end|>
<|im_start|>assistant

1. Structured Output Generation:

# Force the model to complete a specific format
messages = [
    {"role": "system", "content": "You are a helpful assistant that always responds in JSON format."},
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": '{\n  "question": "What\'s the capital of France?",\n  "answer": "'}
]

# The model will continue with just the answer, maintaining JSON structure

2. Code Completion:

# Guide the model to complete code
messages = [
    {"role": "user", "content": "Write a Python function to calculate factorial"},
    {"role": "assistant", "content": "def factorial(n):\n    if n == 0:\n        return 1\n    else:\n        return n * "}
]

# Model will complete the recursive call

3. Step-by-Step Reasoning:

# Guide the model through structured thinking
messages = [
    {"role": "user", "content": "Solve: 2x + 5 = 13"},
    {"role": "assistant", "content": "Let me solve this step by step:\n\nStep 1: "}
]

# Model will continue with the first step

You cannot use add_generation_prompt=True and continue_final_message=True together
您不能同时使用 add_generation_prompt=True 和 continue_final_message=True
The final message must have the "assistant" role when using continue_final_message=True
当使用 continue_final_message=True 时，最后一条消息必须具有 “assistant” 角色。
This feature removes end-of-sequence tokens from the final message
此功能从最后一条消息中删除序列结束 (end-of-sequence) tokens。

# Standard mode - direct answer
standard_messages = [
    {"role": "user", "content": "What is 15 × 24?"},
    {"role": "assistant", "content": "15 × 24 = 360"}
]

# Thinking mode - show reasoning process
thinking_messages = [
    {"role": "user", "content": "What is 15 × 24?"},
    {"role": "assistant", "content": "<|thinking|>\nI need to multiply 15 by 24. Let me break this down:\n15 × 24 = 15 × (20 + 4) = (15 × 20) + (15 × 4) = 300 + 60 = 360\n</|thinking|>\n\n15 × 24 = 360"}
]

# Apply templates
standard_formatted = tokenizer.apply_chat_template(standard_messages, tokenize=False)
thinking_formatted = tokenizer.apply_chat_template(thinking_messages, tokenize=False)

print("Standard mode:")
print(standard_formatted)
print("\nThinking mode:")
print(thinking_formatted)

def create_thinking_example(question, answer, reasoning=None):
    """Create a training example with optional thinking"""
    if reasoning:
        assistant_content = f"<|thinking|>\n{reasoning}\n</|thinking|>\n\n{answer}"
    else:
        assistant_content = answer
    
    return [
        {"role": "user", "content": question},
        {"role": "assistant", "content": assistant_content}
    ]

# Example usage
math_example = create_thinking_example(
    question="What is the derivative of x²?",
    answer="The derivative of x² is 2x",
    reasoning="Using the power rule: d/dx(x^n) = n·x^(n-1)\nFor x²: n=2, so d/dx(x²) = 2·x^(2-1) = 2x"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    },
    {
        "type": "function", 
        "function": {
            "name": "calculate",
            "description": "Perform mathematical calculations",
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "Mathematical expression to evaluate"
                    }
                },
                "required": ["expression"]
            }
        }
    }
]

# Conversation with tool usage
messages = [
    {"role": "system", "content": "You are a helpful assistant with access to tools."},
    {"role": "user", "content": "What's the weather like in Paris?"},
    {
        "role": "assistant", 
        "content": "I'll check the weather in Paris for you.",
        "tool_calls": [
            {
                "id": "call_1",
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "arguments": '{"location": "Paris, France", "unit": "celsius"}'
                }
            }
        ]
    },
    {
        "role": "tool",
        "tool_call_id": "call_1", 
        "content": '{"temperature": 22, "condition": "sunny", "humidity": 60}'
    },
    {
        "role": "assistant",
        "content": "The weather in Paris is currently sunny with a temperature of 22°C and 60% humidity. It's a beautiful day!"
    }
]

# Apply chat template with tools
formatted_with_tools = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=False
)

print("Chat template with tools:")
print(formatted_with_tools)

Chat template with tools:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 01 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful assistant with access to tools.

### Tools

You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:

<tools>
{'type': 'function', 'function': {'name': 'get_weather', 'description': 'Get the current weather for a location', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit'], 'description': 'The temperature unit'}}, 'required': ['location']}}}
{'type': 'function', 'function': {'name': 'calculate', 'description': 'Perform mathematical calculations', 'parameters': {'type': 'object', 'properties': {'expression': {'type': 'string', 'description': 'Mathematical expression to evaluate'}}, 'required': ['expression']}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
...
{"temperature": 22, "condition": "sunny", "humidity": 60}<|im_end|>
<|im_start|>assistant
The weather in Paris is currently sunny with a temperature of 22°C and 60% humidity. It's a beautiful day!<|im_end|>

def format_tool_dataset(examples):
    """Format dataset with tool usage for training"""
    formatted_texts = []
    
    for messages, tools in zip(examples["messages"], examples.get("tools", [None] * len(examples["messages"]))):
        formatted_text = tokenizer.apply_chat_template(
            messages,
            tools=tools,
            tokenize=False,
            add_generation_prompt=False
        )
        formatted_texts.append(formatted_text)
    
    return {"text": formatted_texts}

# View the actual template
print("SmolLM3 Chat Template:")
print(tokenizer.chat_template)

# See what special tokens are used
print("\nSpecial tokens:")
print(f"BOS: {tokenizer.bos_token}")
print(f"EOS: {tokenizer.eos_token}")
print(f"UNK: {tokenizer.unk_token}")
print(f"PAD: {tokenizer.pad_token}")

# Check for custom tokens
special_tokens = tokenizer.special_tokens_map
for name, token in special_tokens.items():
    print(f"{name}: {token}")

# Create a custom template (advanced users only)
custom_template = """
{%- for message in messages %}
    {%- if message['role'] == 'system' %}
        {%- set system_message = message['content'] %}
    {%- endif %}
{%- endfor %}
{%- if system_message is defined %}
<|system|>{{ system_message }}<|end|>
{%- endif %}
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
<|user|>{{ message['content'] }}<|end|>
    {%- elif message['role'] == 'assistant' %}
<|assistant|>{{ message['content'] }}<|end|>
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
<|assistant|>
{%- endif %}
"""

# Apply custom template (be very careful with this!)
# tokenizer.chat_template = custom_template

def debug_chat_template(messages, tokenizer):
    """Debug chat template application"""
    
    # Apply template
    formatted = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    # Tokenize and decode to see actual tokens
    tokens = tokenizer(formatted, return_tensors="pt")
    
    print("=== TEMPLATE DEBUG ===")
    print(f"Input messages: {len(messages)}")
    print(f"Formatted length: {len(formatted)} chars")
    print(f"Token count: {tokens['input_ids'].shape[1]}")
    print("\nFormatted text:")
    print(repr(formatted))  # Shows escape characters
    print("\nTokens:")
    print(tokens['input_ids'][0].tolist()[:20], "...")  # First 20 tokens
    print("\nDecoded tokens:")
    for i, token_id in enumerate(tokens['input_ids'][0][:20]):
        token = tokenizer.decode([token_id])
        print(f"{i:2d}: {token_id:5d} -> {repr(token)}")

# Example usage
debug_messages = [
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there!"}
]

debug_chat_template(debug_messages, tokenizer)

Key Takeaways

Template Consistency: Always use the same template format for training and inference - mismatches can significantly hurt performance
模板一致性：始终对训练和推理使用相同的模板格式 - 不匹配会严重影响性能
Generation Prompts: Use add_generation_prompt=True for inference, False for training data preparation
生成提示：推理时使用 add_generation_prompt=True，训练数据准备时使用 False
Role Structure: Clear role definitions (system, user, assistant, tool) help models understand conversation flow
角色结构：明确的角色定义（系统、用户、助手、工具）有助于模型理解对话流程
Context Management: Leverage SmolLM3's extended context window efficiently by managing conversation history
上下文管理：通过管理对话历史记录，有效地利用 SmolLM3 的扩展上下文窗口
Special Token Handling: Let templates handle special tokens - avoid adding them manually
特殊令牌处理：让模板处理特殊令牌 - 避免手动添加它们

Dual-Mode Reasoning: Use <|thinking|> tags for complex problems requiring step-by-step reasoning
双模式推理：对于需要逐步推理的复杂问题，使用 <|thinking|> 标签
Message Continuation: Use continue_final_message=True for structured output and prefilling responses
消息继续：对于结构化输出和预填充响应，使用 continue_final_message=True
Tool Integration: Modern templates support function calling and tool usage for enhanced capabilities
工具集成：现代模板支持函数调用 (function calling) 和工具使用 (tool usage)，以增强功能
Pipeline Automation: Text generation pipelines handle templates automatically for production use
管道自动化：文本生成管道 (text generation pipelines) 自动处理模板以供生产使用
Multi-Dataset Training: Standardize different dataset formats before combining for training
多数据集训练：在组合训练之前标准化不同的数据集格式

Dataset Preparation: Apply templates with add_generation_prompt=False and add_special_tokens=False for training
数据集准备：对于训练，使用 add_generation_prompt=False 和 add_special_tokens=False 应用模板
Quality Control: Debug templates thoroughly to ensure proper formatting
质量控制：彻底调试模板以确保正确的格式
Performance Monitoring: Incorrect template usage can significantly impact model performance
性能监控：不正确的模板使用会严重影响模型性能
Multimodal Support: Templates extend to vision and audio models with appropriate modifications
多模态支持：通过适当的修改，模板可以扩展到视觉和音频模型

Template mismatch: Using a different template than the model was trained on.
模板不匹配：使用与模型训练时不同的模板。
Double special tokens: Adding special tokens when the template already includes them.
双重特殊令牌：在模板已经包含特殊令牌时添加特殊令牌。
Missing system messages: Not providing enough context for consistent model behavior.
缺少系统消息：没有为一致的模型行为提供足够的上下文。
Inconsistent formatting: Mixing different conversation formats in the same dataset.
格式不一致：在同一数据集中混合不同的对话格式。
Wrong generation prompts: Using incorrect add_generation_prompt settings for your use case.
错误的生成提示：为您的用例使用不正确的 add_generation_prompt 设置。
Ignoring tool syntax: Not properly formatting tool calls and responses.
忽略工具语法：未正确格式化工具调用和响应。
Context overflow: Not managing long conversations within token limits.
上下文溢出：未在令牌限制内管理长对话。

Pipeline usage: Use automated pipelines for consistent template application in production.
管道使用：使用自动化管道 (automated pipelines) 在生产中一致地应用模板。
Error handling: Implement validation for message formats and role sequences.
错误处理：实施消息格式和角色序列的验证 (validation)。
Performance optimization: Cache formatted templates when possible for repeated use.
性能优化：尽可能缓存格式化的模板 (template)，以便重复使用。
Monitoring: Track template application success rates and formatting consistency.
监控：跟踪模板应用成功率和格式一致性。
Version control: Maintain template versions alongside model versions for reproducibility.
版本控制：将模板版本与模型版本一起维护，以实现可重复性 (reproducibility)。

Multimodal templates: Handling images, audio, and video in conversations.
多模态模板 (Multimodal templates)：处理对话中的图像、音频和视频。
Document integration: Including external documents and knowledge bases.
文档集成：包括外部文档和知识库。
Custom template creation: Building specialized templates for domain-specific applications.
自定义模板创建：为特定领域的应用程序构建专用模板。
Template optimization: Performance tuning for high-throughput applications.
模板优化：针对高吞吐量应用程序的性能调优。

Next Steps

Getting Started with Chat Templates - Basic concepts and usage patterns
聊天模板入门 - 基本概念和使用模式
Multimodal Chat Templates - Vision and audio integration
多模态聊天模板 - 视觉和音频集成
Tools and Documents - Function calling and document integration
工具和文档 - 函数调用和文档集成
Advanced Usage - Custom templates and optimization
高级用法 - 自定义模板和优化

SmolLM3 Model Card - Official model documentation and usage examples
SmolLM3 模型卡 - 官方模型文档和使用示例
SmolTalk2 Dataset - Training dataset used for SmolLM3 with template examples
SmolTalk2 数据集 - 用于 SmolLM3 的训练数据集，包含模板示例
TRL Documentation - Training framework with chat template integration
TRL 文档 - 具有聊天模板集成的训练框架

Transformers Chat Templating API - Complete API reference
Transformers 聊天模板 API (Transformers Chat Templating API) - 完整的 API 参考
Jinja2 Template Engine - Template syntax and advanced features
Jinja2 模板引擎 (Jinja2 Template Engine) - 模板语法和高级功能
OpenAI Chat Completions API - Industry standard message format
OpenAI Chat Completions API (OpenAI Chat Completions API) - 行业标准消息格式

Hugging Face Forum - Community discussions and troubleshooting
Hugging Face 论坛 - 社区讨论和故障排除
Discord Server - Real-time help and community interaction
Discord 服务器 - 实时帮助和社区互动
GitHub Issues - Bug reports and feature requests
GitHub Issues - 错误报告和功能请求

Pre-training teaches the model general language understanding (like learning to read).
预训练 (pre-training) 教会模型通用的语言理解（如学习阅读）。
Supervised fine-tuning teaches specific skills and behaviors (like learning to do a specific task).
监督微调 (supervised fine-tuning) 教会特定的技能和行为（如学习执行特定任务）。

Base Model (SmolLM3-3B-Base): Trained on 11T tokens of general text
[N] 基础模型 (SmolLM3-3B-Base)：在 11T 的通用文本上训练
SFT Training: Fine-tuned on curated instruction datasets including SmolTalk2
[N] SFT 训练（SFT Training）：在精选的指令数据集（instruction datasets）上进行微调，包括 SmolTalk2
Preference Alignment: Further refined using techniques like APO (Anchored Preference Optimization)
[N] 偏好对齐（Preference Alignment）：使用诸如 APO (Anchored Preference Optimization, 锚定偏好优化) 等技术进一步优化

Have you tried prompt engineering with existing instruction-tuned models?
[N] 您是否尝试过使用现有的指令调整模型进行提示工程（prompt engineering）？
Do you need consistent output formats that prompting cannot achieve?
[N] 您是否需要提示无法实现的一致的输出格式？
Is your domain specialized enough that general models struggle?
[N] 您的领域是否足够专业以至于通用模型难以胜任？
Do you have high-quality training data (at least 1,000 examples)?
[N] 您是否拥有高质量的训练数据（至少 1,000 个示例）？
Do you have the computational resources for training and evaluation?
[N] 您是否拥有用于训练和评估的计算资源？

SmolTalk2: The dataset used to train SmolLM3, containing high-quality instruction-response pairs.
[N] SmolTalk2：用于训练 SmolLM3 的数据集，包含高质量的指令-响应对。
Domain-specific datasets: For specialized applications (medical, legal, technical).
[N] 领域特定的数据集：用于专门的应用（医疗、法律、技术）。
Custom datasets: Your own curated examples for specific use cases.
[N] 自定义数据集：您自己为特定用例策划的示例。

Input prompt: The user's instruction or question
[N] 输入提示（Input prompt）：用户的指令或问题
Expected response: The ideal assistant response
[N] 预期响应（Expected response）：理想的助手响应
Context (optional): Any additional information needed
[N] 上下文（Context）（可选）：任何需要的附加信息

Local GPU: If you are lucky enough to have a access to a GPU with (at least 16GB of VRAM), you can train your model locally!
[N] 本地 GPU：如果您足够幸运能够访问具有（至少 16GB 显存 (VRAM)）的 GPU，您可以在本地训练您的模型！
Hugging Face Jobs: If you don't have a GPU and don't want to use a cloud provider, you can use Hugging Face Jobs! We'll go into more detail about this in the next section.
[N] Hugging Face Jobs：如果您没有 GPU 并且不想使用云提供商，您可以使用 Hugging Face Jobs！我们将在下一节中详细介绍这一点。
Notebook GPUs: If you like to use a notebook provider like Google Colab, you can use their GPUs!
Notebook GPUs：如果您喜欢使用像 Google Colab 这样的 notebook 提供商，您可以使用他们的 GPU！
Cloud GPU: If you want to take control of your compute resources, you can use a cloud provider like AWS, GCP, or Azure.
Cloud GPU：如果您想控制您的计算资源，您可以使用像 AWS、GCP 或 Azure 这样的云服务提供商。

3. Training Configuration

Start with 5e-5 for SmolLM3; this is conservative and stable.
对于 SmolLM3，从 5e-5 开始；这是一种保守且稳定的方法。
Too high: The model becomes unstable; loss oscillates or explodes.
过高：模型变得不稳定；损失 (loss) 震荡或爆炸。
Too low: The model learns very slowly and may not converge in reasonable time.
过低：模型学习速度非常慢，可能无法在合理的时间内收敛。

Larger batches: More stable gradients, but require more GPU memory.
更大的批次：梯度 (gradients) 更稳定，但需要更多的 GPU 内存。
Smaller batches: Less memory usage, but noisier gradients.
更小的批次：内存使用量更少，但梯度噪声更大。
Use gradient accumulation to achieve larger effective batch sizes.
使用梯度累积 (gradient accumulation) 来实现更大的有效批次大小。

Longer sequences: Can handle more complex conversations.
更长的序列：可以处理更复杂的对话。
Shorter sequences: Faster training, less memory usage.
更短的序列：训练速度更快，内存使用量更少。
Match your use case: Use the typical length of your target conversations.
匹配您的用例：使用您的目标对话的典型长度。

Depends on dataset size: More data usually requires more steps.
取决于数据集大小：更多的数据通常需要更多的步数。
Monitor validation loss: Stop when it stops improving.
监控验证损失 (validation loss)：当它停止改善时停止。
Rule of thumb: Three to five epochs through your dataset.
经验法则：通过数据集进行三到五个 epochs。

Prevents early instability: Helps the model adapt gradually.
防止早期不稳定：帮助模型逐渐适应。
Typical range: 100-500 steps for most SFT tasks.
典型范围：对于大多数 SFT 任务，为 100-500 步。

4. Monitoring and Evaluation

Healthy pattern: Smooth, gradual decrease.
健康模式：平滑、逐渐减少。
Warning signs: Sudden spikes, oscillations, or plateaus.
警告信号：突然的峰值、震荡或停滞。
Typical range: Starts around 2-4, should decrease to 0.5-1.5.
典型范围：从 2-4 左右开始，应降至 0.5-1.5。

Should track training loss: A small gap indicates good generalization.
应跟踪训练损失：一个小差距表明良好的泛化能力。
Growing gap: Sign of overfitting; the model may be memorizing training data.
差距扩大：过度拟合的迹象；模型可能正在记忆训练数据。
Use for early stopping: Stop training when validation loss stops improving.
用于提前停止：当验证损失停止改善时停止训练。

Generate responses: Test the model on held-out prompts during training.
生成响应：在训练期间在保留的提示上测试模型。
Check format consistency: Ensure the model follows desired response patterns.
检查格式一致性：确保模型遵循所需的响应模式。
Monitor for degradation: Watch for repetitive or nonsensical outputs.
监控退化：注意重复或无意义的输出。

Memory spikes: May indicate batch size is too large.
内存峰值：可能表明批量大小（batch size）太大。
Slow training: Could suggest inefficient data loading or processing.
训练缓慢：可能表明数据加载或处理效率低下。

Initial Sharp Drop: Rapid adaptation to new data distribution
初始急剧下降：快速适应新的数据分布
Gradual Stabilization: Learning rate slows as model fine-tunes
逐步稳定：随着模型微调，学习率（Learning rate）减慢
Convergence: Loss values stabilize, indicating training completion
收敛：损失值（Loss values）稳定，表明训练完成

Reducing training steps or epochs
减少训练步数或轮数（epochs）
Increasing dataset size or diversity
增加数据集的大小或多样性
Adding regularization techniques
添加正则化技术（regularization techniques）
Using early stopping based on validation loss
使用基于验证损失（validation loss）的早停法（early stopping）

Learning too slowly (try increasing learning rate)
学习太慢（尝试提高学习率（learning rate））
Struggling with task complexity (check data quality)
难以应对任务的复杂性（检查数据质量）
Hitting architectural limitations (consider different model size)
达到架构限制（考虑不同的模型大小）

Model performs poorly on new, similar examples
模型在新、类似的例子上表现不佳
Outputs lack diversity or creativity
输出缺乏多样性或创造力
Responses are too similar to training examples
响应与训练示例过于相似

Drop-in replacement: API compatible with wandb.init, wandb.log, and wandb.finish.
直接替换：API 与 wandb.init, wandb.log, 和 wandb.finish 兼容。
Local-first design: Dashboard runs locally by default, with optional Hugging Face Spaces hosting.
本地优先设计：仪表板默认在本地运行，可以选择 Hugging Face Spaces 托管。
Free hosting: Everything, including hosting on Hugging Face Spaces, is free.
免费托管：一切都是免费的，包括在 Hugging Face Spaces 上托管。
Lightweight: Fewer than 3,000 lines of Python code, easily extensible.
轻量级：少于 3,000 行 Python 代码，易于扩展。

# Simple Trackio integration
import trackio

# Initialize tracking
trackio.init(project="smollm3-sft")

# Log metrics during training
trackio.log({"train_in_loss": 0.5, "learning_rate": 5e-5})

# Finish tracking
trackio.finish()

export TRACKIO_PROJECT_NAME="my-project"
export TRACKIO_SPACE_ID="username/space_id"

import os

os.environ["TRACKIO_PROJECT_NAME"] = "my-project"
os.environ["TRACKIO_SPACE_ID"] = "username/space_id"

from trl import SFTTrainer

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    args=config,
)

global_step: The total number of optimizer steps taken so far.
global_step: 到目前为止所采取的优化器步骤（optimizer steps）的总数。
epoch: The current epoch number, based on dataset iteration.
epoch: 当前的 epoch 数，基于数据集的迭代。(epoch)
num_tokens: The total number of tokens processed so far.
num_tokens: 到目前为止处理的 token 总数。(tokens)
loss: The average cross-entropy loss computed over non-masked tokens in the current logging interval.
loss: 在当前日志记录间隔中，对非掩码 token 计算的平均交叉熵损失。(cross-entropy loss)
entropy: The average entropy of the model's predicted token distribution over non-masked tokens.
entropy: 模型预测的 token 分布在非掩码 token 上的平均熵。(entropy)
mean_token_accuracy: The proportion of non-masked tokens for which the model’s top-1 prediction matches the ground truth token.
mean_token_accuracy: 模型的前 1 个预测与 ground truth token 匹配的非掩码 token 的比例。(mean_token_accuracy)
learning_rate: The current learning rate, which may change dynamically if a scheduler is used.
learning_rate: 当前的学习率，如果使用调度器，可能会动态变化。(learning rate)
grad_norm: The L2 norm of the gradients, computed before gradient clipping.
grad_norm: 梯度在梯度裁剪之前计算的 L2 范数。(grad_norm)

# Standard language modeling
{"text": "The sky is blue."}

# Conversational language modeling
{"messages": [{"role": "user", "content": "What color is the sky?"},
              {"role": "assistant", "content": "It is blue."}]}

# Standard prompt-completion
{"prompt": "The sky is",
 "completion": " blue."}

# Conversational prompt-completion
{"prompt": [{"role": "user", "content": "What color is the sky?"}],
 "completion": [{"role": "assistant", "content": "It is blue."}]}

from datasets import load_dataset

dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT", "en")

def preprocess_function(example):
    return {
        "prompt": [{"role": "user", "content": example["Question"]}],
        "completion": [
            {"role": "assistant", "content": f"<think>{example['Complex_CoT']}</think>{example['Response']}"}
        ],
    }

dataset = dataset.map(preprocess_function, remove_columns=["Question", "Response", "Complex_CoT"])
print(next(iter(dataset["train"])))

{
    "prompt": [
        {
            "content": "Given the symptoms of sudden weakness in the left arm and leg, recent long-distance travel, and the presence of swollen and tender right lower leg, what specific cardiac abnormality is most likely to be found upon further evaluation that could explain these findings?",
            "role": "user",
        }
    ],
    "completion": [
        {
            "content": "<think>Okay, let's see what's going on here. We've got sudden weakness [...] clicks into place!</think>The specific cardiac abnormality most likely to be found in [...] the presence of a PFO facilitating a paradoxical embolism.",
            "role": "assistant",
        }
    ],
}

Production ready: Used by major organizations and research labs.
Production ready: 被主要组织和研究实验室使用。
Comprehensive: Supports SFT, DPO, ORPO, PPO, and more advanced techniques.
Comprehensive: 支持 SFT、DPO、ORPO、PPO 和更高级的技术。(SFT) (DPO) (ORPO) (PPO)
Efficient: Optimized for memory usage and training speed.
Efficient: 针对内存使用和训练速度进行了优化。
Flexible: Works with any Hugging Face model.
Flexible: 适用于任何 Hugging Face 模型。
CLI support: Command-line tools for scalable training workflows.
CLI support: 用于可扩展训练工作流程的命令行工具。(CLI)

SFTTrainer: The core class for supervised fine-tuning
SFTTrainer: 用于监督微调的核心类。(SFTTrainer)
SFTConfig: Configuration management for training parameters
SFTConfig: 用于训练参数的配置管理。(SFTConfig)
CLI Tools: Command-line interface for production workflows
CLI Tools: 用于生产工作流程的命令行界面。(CLI)
Integration: Seamless integration with Hugging Face Hub, Trackio, Weights & Biases, and more
Integration: 与 Hugging Face Hub、Trackio、Weights & Biases 等无缝集成。

Transformers: Model loading and inference.
Transformers: 模型加载和推理。(Transformers)
Datasets: Data processing and management.
Datasets: 数据处理和管理。(Datasets)
Accelerate: Distributed training and optimization.
Accelerate: 分布式训练和优化。(Accelerate)
PEFT: Parameter-efficient fine-tuning (LoRA, QLoRA).
PEFT: 参数高效微调 (LoRA, QLoRA)。(PEFT) (LoRA) (QLoRA)

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
import trackio as wandb

# Initialize experiment tracking
wandb.init(project="smollm3-sft", name="my-first-sft-run")

# Load SmolLM3 base model
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base")
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base")

# Load SmolTalk2 dataset
dataset = load_dataset("HuggingFaceTB/smoltalk2_everyday_convs_think")

# Configure training with Trackio integration
config = SFTConfig(
    output_dir="./smollm3-finetuned",
    per_device_train_batch_size=4,
    learning_rate=5e-5,
    max_steps=1000,
    report_to="trackio",  # Enable Trackio logging
)

# Train!
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    args=config,
)
trainer.train()

# Fine-tune SmolLM3 using TRL CLI with Trackio tracking
trl sft \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B-Base \
    --dataset_name HuggingFaceTB/smoltalk2_everyday_convs_think \
    --output_dir ./smollm3-sft-model \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-5 \
    --max_steps 1000 \
    --logging_steps 50 \
    --save_steps 200 \
    --report_to trackio \
    --push_to_hub \
    --hub_model_id your-username/smollm3-custom

SFT is Essential: It's the bridge between base models and instruction-following assistants
SFT is Essential (SFT至关重要): It's the bridge between base models and instruction-following assistants
Data Quality Matters: High-quality datasets lead to better fine-tuned models - invest time in curation
Data Quality Matters (数据质量至关重要): High-quality datasets lead to better fine-tuned models - invest time in curation
Monitor Carefully: Watch both loss curves and actual outputs to catch issues early
Monitor Carefully (仔细监控): Watch both loss curves (损失曲线) and actual outputs to catch issues early
TRL Simplifies Everything: From research to production, TRL provides the tools you need
TRL Simplifies Everything (TRL简化一切): From research to production, TRL provides the tools you need
SmolLM3 is Perfect for Learning: Powerful enough to be useful, small enough to be accessible
SmolLM3 is Perfect for Learning (SmolLM3非常适合学习): Powerful enough to be useful, small enough to be accessible
Multiple Approaches: Both programmatic and CLI workflows for different use cases
Multiple Approaches (多种方法): Both programmatic and CLI (命令行界面) workflows for different use cases

Next Steps

Resources and Further Reading

Training with Hugging Face Jobs - Cloud-based training with managed infrastructure
Training with Hugging Face Jobs - Cloud-based training with managed infrastructure
Trackio Documentation - Free, lightweight experiment tracking
Trackio Documentation - Free, lightweight experiment tracking
TRL Documentation - Comprehensive guide to all TRL features
TRL Documentation - Comprehensive guide to all TRL features
SFTTrainer API Reference - Detailed parameter documentation
SFTTrainer API Reference (SFTTrainer API参考) - Detailed parameter documentation
SmolTalk2 Dataset - The dataset that trained SmolLM3
SmolTalk2 Dataset (SmolTalk2数据集) - The dataset that trained SmolLM3
SmolLM3 Model Card - Official model documentation
SmolLM3 Model Card (SmolLM3模型卡片) - Official model documentation
TRL CLI Documentation - Command-line interface guide
TRL CLI Documentation (TRL CLI文档) - Command-line interface guide

Learning Objectives

Master SmolLM3's chat template system
Master SmolLM3's chat template system
Fine-tune SmolLM3 on real datasets using both Python APIs and CLI tools
Fine-tune SmolLM3 on real datasets using both Python APIs and CLI tools (命令行工具)
Work with the SmolTalk2 dataset that was used to train the original model
Work with the SmolTalk2 dataset that was used to train the original model
Compare base model vs fine-tuned model performance
比较基础模型与微调模型的性能
Deploy your models to Hugging Face Hub
Deploy your models to Hugging Face Hub
Understand production workflows for scaling fine-tuning
Understand production workflows for scaling fine-tuning

Environment Setup

[!WARNING]

• You need a GPU with at least 8GB VRAM for training. CPU/MPS can run formatting and dataset exploration, but training larger models will likely fail.
• First run will download several GB of model weights; ensure 15GB+ free disk and a stable connection.
• If you need access to private repos, authenticate with Hugging Face Hub via login().

# Install required packages (run in Colab or your environment)
pip install "transformers>=4.36.0" "trl>=0.7.0" "datasets>=2.14.0" "torch>=2.0.0"
pip install "accelerate>=0.24.0" "peft>=0.7.0" "trackio"

# Import necessary libraries
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from datasets import load_dataset
import json
from typing import Optional, Dict, Any

if torch.cuda.is_available():
    device = "cuda"
    print(f"Using CUDA GPU: {torch.cuda.get_device_name()}")
    print(f"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f}GB")
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    device = "mps"
    print("Using Apple MPS")
else:
    device = "cpu"
    print("Using CPU - you will need to use a GPU to train models")

# Authenticate with Hugging Face (optional, for private models)
from huggingface_hub import login
# login()  # Uncomment if you need to access private models

Using CUDA GPU: NVIDIA A100-SXM4-40GB
GPU memory: 42.5GB

# Load both base and instruct models for comparison
base_model_name = "HuggingFaceTB/SmolLM3-3B-Base"
instruct_model_name = "HuggingFaceTB/SmolLM3-3B"

# Load tokenizers
base_tokenizer = AutoTokenizer.from_pretrained(base_model_name)
instruct_tokenizer = AutoTokenizer.from_pretrained(instruct_model_name)

# Load models (use smaller precision for memory efficiency)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    dtype=torch.bfloat16,
    device_map="auto"
)

instruct_model = AutoModelForCausalLM.from_pretrained(
    instruct_model_name,
    dtype=torch.bfloat16,
    device_map="auto"
)

print("Models loaded successfully!")

tokenizer_config.json: 
 50.4k/? [00:00<00:00, 5.09MB/s]
tokenizer.json: 100%
 17.2M/17.2M [00:02<00:00, 10.7MB/s]
special_tokens_map.json: 100%
 151/151 [00:00<00:00, 21.5kB/s]
tokenizer_config.json: 
 50.4k/? [00:00<00:00, 5.45MB/s]
tokenizer.json: 100%
 17.2M/17.2M [00:00<00:00, 472kB/s]
special_tokens_map.json: 100%
 289/289 [00:00<00:00, 35.0kB/s]
chat_template.jinja: 
 5.60k/? [00:00<00:00, 577kB/s]
config.json: 100%
 943/943 [00:00<00:00, 121kB/s]
model.safetensors.index.json: 
 26.9k/? [00:00<00:00, 2.81MB/s]
Fetching 2 files: 100%
 2/2 [00:32<00:00, 32.11s/it]
model-00001-of-00002.safetensors: 100%
 4.97G/4.97G [00:31<00:00, 247MB/s]
model-00002-of-00002.safetensors: 100%
 1.18G/1.18G [00:17<00:00, 57.2MB/s]
Loading checkpoint shards: 100%
 2/2 [00:01<00:00,  1.18it/s]
generation_config.json: 100%
 126/126 [00:00<00:00, 17.1kB/s]
config.json: 
 1.92k/? [00:00<00:00, 229kB/s]
model.safetensors.index.json: 
 26.9k/? [00:00<00:00, 3.14MB/s]
Fetching 2 files: 100%
 2/2 [00:32<00:00, 32.38s/it]
model-00002-of-00002.safetensors: 100%
 1.18G/1.18G [00:17<00:00, 92.1MB/s]
model-00001-of-00002.safetensors: 100%
 4.97G/4.97G [00:31<00:00, 182MB/s]
Loading checkpoint shards: 100%
 2/2 [00:01<00:00,  1.14it/s]
generation_config.json: 100%
 182/182 [00:00<00:00, 21.0kB/s]
Models loaded successfully!

# Create different types of conversations to test
conversations = {
    "simple_qa": [
        {"role": "user", "content": "What is machine learning?"},
    ],
    
    "with_system": [
        {"role": "system", "content": "You are a helpful AI assistant specialized in explaining technical concepts clearly."},
        {"role": "user", "content": "What is machine learning?"},
    ],
    
    "multi_turn": [
        {"role": "system", "content": "You are a math tutor."},
        {"role": "user", "content": "What is calculus?"},
        {"role": "assistant", "content": "Calculus is a branch of mathematics that deals with rates of change and accumulation of quantities."},
        {"role": "user", "content": "Can you give me a simple example?"},
    ],
    
    "reasoning_task": [
        {"role": "user", "content": "Solve step by step: If a train travels 120 miles in 2 hours, what is its average speed?"},
    ]
}

for conv_type, messages in conversations.items():
    print(f"--- {conv_type.upper()} ---")
    
    # Format without generation prompt (for completed conversations)
    formatted_complete = instruct_tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=False
    )
    
    # Format with generation prompt (for inference)
    formatted_prompt = instruct_tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    print("Complete conversation format:")
    print(formatted_complete)
    print("\nWith generation prompt:")
    print(formatted_prompt)
    print("\n" + "="*50 + "\n")

--- SIMPLE_QA ---
Complete conversation format:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

<|im_start|>user
What is machine learning?<|im_end|>


With generation prompt:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

<|im_start|>user
What is machine learning?<|im_end|>
<|im_start|>assistant
<think>

</think>


==================================================

--- WITH_SYSTEM ---
Complete conversation format:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant specialized in explaining technical concepts clearly.

<|im_start|>user
What is machine learning?<|im_end|>


With generation prompt:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant specialized in explaining technical concepts clearly.

<|im_start|>user
What is machine learning?<|im_end|>
<|im_start|>assistant
<think>

</think>


==================================================

--- MULTI_TURN ---
Complete conversation format:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a math tutor.

<|im_start|>user
What is calculus?<|im_end|>
<|im_start|>assistant
<think>

</think>
Calculus is a branch of mathematics that deals with rates of change and accumulation of quantities.<|im_end|>
<|im_start|>user
Can you give me a simple example?<|im_end|>


With generation prompt:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a math tutor.

<|im_start|>user
What is calculus?<|im_end|>
<|im_start|>assistant
<think>

</think>
Calculus is a branch of mathematics that deals with rates of change and accumulation of quantities.<|im_end|>
<|im_start|>user
Can you give me a simple example?<|im_end|>
<|im_start|>assistant
<think>

</think>


==================================================

--- REASONING_TASK ---
Complete conversation format:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.

<|im_start|>user
Solve step by step: If a train travels 120 miles in 2 hours, what is its average speed?<|im_end|>


With generation prompt:
<|im_start|>system
## Metadata

Knowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.

<|im_start|>user
Solve step by step: If a train travels 120 miles in 2 hours, what is its average speed?<|im_end|>
<|im_start|>assistant


==================================================

# Test the same prompt on both models
test_prompt = "Explain quantum computing in simple terms."

# Prepare the prompt for base model (no chat template)
base_inputs = base_tokenizer(test_prompt, return_tensors="pt").to(device)

# Prepare the prompt for instruct model (with chat template)
instruct_messages = [{"role": "user", "content": test_prompt}]
instruct_formatted = instruct_tokenizer.apply_chat_template(
    instruct_messages, 
    tokenize=False, 
    add_generation_prompt=True
)
instruct_inputs = instruct_tokenizer(instruct_formatted, return_tensors="pt").to(device)

# Generate responses
print("=== Model comparison ===\n")

print("🤖 BASE MODEL RESPONSE:")
with torch.no_grad():
    base_outputs = base_model.generate(
        **base_inputs,
        max_new_tokens=150,
        temperature=0.7,
        do_sample=True,
        pad_token_id=base_tokenizer.eos_token_id
    )
    base_response = base_tokenizer.decode(base_outputs[0], skip_special_tokens=True)
    print(base_response[len(test_prompt):])  # Show only the generated part

print("\n" + "="*50)
print("Instruct model response:")
with torch.no_grad():
    instruct_outputs = instruct_model.generate(
        **instruct_inputs,
        max_new_tokens=150,
        temperature=0.7,
        do_sample=True,
        pad_token_id=instruct_tokenizer.eos_token_id
    )
    instruct_response = instruct_tokenizer.decode(instruct_outputs[0], skip_special_tokens=True)
    # Extract only the assistant's response
    assistant_start = instruct_response.find("<|im_start|>assistant\n") + len("<|im_start|>assistant\n")
    assistant_response = instruct_response[assistant_start:].split("<|im_end|>")[0]
    print(assistant_response)

=== Model comparison ===

🤖 BASE MODEL RESPONSE:
 Why is it thought to be superior to our current technology? How is it superior? What is it's limit?
Quantum computing is based on the fact that in quantum mechanics, a particle can be in multiple states at the same time. This is called superposition. But a single particle can not be in multiple locations at the same time. That is called entanglement. So, how can you have a particle in multiple locations at the same time? Quantum mechanics says that if you measure the location of a particle, it will randomly jump to a particular location. So, if you have 1000 particles, you can have each particle in 1000 different locations at the same time.
This is very useful for solving problems. For example

==================================================
Instruct model response:
nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

user
Explain quantum computing in simple terms.
assistant
<think>

</think>
Quantum computing is a type of computing that uses quantum bits, or qubits, to perform calculations. In traditional computers, we use bits, which can be either 0 or 1. But in quantum computing, we use qubits that can exist in multiple states at once, like both 0 and 1 simultaneously.

Think of it like flipping a coin. A regular coin can land on either heads or tails, but a quantum coin can land on both heads and tails at the same time. This property is called superposition.

Another unique aspect of quantum computing is entanglement. Imagine two coins that are linked together. If one coin lands on heads, the other coin will always land on tails, no matter how far apart they are

# Test SmolLM3's reasoning capabilities
reasoning_prompts = [
    "What is 15 × 24? Show your work.",
    "A recipe calls for 2 cups of flour for 12 cookies. How much flour is needed for 30 cookies?",
    "If I have $50 and spend $18.75 on lunch and $12.30 on a book, how much money do I have left?"
]

print("=== TESTING REASONING CAPABILITIES ===\n")

for i, prompt in enumerate(reasoning_prompts, 1):
    print(f"Problem {i}: {prompt}")
    
    messages = [{"role": "user", "content": prompt}]
    formatted_prompt = instruct_tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = instruct_tokenizer(formatted_prompt, return_tensors="pt").to(device)
    
    with torch.no_grad():
        outputs = instruct_model.generate(
            **inputs,
            max_new_tokens=200,
            temperature=0.3,  # Lower temperature for more consistent reasoning
            do_sample=True,
            pad_token_id=instruct_tokenizer.eos_token_id
        )
        response = instruct_tokenizer.decode(outputs[0], skip_special_tokens=True)
        assistant_start = response.find("<|im_start|>assistant\n") + len("<|im_start|>assistant\n")
        assistant_response = response[assistant_start:].split("<|im_end|>")[0]
        print(f"Answer: {assistant_response}")
    
    print("\n" + "-"*50 + "\n")

=== TESTING REASONING CAPABILITIES ===

Thinking prompt: /no_think
Problem 1: What is 15 × 24? Show your work.
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

user
What is 15 × 24? Show your work.
assistant
<think>

</think>
To find the product of 15 and 24, we can use the standard multiplication algorithm. Here's how we can do it step by step:

First, we multiply 15 by 4 (the units digit of 24):

Next, we multiply 15 by 20 (the tens digit of 24, shifted one place to the left):

Now, we add the two partial products:

So, 15 × 24 = 360.

--------------------------------------------------

Problem 2: A recipe calls for 2 cups of flour for 12 cookies. How much flour is needed for 30 cookies?
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

user
A recipe calls for 2 cups of flour for 12 cookies. How much flour is needed for 30 cookies?
assistant
<think>

</think>
To solve this problem, we need to determine the amount of flour needed per cookie and then multiply that by the number of cookies we want to make.

First, let's find out how much flour is needed per cookie. The recipe calls for 2 cups of flour for 12 cookies. To find the amount of flour per cookie, we divide the total amount of flour by the number of cookies:

2 cups / 12 cookies = 1/6 cup per cookie

Now that we know how much flour is needed per cookie, we can multiply that by the number of cookies we want to make (30):

1/6 cup per cookie * 30 cookies = 5 cups

So, to make 30 cookies, you would need 5 cups of flour.

--------------------------------------------------

Problem 3: If I have $50 and spend $18.75 on lunch and $12.30 on a book, how much money do I have left?
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /no_think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face.

user
If I have $50 and spend $18.75 on lunch and $12.30 on a book, how much money do I have left?
assistant
<think>

</think>
To find out how much money you have left, you need to subtract the total amount spent from your initial amount.

First, calculate the total amount spent on lunch and the book:
$18.75 (lunch) + $12.30 (book) = $31.05

Now, subtract the total amount spent from your initial amount:
$50 (initial amount) - $31.05 (total spent) = $18.95

So, you have $18.95 left.

--------------------------------------------------

Thinking prompt: /think
Problem 1: What is 15 × 24? Show your work.
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.

user
What is 15 × 24? Show your work.
assistant
<think>
Okay, let's see. I need to calculate 15 multiplied by 24. Hmm, how do I do that? I remember there are a few methods. Maybe the standard multiplication algorithm? Or maybe breaking it down into smaller parts. Let me try both ways to make sure I get the right answer.

First, the standard way. Let me write it out like I'm doing long multiplication. So, 15 times 24. I can think of 24 as 20 + 4. So maybe I can break it down into 15 times 20 plus 15 times 4. That might be easier.

Starting with 15 times 20. Well, 15 times 2 is 30, so adding a zero at the end makes it 300. So 15 × 20 = 300. Got that part.

Now, 15 times 4. Let me calculate that. 15 times 4 is 60. Right? Because 10

--------------------------------------------------

Problem 2: A recipe calls for 2 cups of flour for 12 cookies. How much flour is needed for 30 cookies?
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.

user
A recipe calls for 2 cups of flour for 12 cookies. How much flour is needed for 30 cookies?
assistant
<think>
Okay, so I need to figure out how much flour is needed for 30 cookies if the recipe calls for 2 cups of flour for 12 cookies. Hmm, let's see. I think this is a proportion problem. If 12 cookies require 2 cups, then I need to find out how much flour is needed for 30 cookies. 

First, maybe I should determine how much flour is needed per cookie. If 12 cookies take 2 cups, then per cookie it would be 2 divided by 12. Let me write that down: 2 cups / 12 cookies. That simplifies to 1/6 cup per cookie. Wait, 2 divided by 12 is 1/6? Let me check that again. 2 divided by 12 is indeed 1/6. Yeah, because 12 divided by 6 is 2, so 2 divided by 12 is 1/6. So each cookie needs 

--------------------------------------------------

Problem 3: If I have $50 and spend $18.75 on lunch and $12.30 on a book, how much money do I have left?
Answer: nowledge Cutoff Date: June 2025
Today Date: 03 September 2025
Reasoning Mode: /think

## Custom Instructions

You are a helpful AI assistant named SmolLM, trained by Hugging Face. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracking, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> Thought section </think> Solution section. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion.

user
If I have $50 and spend $18.75 on lunch and $12.30 on a book, how much money do I have left?
assistant
<think>
Okay, let's see. The problem is about calculating how much money is left after spending on lunch and a book. I need to start with the initial amount of $50 and then subtract the amounts spent on lunch and the book.

First, I should add up the total amount spent. The lunch cost $18.75 and the book cost $12.30. To add these two amounts together, I need to make sure they are in the same units, which they already are since both are in dollars. So, adding $18.75 and $12.30. Let me do that step by step.

Starting with the dollars: 18 dollars plus 12 dollars is 30 dollars. Then the cents: 75 cents plus 30 cents is 105 cents. Now, 105 cents is equal to $1.05 because 100 cents make a dollar, so 105 cents is 1 dollar and 5 cents. Therefore, adding the dollars and cents together

--------------------------------------------------

Different chat template formats for various conversation types
各种对话类型的不同聊天模板格式 (chat template formats)
Clear differences between base model and instruct model responses
基础模型 (base model) 和指导模型 (instruct model) 响应之间的明显差异
SmolLM3's reasoning capabilities in action
SmolLM3 的推理能力正在发挥作用

# Load and explore the SmolTalk2 dataset
print("=== EXPLORING SMOLTALK2 DATASET ===\n")

# Load the SFT subset
dataset_dict = load_dataset("HuggingFaceTB/smoltalk2", "SFT")
print(f"Total splits: {len(dataset_dict)}")
print(f"Available splits: {list(dataset_dict.keys())}")
print(f"Number of total rows: {sum([dataset_dict[d].num_rows for d in dataset_dict])}")
print(f"Dataset structure: {dataset_dict}")

=== EXPLORING SMOLTALK2 DATASET ===

Resolving data files: 100%
 124/124 [00:00<00:00, 9963.48it/s]
Resolving data files: 100%
 113/113 [00:00<00:00, 57.54it/s]
Resolving data files: 100%
 113/113 [00:00<00:00, 114.07it/s]
Loading dataset shards: 100%
 105/105 [00:00<00:00, 2570.62it/s]
Total splits: 25
Available splits: ['LongAlign_64k_Qwen3_32B_yarn_131k_think', 'OpenThoughts3_1.2M_think', 'aya_dataset_Qwen3_32B_think', 'multi_turn_reasoning_if_think', 's1k_1.1_think', 'smolagents_toolcalling_traces_think', 'smoltalk_everyday_convs_reasoning_Qwen3_32B_think', 'smoltalk_multilingual8_Qwen3_32B_think', 'smoltalk_systemchats_Qwen3_32B_think', 'table_gpt_Qwen3_32B_think', 'LongAlign_64k_context_lang_annotated_lang_6_no_think', 'Mixture_of_Thoughts_science_no_think', 'OpenHermes_2.5_no_think', 'OpenThoughts3_1.2M_no_think_no_think', 'hermes_function_calling_v1_no_think', 'smoltalk_multilingual_8languages_lang_5_no_think', 'smoltalk_smollm3_everyday_conversations_no_think', 'smoltalk_smollm3_explore_instruct_rewriting_no_think', 'smoltalk_smollm3_smol_magpie_ultra_no_think', 'smoltalk_smollm3_smol_rewrite_no_think', 'smoltalk_smollm3_smol_summarize_no_think', 'smoltalk_smollm3_systemchats_30k_no_think', 'table_gpt_no_think', 'tulu_3_sft_personas_instruction_following_no_think', 'xlam_traces_no_think']
Number of total rows: 3383242
Dataset structure: DatasetDict({
    LongAlign_64k_Qwen3_32B_yarn_131k_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 7526
    })
    OpenThoughts3_1.2M_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 1133524
    })
    aya_dataset_Qwen3_32B_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 15222
    })
    multi_turn_reasoning_if_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 28217
    })
    s1k_1.1_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 835
    })
    smolagents_toolcalling_traces_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 9079
    })
    smoltalk_everyday_convs_reasoning_Qwen3_32B_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 2057
    })
    smoltalk_multilingual8_Qwen3_32B_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 244736
    })
    smoltalk_systemchats_Qwen3_32B_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 27436
    })
    table_gpt_Qwen3_32B_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 13201
    })
    LongAlign_64k_context_lang_annotated_lang_6_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 6249
    })
    Mixture_of_Thoughts_science_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 86110
    })
    OpenHermes_2.5_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 384900
    })
    OpenThoughts3_1.2M_no_think_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 435193
    })
    hermes_function_calling_v1_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 8961
    })
    smoltalk_multilingual_8languages_lang_5_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 254047
    })
    smoltalk_smollm3_everyday_conversations_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 2260
    })
    smoltalk_smollm3_explore_instruct_rewriting_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 30391
    })
    smoltalk_smollm3_smol_magpie_ultra_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 406843
    })
    smoltalk_smollm3_smol_rewrite_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 53262
    })
    smoltalk_smollm3_smol_summarize_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 96061
    })
    smoltalk_smollm3_systemchats_30k_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 33997
    })
    table_gpt_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 13203
    })
    tulu_3_sft_personas_instruction_following_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 29970
    })
    xlam_traces_no_think: Dataset({
        features: ['messages', 'chat_template_kwargs', 'source'],
        num_rows: 59962
    })
})

# Function to process different dataset formats
def process_qa_dataset(examples, question_col, answer_col):
    """Process Q&A datasets into chat format"""
    processed = []
    
    for question, answer in zip(examples[question_col], examples[answer_col]):
        messages = [
            {"role": "user", "content": question},
            {"role": "assistant", "content": answer}
        ]
        processed.append(messages)
    
    return {"messages": processed}

def process_instruction_dataset(examples):
    """Process instruction-following datasets"""
    processed = []
    
    for instruction, response in zip(examples["instruction"], examples["response"]):
        messages = [
            {"role": "user", "content": instruction},
            {"role": "assistant", "content": response}
        ]
        processed.append(messages)
    
    return {"messages": processed}

# Example: Process GSM8K math dataset
print("=== PROCESSING GSM8K DATASET ===\n")

gsm8k = load_dataset("openai/gsm8k", "main", split="train[:100]")  # Small subset for demo
print(f"Original GSM8K example: {gsm8k[0]}")

# Convert to chat format
def process_gsm8k(examples):
    processed = []
    for question, answer in zip(examples["question"], examples["answer"]):
        messages = [
            {"role": "system", "content": "You are a math tutor. Solve problems step by step."},
            {"role": "user", "content": question},
            {"role": "assistant", "content": answer}
        ]
        processed.append(messages)
    return {"messages": processed}

gsm8k_processed = gsm8k.map(process_gsm8k, batched=True, remove_columns=gsm8k.column_names)
print(f"Processed example: {gsm8k_processed[0]}")

=== PROCESSING GSM8K DATASET ===

README.md: 
 7.94k/? [00:00<00:00, 572kB/s]
main/train-00000-of-00001.parquet: 100%
 2.31M/2.31M [00:01<00:00, 42.6kB/s]
main/test-00000-of-00001.parquet: 100%
 419k/419k [00:00<00:00, 813kB/s]
Generating train split: 100%
 7473/7473 [00:00<00:00, 321312.49 examples/s]
Generating test split: 100%
 1319/1319 [00:00<00:00, 97120.71 examples/s]
Original GSM8K example: {'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72'}
Map: 100%
 100/100 [00:00<00:00, 4792.50 examples/s]
Processed example: {'messages': [{'content': 'You are a math tutor. Solve problems step by step.', 'role': 'system'}, {'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'role': 'user'}, {'content': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'role': 'assistant'}]}

# Function to apply chat templates to processed datasets
def apply_chat_template_to_dataset(dataset, tokenizer):
    """Apply chat template to dataset for training"""
    
    def format_messages(examples):
        formatted_texts = []
        
        for messages in examples["messages"]:
            # Apply chat template
            formatted_text = tokenizer.apply_chat_template(
                messages,
                tokenize=False,
                add_generation_prompt=False  # We want the complete conversation
            )
            formatted_texts.append(formatted_text)
        
        return {"text": formatted_texts}
    
    return dataset.map(format_messages, batched=True)

# Apply to our processed GSM8K dataset
gsm8k_formatted = apply_chat_template_to_dataset(gsm8k_processed, instruct_tokenizer)
print("=== FORMATTED TRAINING DATA ===")
print(gsm8k_formatted[0]["text"])

=== PROCESSING GSM8K DATASET ===

README.md: 
 7.94k/? [00:00<00:00, 572kB/s]
main/train-00000-of-00001.parquet: 100%
 2.31M/2.31M [00:01<00:00, 42.6kB/s]
main/test-00000-of-00001.parquet: 100%
 419k/419k [00:00<00:00, 813kB/s]
Generating train split: 100%
 7473/7473 [00:00<00:00, 321312.49 examples/s]
Generating test split: 100%
 1319/1319 [00:00<00:00, 97120.71 examples/s]
Original GSM8K example: {'question': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'answer': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72'}
Map: 100%
 100/100 [00:00<00:00, 4792.50 examples/s]
Processed example: {'messages': [{'content': 'You are a math tutor. Solve problems step by step.', 'role': 'system'}, {'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'role': 'user'}, {'content': 'Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips altogether in April and May.\n#### 72', 'role': 'assistant'}]}

# Import required libraries for fine-tuning
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import SFTTrainer, SFTConfig, DataCollatorForCompletionOnlyLM
from datasets import load_dataset
import torch
import wandb  # Optional: for experiment tracking

# Initialize Weights & Biases (optional)
# wandb.init(project="smollm3-finetuning")

# Load SmolLM3 base model for fine-tuning
model_name = "HuggingFaceTB/SmolLM3-3B-Base"
new_model_name = "SmolLM3-Custom-SFT"

print(f"Loading {model_name}...")
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token  # Set padding token
tokenizer.padding_side = "right"  # Padding on the right for generation

print(f"Model loaded! Parameters: {model.num_parameters():,}")

Loading HuggingFaceTB/SmolLM3-3B-Base...
Model loaded! Parameters: 3,075,098,624

# Load and prepare training dataset
print("=== PREPARING DATASET ===\n")

# Option 1: Use SmolTalk2 (recommended for beginners)
dataset = load_dataset("HuggingFaceTB/smoltalk2", "SFT")
train_dataset = dataset["smoltalk_everyday_convs_reasoning_Qwen3_32B_think"].select(range(1000))  # Use subset for faster training

# Option 2: Use your own processed dataset from Exercise 2
# train_dataset = gsm8k_formatted.select(range(500))

print(f"Training examples: {len(train_dataset)}")
print(f"Example: {train_dataset[0]}")

# Prepare the dataset for SFT
def format_chat_template(example):
    """Format the messages using the chat template"""
    if "messages" in example:
        # SmolTalk2 format
        messages = example["messages"]
    else:
        # Custom format - adapt as needed
        messages = [
            {"role": "user", "content": example["instruction"]},
            {"role": "assistant", "content": example["response"]}
        ]
    
    # Apply chat template
    text = instruct_tokenizer.apply_chat_template(
        messages, 
        tokenize=False,
        add_generation_prompt=False
    )
    return {"text": text}

# Apply formatting
formatted_dataset = train_dataset.map(format_chat_template)
formatted_dataset = formatted_dataset.remove_columns(
    [col for col in formatted_dataset.column_names if col != "text"]
)
print(f"Formatted example: {formatted_dataset[0]['text'][:200]}...")

=== PREPARING DATASET ===
Training examples: 1000
Example: {'messages': [{'content': 'Solve the problem step by step.', 'role': 'system'}, {'content': 'Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?', 'role': 'user'}]}
Formatted example: You are a math tutor. Solve problems step by step. Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?...

# Configure training parameters
training_config = SFTConfig(
    # Model and data
    output_dir=f"./{new_model_name}",
    dataset_text_field="text",
    max_length=2048,
    
    # Training hyperparameters
    per_device_train_batch_size=2,  # Adjust based on your GPU memory
    gradient_accumulation_steps=2,
    learning_rate=5e-5,
    num_train_epochs=1,  # Start with 1 epoch
    max_steps=500,  # Limit steps for demo
    
    # Optimization
    warmup_steps=50,
    weight_decay=0.01,
    optim="adamw_torch",
    
    # Logging and saving
    logging_steps=10,
    save_steps=100,
    eval_steps=100,
    save_total_limit=2,
    
    # Memory optimization
    dataloader_num_workers=0,
    group_by_length=True,  # Group similar length sequences
    
    # Hugging Face Hub integration
    push_to_hub=False,  # Set to True to upload to Hub
    hub_model_id=f"your-username/{new_model_name}",
    
    # Experiment tracking
    report_to=["trackio"],  # Use trackio for experiment tracking
    run_name=f"{new_model_name}-training",
)

print("Training configuration set!")
print(f"Effective batch size: {training_config.per_device_train_batch_size * training_config.gradient_accumulation_steps}")

Training configuration set!
Effective batch size: 4

# LoRA configuration with PEFT
from peft import LoraConfig

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# Create SFTTrainer with LoRA enabled
from trl import SFTTrainer

lora_trainer = SFTTrainer(
    model=model,
    train_dataset=formatted_dataset,  # dataset with a "text" field or messages + dataset_text_field in config
    args=training_config,
    peft_config=peft_config,  # << enable LoRA
)

print("Starting LoRA training…")
lora_trainer.train()

trainer = SFTTrainer(
    model=model,
    train_dataset=formatted_dataset,
    args=config,
)

trainer.train()

# Test the fine-tuned model
print("=== AFTER TRAINING ===")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(response[len(formatted_prompt):])

# Optional: Push to Hugging Face Hub
if training_config.push_to_hub:
    trainer.push_to_hub(
        commit_message="Fine-tuned SmolLM3 with custom dataset",
        tags=["smol-course", "sft", "instruction-tuning"]
    )
    print(f"Model pushed to Hub: {training_config.hub_model_id}")

# Fine-tune SmolLM3 using TRL CLI
trl sft \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B-Base \
    --dataset_name HuggingFaceTB/smoltalk2_everyday_convs_think \
    --output_dir ./smollm3-sft-cli \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 2 \
    --learning_rate 5e-5 \
    --num_train_epochs 1 \
    --max_length 2048 \
    --logging_steps 10 \
    --save_steps 500 \
    --warmup_steps 100 \
    --bf16 True \
    --push_to_hub \
    --hub_model_id your-username/smollm3-sft-cli

# Model and dataset
model_name_or_path: HuggingFaceTB/SmolLM3-3B-Base
dataset_name: HuggingFaceTB/smoltalk2_everyday_convs_think
output_dir: ./smollm3-advanced-sft

# Training hyperparameters
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 3e-5
num_train_epochs: 2
max_length: 4096

# Optimization
warmup_steps: 200
weight_decay: 0.01
optim: adamw_torch
lr_scheduler_type: cosine

# Memory and performance
bf16: true
dataloader_num_workers: 4
group_by_length: true
remove_unused_columns: false

# Logging and evaluation
logging_steps: 25
eval_steps: 250
save_steps: 500
eval_strategy: steps
load_best_model_at_end: true
metric_for_best_model: eval_loss

# Hub integration
push_to_hub: true
hub_model_id: your-username/smollm3-advanced
hub_strategy: every_save

# Run training with config file
trl sft --config sft_config.yaml

Reduce per_device_train_batch_size to 1
将 per_device_train_batch_size 减小到 1
Reduce max_length to 1024 or 512
将 max_length 减小到 1024 或 512
Use torch.cuda.empty_cache() to clear GPU memory
使用 torch.cuda.empty_cache() 清理 GPU 内存

Check your internet connection
检查您的互联网连接
Try using device_map="cpu" for CPU loading
尝试使用 device_map="cpu" 进行 CPU 加载
Use a smaller model like HuggingFaceTB/SmolLM3-1.7B for testing
使用较小的模型，例如 HuggingFaceTB/SmolLM3-1.7B 进行测试

Make sure your dataset is properly formatted
确保您的数据集格式正确
Check that all examples have reasonable length (not too long)
检查所有示例是否具有合理的长度（不要太长）
Monitor the training loss - it should decrease steadily
监控训练损失（training loss）- 它应该稳定下降

SmolLM3's chat template system and dual-mode reasoning
SmolLM3 的聊天模板系统和双模式推理
Dataset processing and preparation techniques
数据集处理和准备技术
Supervised fine-tuning with Python APIs
使用 Python API 的监督式微调（Supervised fine-tuning）
Production workflows using CLI tools
使用 CLI 工具的生产工作流程（Production workflows）
Distributed training setups
分布式训练设置

Resources for Further Learning

TRL Documentation - Complete reference
TRL 文档 - 完整参考
SmolLM3 Model Card - Model details
SmolLM3 模型卡片 - 模型详情
SmolTalk2 Dataset - Training data
SmolTalk2 数据集 - 训练数据
Hugging Face Hub - Share your models
Hugging Face Hub - 分享您的模型
Discord Community - Get help and discuss
Discord 社区 - 获取帮助并讨论

Scalable Infrastructure: Access to high-end GPUs (A100, L4, etc.) without hardware investment
可扩展的基础设施：无需硬件投资即可访问高端 GPU（A100、L4 等）
Zero Setup: No need to manage CUDA drivers, Docker containers, or environment configurations
零设置：无需管理 CUDA 驱动程序、Docker 容器或环境配置
Cost Effective: Pay only for compute time used, with automatic shutdown after completion
经济高效：只需为使用的计算时间付费，并在完成后自动关闭
Integrated Workflow: Seamless integration with Hugging Face Hub for model storage and sharing
集成工作流程：与 Hugging Face Hub 无缝集成，用于模型存储和共享
Monitoring: Built-in logging and progress tracking through the Hub interface
监控：通过 Hub 界面内置日志记录和进度跟踪

A Pro, Team, or Enterprise Hugging Face plan which you can get here
一个 Pro、Team 或 Enterprise Hugging Face 计划，您可以在这里获得
Authentication via hf auth login
通过 hf auth login 进行身份验证

# sft_training.py
# /// script
# dependencies = [
#     "trl[sft]>=0.7.0",
#     "transformers>=4.36.0", 
#     "datasets>=2.14.0",
#     "accelerate>=0.24.0",
#     "peft>=0.7.0"
# ]
# ///

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base")
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base")

# Load dataset
dataset = load_dataset("HuggingFaceTB/smoltalk2", "SFT")

# Configure training
config = SFTConfig(
    output_dir="./smollm3-jobs-sft",
    per_device_train_batch_size=4,
    learning_rate=5e-5,
    max_steps=1000,
    logging_steps=50,
    save_steps=200,
    push_to_hub=True,
    hub_model_id="your-username/smollm3-jobs-sft"
)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["smoltalk_everyday_convs_reasoning_Qwen3_32B_think"],
    args=config,
)
trainer.train()

# Run the UV script on Jobs
hf jobs uv run \
    --flavor a10g-large \
    --timeout 2h \
    --secrets HF_TOKEN \
    sft_training.py

a10g-large: 24GB GPU memory, cost-effective for most SFT tasks
a10g-large：24GB GPU 内存，对于大多数 SFT 任务来说经济高效
a100-large: 40GB GPU memory, fastest training with larger batch sizes
a100-large：40GB GPU 内存，使用更大的批量大小进行最快的训练
l4x1: 24GB GPU memory, multi-GPU setup for distributed training
l4x1：24GB GPU 内存，用于分布式训练的多 GPU 设置

a100-large: Required for 7B+ models
a100-large：7B+ 模型必需
l4x4: Multi-GPU setup for distributed training
l4x4：用于分布式训练的多 GPU 设置

t4-small: 16GB GPU memory, slower but economical for experimentation
t4-small：16GB GPU 内存，速度较慢，但对于实验来说经济
l4x1: 24GB GPU memory, good balance of cost and performance
l4x1：24GB GPU 内存，成本和性能之间的良好平衡

# Use TRL's maintained SFT script directly
hf jobs uv run \
    --flavor a10g-large \
    --timeout 2h \
    --secrets HF_TOKEN \
    "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B-Base \
    --dataset_name HuggingFaceTB/smoltalk2_everyday_convs_think \
    --learning_rate 5e-5 \
    --per_device_train_batch_size 4 \
    --max_steps 1000 \
    --output_dir smollm3-sft-jobs \
    --push_to_hub \
    --hub_model_id your-username/smollm3-sft \
    --report_to trackio

hf jobs uv run \
    --flavor a10g-large \
    --timeout 3h \
    --secrets HF_TOKEN=your_token \
    --secrets WANDB_API_KEY=your_wandb_key \
    --env WANDB_PROJECT=smollm3-sft \
    --env CUDA_VISIBLE_DEVICES=0 \
    my_sft_training.py

# List all jobs
hf jobs ps -a

# Get detailed job information  
hf jobs inspect <job_id>

# Stream job logs in real-time
hf jobs logs <job_id> --follow

# Cancel a running job if needed
hf jobs cancel <job_id>

hf jobs uv run \
  --flavor a10g-large \
  --timeout 2h \
  --secrets HF_TOKEN \
  "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" \
  --model_name_or_path HuggingFaceTB/SmolLM3-3B-Base \
  --dataset_name HuggingFaceTB/smoltalk2_everyday_convs_think \
  --output_dir smollm3-lora-sft-jobs \
  --per_device_train_batch_size 4 \
  --learning_rate 5e-5 \
  --max_steps 1000 \
  --report_to trackio \
  --push_to_hub \
  --hub_model_id your-username/smollm3-lora-sft \
  --use_peft \
  --lora_r 8 \
  --lora_alpha 16 \
  --lora_dropout 0.05 \
  --lora_target_modules all-linear

Confirm flag names in the TRL SFT script before running: https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py.
运行前请在 TRL SFT 脚本中确认标志名称：https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py。
LoRA trains small adapters, which you can keep separate or merge later for deployment.
LoRA (Low-Rank Adaptation) 训练小型适配器，您可以将其保持分离状态，或稍后合并以进行部署。

l4x1: ~$3-4 per hour (24GB GPU memory)
l4x1：每小时约 3-4 美元（24GB GPU 内存）
a10g-large: ~$4-6 per hour (24GB GPU memory)
a10g-large：每小时约 4-6 美元（24GB GPU 内存）
a100-large: ~$8-12 per hour (40GB GPU memory)
a100-large：每小时约 8-12 美元（40GB GPU 内存）

Use smaller batch sizes with gradient accumulation to fit on cheaper GPUs
使用较小的批量大小和梯度累积 (gradient accumulation) 以适应更便宜的 GPU
Start with shorter training runs (500 steps) to validate your setup
首先进行较短的训练（500 步）以验证您的设置
Use l4x1 for initial experiments, then scale to faster GPUs for production
初始实验使用 l4x1，然后扩展到更快的 GPU 进行生产
Set appropriate timeouts to avoid unexpected charges
设置适当的超时时间以避免意外收费

Reduce per_device_train_batch_size
减少 per_device_train_batch_size
Enable gradient checkpointing
启用梯度检查点 (gradient checkpointing)
Use smaller max_length
使用较小的 max_length

Increase timeout parameter
增加 timeout 参数
Reduce training steps or use more powerful hardware
减少训练步骤或使用更强大的硬件
Optimize data loading and preprocessing
优化数据加载和预处理

Ensure HF_TOKEN is correctly set as a secret
确保 HF_TOKEN 已正确设置为 secret
Verify your Hugging Face account has the required plan
验证您的 Hugging Face 帐户是否具有所需的计划
Check token permissions for model uploads
检查模型上传的令牌权限

Resources and Further Reading

Hugging Face Jobs Documentation - Complete Jobs guide
Hugging Face Jobs 文档 - 完整的 Jobs 指南
TRL Jobs Training Guide - TRL-specific Jobs examples
TRL Jobs 训练指南 - TRL 相关的 Jobs 示例
Jobs Pricing - Current pricing for different hardware flavors
Jobs 定价 - 不同硬件版本的当前定价
Jobs CLI Reference - Command-line interface details
Jobs CLI 参考 - 命令行界面详细信息

Submit your final project!

1. Read the written guide for the chapter ✅
2. Train a model using what you learned in the chapter.
3. Push the model to the Hugging Face Hub.
4. Evaluate the model using hf jobs.
5. Open a pull request on the leaderboard.

On this page we will go through each step.

1. Read the written guide for the chapter and 2. Train a model using what you learned in the chapter.

3. Push the model to the Hugging Face Hub

hf jobs uv run \
    --flavor a100-large \
    --secrets HF_TOKEN \
    "trl/scripts/sft.py" \
    ...
    --push_to_hub # this will push the model to the Hugging Face Hub

Your trained model will be available at your-username/your-model-name. For detailed documentation, check out the checkpoints documentation from transformers.

4. Evaluate the model using hf jobs

hf jobs uv run \ # run a hf jobs job with uv
--flavor a10g-large \ # select the machine size
--with "lighteval[vllm]" \ # install lighteval with vllm dependencies
s HF_TOKEN \ # share the huggingface write token
lighteval vllm "model_name=<your-username>/<your-model-name>" "lighteval|gsm8k|0|0" --push-to-hub --results-org <your-username>

5. Open a pull request on the leaderboard space

add your model's results to submissions.json
将你的模型的结果添加到 submissions.json
share you evaluation command (using hf jobs) in the PR text.
在 PR 文本中分享你的评估命令（使用 hf jobs）。

Add your model's results to submissions.json

{
    "submissions": [

        ... # existing submissions
        
        {
            "username": "<your-username>",
            "model_name": "<your-model-name>",
            "chapter": "1",
            "submission_date": "<your-submission-date>",
            "results-dataset": "<your-results-dataset>"
        }
    ]
}

hf jobs uv run ...

Wait for the PR to be merged

Test your knowledge

Adapt the base model to follow instructions through supervised fine-tuning.
通过监督式微调 (supervised fine-tuning) 调整基础模型以遵循指令。
Directly optimize the model using preference data through Direct Preference Optimization.
通过直接偏好优化 (Direct Preference Optimization, DPO) 使用偏好数据直接优化模型。

What You'll Build

You'll master DPO hyperparameter configuration and tuning techniques.
你将掌握 DPO 超参数配置和调整技术。
You'll compare DPO results with baseline instruction-tuned models.
你将比较 DPO 结果与基线指令调整模型。
You'll evaluate model safety and alignment quality using standard benchmarks.
你将使用标准基准评估模型的安全性和对齐质量。
You'll submit your aligned model to the course leaderboard.
你会将对齐的模型提交到课程排行榜 (leaderboard)。
Finally, you'll explore how to deploy aligned models for practical applications.
最后，您将探索如何部署对齐的模型以用于实际应用。

Training a reward model to predict human preferences based on preferred and rejected responses.
训练一个奖励模型，以根据首选和拒绝的响应来预测人类偏好。
Using reinforcement learning algorithms like PPO to optimize the policy against the reward model.
使用诸如 PPO 之类的强化学习算法来根据奖励模型优化策略。

First, training a SFT model to follow instructions.
首先，训练一个 SFT 模型以遵循指令。
Then, training a DPO model to directly optimize the language model using preference data itself.
然后，训练一个 DPO 模型以使用偏好数据本身直接优化语言模型。

π_θ is the model being trained
π_θ 是正在训练的模型
π_ref is the reference model (usually the SFT model)
π_ref 是参考模型（通常是 SFT 模型）
y_w is the preferred (winning) response
y_w 是首选（获胜）响应
y_l is the rejected (losing) response
y_l 是拒绝（失败）的响应
β is a temperature parameter controlling optimization strength
β 是控制优化强度的温度参数
σ is the sigmoid function
σ 是 sigmoid 函数

from trl import DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM3-3B")
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

# Configure DPO training
training_args = DPOConfig(
    beta=0.1,                    # Temperature parameter
    learning_rate=5e-7,          # Lower LR for stability
    max_prompt_length=512,       # Maximum prompt length
    max_length=1024,             # Maximum total length
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
)

# Initialize trainer
trainer = DPOTrainer(
    model=model,
    args=training_args,
    train_dataset=preference_dataset,
    processing_class=tokenizer,
)

# Train the model
trainer.train()

Training SmolLM3 with your preference data
使用您的偏好数据训练SmolLM3 (Training SmolLM3 with your preference data)
Evaluating alignment quality and model performance
评估对齐质量和模型性能 (Evaluating alignment quality and model performance)
Deploying your aligned model
部署您已对齐的模型 (Deploying your aligned model)

Environment Setup

# Install required packages
pip install "transformers>=4.56.1" "trl>=0.23.0" "datasets>=4.1.0" "torch>=2.8.0"
pip install "accelerate>=1.10.1" "peft>=0.17.0" "trackio"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
from trl import DPOTrainer, DPOConfig
import json

# Check available device
if torch.cuda.is_available():
    device = "cuda"
    print(f"Using CUDA GPU: {torch.cuda.get_device_name()}")
    print(f"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f}GB")
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
    device = "mps"
    print("Using Apple MPS")
else:
    device = "cpu"
    print("Using CPU - you will need HF Jobs with GPU for actual training")

# Authenticate with Hugging Face
from huggingface_hub import login
login()  # Required for HF Jobs and model uploads

# Load a preference dataset to understand the format
dataset = load_dataset("Anthropic/hh-rlhf", split="train")
print(f"Dataset size: {len(dataset)}")
print("Dataset features:", dataset.features.keys())

# Examine a preference pair
sample = dataset[0]
print(f"\nChosen (Preferred): {sample['chosen'][:200]}...")
print(f"\nRejected (Non-preferred): {sample['rejected'][:200]}...")

# This shows how DPO learns to prefer "chosen" responses over "rejected" ones

Dataset size: 112000
Dataset features: ['prompt', 'chosen', 'rejected']

=== DPO Preference Pair Example ===
Prompt: Human: What are some cuss words in english?

Chosen (Preferred): Assistant: I understand you're curious about profanity in English. While I can acknowledge that such words exist, I'd prefer not to list them explicitly. These are generally considered offensive or inappropriate in most contexts...

Rejected (Non-preferred): Assistant: Here are some common cuss words in English: [explicit list follows]...

# Load a small subset for local testing
small_dataset = dataset.select(range(1000))

# Load SmolLM3-3B-Instruct model
model_name = "HuggingFaceTB/SmolLM3-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Configure DPO training for local testing
training_args = DPOConfig(
    beta=0.1,                           # Preference optimization strength
    learning_rate=5e-7,                 # Lower than SFT
    per_device_train_batch_size=1,      # Small batch for local testing
    gradient_accumulation_steps=4,      # Effective batch size = 4
    max_steps=50,                       # Very short for testing
    logging_steps=10,
    output_dir="./local_dpo_test",
    report_to="trackio",
)

# Create trainer (but don't train yet - save resources for HF Jobs)
trainer = DPOTrainer(
    model=model,
    args=training_args,
    train_dataset=small_dataset,
    processing_class=tokenizer,
)

print("Local DPO trainer configured successfully!")
print("Ready to scale to HF Jobs for full training...")

# dpo_training.py
# /// script
# dependencies = [
#     "trl[dpo]>=0.7.0",
#     "transformers>=4.36.0", 
#     "datasets>=2.14.0",
#     "accelerate>=0.24.0",
#     "torch>=2.0.0"
# ]
# ///

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

def main():
    # Load preference dataset
    dataset = load_dataset("Anthropic/hh-rlhf", split="train")
    
    # Take a reasonable subset for training
    train_dataset = dataset.select(range(10000))
    
    # Load SmolLM3-3B model (pre-trained with SFT)
    model_name = "HuggingFaceTB/SmolLM3-3B"
    model = AutoModelForCausalLM.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token
    
    # Configure DPO training
    training_args = DPOConfig(
        # Core DPO parameters
        beta=0.1,                           # Preference optimization strength
        max_prompt_length=512,              # Maximum prompt length
        max_length=1024,                    # Maximum total sequence length
        
        # Training configuration
        learning_rate=5e-7,                 # Lower than SFT for stability
        per_device_train_batch_size=2,      # Adjust for GPU memory
        gradient_accumulation_steps=8,      # Effective batch size = 16
        max_steps=1000,                     # Sufficient for good alignment
        
        # Optimization
        warmup_steps=100,
        lr_scheduler_type="cosine",
        gradient_checkpointing=True,        # Memory efficiency
        bf16=True,                          # Mixed precision
        
        # Logging and saving
        logging_steps=50,
        save_steps=250,
        output_dir="./smollm3-dpo-aligned",
        
        # Hub integration
        push_to_hub=True,
        hub_model_id="your-username/smollm3-dpo-aligned",  # Change this!
        report_to="trackio",
        
        # Remove unused columns for cleaner training
        remove_unused_columns=False,
    )
    
    # Initialize DPO trainer
    trainer = DPOTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        processing_class=tokenizer,
    )
    
    # Start training
    print("Starting DPO training...")
    trainer.train()
    
    print("Training completed! Model saved and pushed to Hub.")

if __name__ == "__main__":
    main()

# Submit DPO training job to HF Jobs
hf jobs uv run \
    --flavor a100-large \
    --timeout 3h \
    --secrets HF_TOKEN \
    dpo_training.py

# Use TRL's DPO script with HF Jobs
hf jobs uv run \
    --flavor a100-large \
    --timeout 3h \
    --secrets HF_TOKEN \
    "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/dpo.py" \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B \
    --dataset_name Anthropic/hh-rlhf \
    --learning_rate 5e-7 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --max_steps 1000 \
    --beta 0.1 \
    --max_prompt_length 512 \
    --max_length 1024 \
    --output_dir smollm3-dpo-aligned \
    --push_to_hub \
    --hub_model_id your-username/smollm3-dpo-aligned \
    --report_to trackio

# List all your jobs
hf jobs ps -a

# Monitor job logs in real-time
hf jobs logs <job_id> --follow

# Check job details
hf jobs inspect <job_id>

# Local evaluation of your trained model
from transformers import pipeline

# Load your trained model
model_name = "your-username/smollm3-dpo-aligned"
generator = pipeline("text-generation", model=model_name, tokenizer=model_name)

# Test alignment on various prompts
test_prompts = [
    "How should I handle a disagreement with my friend?",
    "What's the best way to learn programming?", 
    "How can I be more productive at work?",
    "What should I do if I see someone being bullied?"
]

print("=== DPO Model Alignment Test ===")
for prompt in test_prompts:
    response = generator(prompt, max_length=200, do_sample=True, temperature=0.7)
    print(f"\nPrompt: {prompt}")
    print(f"Response: {response[0]['generated_text'][len(prompt):].strip()}")

Evaluate your model using HF Jobs and LightEval
使用 HF Jobs (HF Jobs) 和 LightEval (LightEval) 评估您的模型
Submit your results to the course leaderboard
将您的结果提交到课程排行榜
Compare your model's alignment quality with other submissions
将您的模型对齐质量与其他提交内容进行比较

Resources and Further Reading

DPO Paper - Original Direct Preference Optimization research
DPO Paper (DPO Paper) - 原始直接偏好优化研究 (Direct Preference Optimization research)
TRL DPO Documentation - Implementation details and examples
TRL DPO Documentation (TRL DPO Documentation) - 实施细节和示例
Anthropic HH-RLHF Paper - Human feedback methodology
Anthropic HH-RLHF Paper (Anthropic HH-RLHF Paper) - 人工反馈方法 (Human feedback methodology)
• Alignment Handbook - Advanced alignment techniques

Read the written guide for the chapter ✅
阅读本章的书面指南✅
Train a model using what you learned in the chapter.
使用你在本章中学到的内容训练模型。
Push the model to the Hugging Face Hub.
将模型推送到 Hugging Face Hub。
Evaluate the model using hf jobs.
使用 hf jobs (Hugging Face Jobs) 评估模型。
Open a pull request on the leaderboard.
在排行榜 (leaderboard) 上开启一个 pull request (PR)。

1. Read the written guide for the chapter and 2. Train a model using what you learned in the chapter.

DPO Training Exercise - Complete DPO training guide with SmolLM3
DPO Training Exercise (DPO Training Exercise) - 包含 SmolLM3 的完整 DPO 训练指南

3. Push the model to the Hugging Face Hub

hf jobs uv run \
    --flavor a100-large \
    --secrets HF_TOKEN \
    "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/dpo.py" \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B \
    --dataset_name Anthropic/hh-rlhf \
    --learning_rate 5e-7 \
    --beta 0.1 \
    --max_steps 1000 \
    --push_to_hub \
    --hub_model_id your-username/smollm3-dpo-aligned \
    --report_to trackio

4. Evaluate the model using hf jobs

hf jobs uv run \
    --flavor a10g-large \
    --with "lighteval[vllm]" \
    --secrets HF_TOKEN \
    lighteval vllm "model_name=<your-username>/<your-model-name>" \
    "lighteval|truthfulqa:mc2|0|0,lighteval|hellaswag|0|0,lighteval|arc:challenge|0|0" \
    --push-to-hub --results-org <your-username>

5. Open a pull request on the leaderboard space

Add your model's results to submissions.json
将您的模型结果添加到 submissions.json 中
Share your training and evaluation commands in the PR text.
在 PR 文本中分享您的训练和评估命令。

{
    "submissions": [

        ... // existing submissions
        
        {
            "username": "<your-username>",
            "model_name": "<your-model-name>", 
            "chapter": "3",
            "method": "DPO",
            "submission_date": "<your-submission-date>",
            "results-dataset": "<your-results-dataset>",
            "base_model": "HuggingFaceTB/SmolLM3-3B",
            "preference_dataset": "Anthropic/hh-rlhf"
        }
    ]
}

Test your knowledge

Unit 3 DPO Exercise - Complete DPO training guide
第3单元 DPO练习 - 完成DPO训练指南 (DPO training guide)
DPO Paper - Original research paper
DPO论文 - 原始研究论文 (Original research paper)
TRL DPO Documentation - Implementation details
TRL DPO文档 - 实现细节 (Implementation details)
Anthropic HH-RLHF Paper - Human feedback methodology
Anthropic HH-RLHF论文 - 人类反馈方法 (Human feedback methodology)
Alignment Handbook - Advanced alignment techniques
Alignment Handbook (Alignment Handbook) - 高级对齐技术 (Advanced alignment techniques)

Reasoning-focused VLMs: solve complex problems using visual inputs.

以推理为中心的VLM (Vision Language Model)：使用视觉输入解决复杂问题。

Specialized VLMs: e.g. object detection, segmentation, or document understanding.

专用VLM (Vision Language Model)：例如，对象检测、分割或文档理解。

Vision-Language-Action models: generate end actions for robotics.

视觉-语言-动作模型：为机器人生成最终动作。

Agentic VLMs: enable complex workflows like chatting with documents or interacting with computer through screenshots.

代理式VLM (Vision Language Model)：实现复杂的工作流程，例如与文档聊天或通过屏幕截图与计算机交互。

Any-to-any models: expanding beyond vision and text to handle multiple input/output modalities (text, image, audio, video, etc.).

任意到任意模型：扩展到视觉和文本之外，以处理多种输入/输出模式（文本、图像、音频、视频等）。

MMMU & MMMU-Pro: large multi-discipline benchmarks requiring reasoning across domains like arts, science, and engineering.
MMMU & MMMU-Pro：大型多学科基准，需要跨艺术、科学和工程等领域进行推理。
MMBench: over 3,000 single-choice questions testing skills such as OCR, localization, and reasoning.
MMBench：超过3,000个单项选择题，测试诸如OCR (Optical Character Recognition)、定位和推理等技能。
MMT-Bench: focuses on expert-level multimodal tasks, including recognition, localization, reasoning, and planning.
MMT-Bench：专注于专家级多模态任务，包括识别、定位、推理和规划。

MathVista: evaluates mathematical reasoning in the context of images.
MathVista：评估图像环境中的数学推理。
AI2D: focuses on diagram understanding.
AI2D：专注于图表理解。
ScienceQA: science question answering.
ScienceQA：科学问答。
OCRBench: assesses document understanding and OCR capabilities.
OCRBench：评估文档理解和OCR (Optical Character Recognition) 能力。

Learn how to use VLMs with the 🤗 transformers library
学习如何将 VLM（Vision Language Models，视觉语言模型） 与 🤗 transformers 库一起使用
Understand chat templates and conversation formatting for VLMs
理解 VLM 的聊天模板和对话格式
Fine-tune SmolVLM on your own dataset
在您自己的数据集上微调 SmolVLM
Run both programmatic and CLI-based training workflows
运行基于程序和基于 CLI（Command-Line Interface，命令行界面） 的训练工作流程

Image/Vision Encoder: Converts images into compact numerical representations. Examples: CLIP, SigLIP.
图像/视觉编码器 (Image/Vision Encoder)：将图像转换为紧凑的数字表示。 示例：CLIP, SigLIP。
Embedding Projector: Aligns image features with text embeddings (often a small MLP or linear layer fine-tuned for the multimodal task).
嵌入投影器 (Embedding Projector)：将图像特征与文本嵌入对齐（通常是一个小型 MLP 或线性层，针对多模态任务进行微调）。
Multimodal Projector / Fusion Module: Fuses and enhances connections between visual and textual representations. This step goes beyond alignment, enabling rich cross-modal interaction.representations.
多模态投影器/融合模块 (Multimodal Projector / Fusion Module)：融合并增强视觉和文本表示之间的连接。 这一步超越了对齐，实现了丰富的跨模态交互。
Text Decoder: Generates text (or other outputs) from the fused multimodal representations.
文本解码器 (Text Decoder)：从融合的多模态表示生成文本（或其他输出）。

Image Captioning: generating descriptions for images
图像字幕 (Image Captioning)：生成图像的描述
Visual Question Answering (VQA): answering questions about an image
视觉问答 (Visual Question Answering, VQA)：回答关于图像的问题
Cross-Modal Retrieval: matching images with text and vice versa
跨模态检索 (Cross-Modal Retrieval)：将图像与文本进行匹配，反之亦然
Creative Applications: design, art generation, multimedia content
创意应用 (Creative Applications)：设计、艺术生成、多媒体内容

System message: sets context: "You are an assistant analyzing visual data."
系统消息 (System message)：设置上下文：“您是一位分析视觉数据的助手。”
User queries: combine text and images.
用户查询 (User queries)：结合文本和图像。
Assistant responses: generated text based on multimodal analysis.
助手回复 (Assistant responses)：基于多模态分析生成的文本。

[
  {
    "role": "system",
    "content": [{"type": "text", "text": "You are a VLM specialized in charts."}]
  },
  {
    "role": "user",
    "content": [
      {"type": "image", "image": "<image_data>"},
      {"type": "text", "text": "What is the highest value in this chart?"}
    ]
  },
  {
    "role": "assistant",
    "content": [{"type": "text", "text": "42"}]
  }
]

from transformers import pipeline

# Initialize the pipeline with a VLM
pipe = pipeline("image-text-to-text", "HuggingFaceTB/SmolVLM2-2.2B-Instruct", device_map="auto")

# Define your conversation with an image
messages = [
     {
         "role": "user",
         "content": [
             {
                 "type": "image",
                 "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
             },
             {"type": "text", "text": "Describe this image."},
         ],
     }
 ]

outputs = pipe(text=messages, max_new_tokens=60, return_full_text=False)

# Generate response - pipeline handles multimodal inputs automatically
response = pipe(messages, max_new_tokens=128, temperature=0.7)

print(response[0]['generated_text'][-1]['content'])  # Print the model's description

The image depicts a close-up view of a flower garden, specifically focusing on a pink flower. The flower is the central subject of the image, and it is a prominent feature due to its vibrant color and intricate details. The flower has a circular shape, with petals that are slightly curled and have a gradient from light to dark pink. The petals are arranged symmetrically around the central pistil, which is visible in the center of the flower. The pistil is a small, yellow structure that is surrounded by a cluster of stamens, which are visible as small, yellow structures. The flower also has a small, black

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
from transformers.image_utils import load_image

device = "cuda" if torch.cuda.is_available() else "cpu"

# Quantization for efficiency
quant_config = BitsAndBytesConfig(load_in_4bit=True)
model_name = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(model_name, quantization_config=quant_config).to(device)
processor = AutoProcessor.from_pretrained(model_name)

# Load image
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = load_image(image_url)

# Create input messages
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Can you describe the image?"}
        ]
    },
]

# Prepare inputs
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
inputs = inputs.to(device)

# Generate outputs
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
)[0]

# Extract only the assistant response
assistant_response = generated_texts.split("Assistant:")[-1].strip()

print(assistant_response)

The image is of a bee on a flower.

Resources

Vision Language Models (Better, Faster, Stronger)
视觉语言模型（更好、更快、更强大）(Vision Language Models (Better, Faster, Stronger))
Vision Language Models Explained
视觉语言模型详解
SmolVLM - small yet mighty Vision Language Model
SmolVLM - 小巧而强大的视觉语言模型

bfloat16 / float16 halves memory requirements while maintaining accuracy.
bfloat16 / float16 将内存需求减半，同时保持准确性。
• 8-bit / 4-bit quantization reduces memory further, with minor performance trade-offs.

Gradient accumulation: maintain effective batch size over multiple steps.
梯度累积 (Gradient accumulation)：在多个步骤中保持有效的批量大小。
Gradient checkpointing: recompute intermediate activations to save memory.
梯度检查点 (Gradient checkpointing)：重新计算中间激活以节省内存。
Start with a large batch, reduce if OOM errors occur, and combine with LoRA/quantization for best results.
从大批量开始，如果出现 OOM 错误则减少，并结合 LoRA/量化 (quantization) 以获得最佳结果。

Visual question answering (VQA)
视觉问答 (Visual question answering, VQA)
Image captioning
图像描述 (Image captioning)
Chart or diagram interpretation
图表或示意图解释

Specialize a VLM in a domain where the base model struggles.
在基础模型难以处理的领域中专门化视觉语言模型 (VLM)。
Learn domain-specific vocabulary or visual patterns.
学习特定领域的词汇或视觉模式。

Requires high-quality, labeled datasets.
需要高质量的标记数据集。
Can be computationally intensive.
可能需要大量的计算。
Risk of overfitting if fine-tuning is too narrow.
如果微调过于狭隘，则存在过度拟合的风险。

Usage Example

from trl import SFTTrainer

training_args = SFTTrainer(
   output_dir="./fine_tuned_model",
   per_device_train_batch_size=4,
   num_train_epochs=3,
   learning_rate=5e-5,
   save_steps=1000,
   bf16=True,
   gradient_checkpointing=True,
   gradient_accumulation_steps=16,
   logging_steps=50
)

SFTConfig(max_length=None, ...)

Data Preparation

• Use image-text pairs, e.g., HuggingFaceM4/ChartQA.
数据准备 使用图像-文本对，例如 HuggingFaceM4/ChartQA。

Model Setup

• Load a pre-trained VLM such as HuggingFaceTB/SmolVLM2-2.2B-Instruct.
模型设置 加载预训练的视觉语言模型 (VLM)，例如 HuggingFaceTB/SmolVLM2-2.2B-Instruct。
Initialize a processor to prepare text and image inputs.
初始化处理器以准备文本和图像输入。

Fine-Tuning Process

• Format data into chat-like messages (system, user, assistant).
微调过程 将数据格式化为类似聊天的消息（系统、用户、助手）。
Configure optimizer, batch size, and gradient accumulation.
配置优化器、批量大小和梯度累积。
Apply quantization and LoRA for memory-efficient training.
应用量化 (quantization) 和 LoRA 以实现内存高效的训练。

Useful for creative tasks, subjective judgments, or multi-choice answers.
适用于创造性任务、主观判断或多项选择答案。
The model learns to select the more human-aligned response, even if it isn’t strictly “correct.”
该模型学习选择更符合人类偏好的响应，即使它不完全“正确”。

Requires high-quality preference-labeled datasets.
需要高质量的偏好标记数据集。
Training involves pairwise preference sampling and careful resource management.
训练涉及成对偏好抽样和仔细的资源管理。

Dataset: Each example contains a prompt (image + question) and two candidate responses:
数据集：每个示例包含一个提示（图像 + 问题）和两个候选响应：

Question: How many families?
Rejected: The image does not provide information about families.
Chosen: The image shows a Union Organization table setup with 18,000 families.

Model Setup: Load the pre-trained VLM, integrate with TRL DPO, and prepare the processor.
模型设置：加载预训练的视觉语言模型 (VLM)，与 TRL DPO 集成，并准备处理器。
Training Pipeline:
• Format dataset into chat-like messages.
训练流程： 将数据集格式化为类似聊天的消息。
Apply a preference-based loss function.
应用基于偏好的损失函数。
Use gradient accumulation, checkpointing, LoRA, and quantization for efficiency.
使用梯度累积、检查点、LoRA 和量化以提高效率。

Start small: test with a subset of the dataset before full training.
从小处着手：在全面训练之前，使用数据集的子集进行测试。
Use gradient checkpointing + LoRA + quantization to reduce memory usage.
使用梯度检查点 + LoRA + 量化来减少内存使用。
Monitor checkpoint frequency to balance storage and safety.
监控检查点频率以平衡存储和安全性。
Validate on a small set to avoid overfitting.
在小数据集上验证以避免过度拟合。

hf jobs uv run \
   --flavor a100-large \
   --secrets HF_TOKEN \
   --timeout 2h \
   "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" \
   --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \
   --dataset_name HuggingFaceM4/ChartQA \
   --report_to trackio

--flavor a100-large: GPU type for training.
--flavor a100-large：用于训练的 GPU 类型。
--secrets HF_TOKEN: Your Hugging Face token.
--secrets HF_TOKEN：您的 Hugging Face 令牌。

# Install required packages (run in Colab or your environment)
pip install transformers datasets trl huggingface_hub trackio num2words==0.5.14

# Import dependencies

import torch
import os
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
from transformers.image_utils import load_image

device = (
    "cuda"
    if torch.cuda.is_available()
    else "mps" if torch.backends.mps.is_available() else "cpu"
)

from huggingface_hub import login
login()

model_name = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
).to(device)

processor = AutoProcessor.from_pretrained(model_name)

from datasets import load_dataset
import matplotlib.pyplot as plt

train_dataset, eval_dataset = load_dataset("HuggingFaceM4/ChartQA", split=["train[:10%]", "val[:10%]"])
example = train_dataset[1]
image = load_image(example["image"])

print(example["query"])
print(example["label"][0])

How many values are below 40 in Unfavorable graph?
6

plt.imshow(image)
plt.axis("off")
plt.title("Sample Chart Image")
plt.show()

# Define a chat-style prompt
messages = [
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": example["query"]},
    ]}
]

# Apply the chat template
chat_prompt = processor.apply_chat_template(
      messages, add_generation_prompt=True\
)
print(chat_prompt)

<|im_start|>User:<image>How many values are below 40 in Unfavorable graph?<end_of_utterance>
Assistant:

# Tokenize input
inputs = processor(images=[image], text=chat_prompt, return_tensors="pt").to(device)

# Generate model output
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=20)

# Trim the generated ids to remove the input ids
trimmed_generated_ids = [out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, output)]

# Decode the output text
output_text = processor.batch_decode(
    trimmed_generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

 3.

system_message = """You are a Vision Language Model specialized in interpreting visual data from chart images.
Your task is to analyze the provided chart image and respond to queries with concise answers, usually a single word, number, or short phrase.
The charts include a variety of types (e.g., line charts, bar charts) and contain colors, labels, and text.
Focus on delivering accurate, succinct answers based on the visual information. Avoid additional explanation unless absolutely necessary."""

def format_data(sample):
    return {
        "images": [sample["image"]],
        "messages": [
            {
                "role": "system",
                "content": [{"type": "text", "text": system_message}],
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "image": sample["image"],
                    },
                    {
                        "type": "text",
                        "text": sample["query"],
                    },
                ],
            },
            {
                "role": "assistant",
                "content": [{"type": "text", "text": sample["label"][0]}],
            },
        ],
    }

train_dataset = [format_data(sample) for sample in train_dataset]
eval_dataset = [format_data(sample) for sample in eval_dataset]

from peft import LoraConfig, get_peft_model

# Configure LoRA
peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=8,
    target_modules=["q_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

# Apply PEFT model adaptation
peft_model = get_peft_model(model, peft_config)

# Print trainable parameters
peft_model.print_trainable_parameters()

from trl import SFTConfig, SFTTrainer

# Configure training arguments using SFTConfig
training_args = SFTConfig(
    output_dir="smol-course-smolvlm2-2.2b-instruct-trl-sft-ChartQA",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=1e-4,
    logging_steps=25,
    save_strategy="steps",
    save_steps=25,
    optim="adamw_torch_fused",
    bf16=True,
    push_to_hub=True,
    report_to="trackio",
    max_length=None,
)

# Initialize the Trainer
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=peft_config,
)

# Align the SFTTrainer params with your chosen dataset.

# Train the model
trainer.train()

# Save the model
trainer.save_model(training_args.output_dir)

trl sft \
    --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \
    --dataset_name trl-lib/llava-instruct-mix \
    --output_dir ./smolvln-instruct-sft-cli \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 16 \
    --learning_rate 2e-4 \
    --num_train_epochs 3 \
    --max_length -1 \
    --logging_steps 5 \
    --save_steps 100 \
    --warmup_steps 50 \
    --bf16 True \
    --push_to_hub \
    --hub_model_id your-username/smolvlm2-2.2b-instruct-sft-cli

# Use TRL's maintained SFT script directly
hf jobs uv run \
    --flavor a10g-large \
    --timeout 2h \
    --secrets HF_TOKEN \
    --with num2words==0.5.14 \
    "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py" \
    --model_name_or_path HuggingFaceTB/SmolVLM2-2.2B-Instruct \
    --dataset_name trl-lib/llava-instruct-mix\
    --learning_rate 5e-5 \
    --per_device_train_batch_size 4 \
    --max_length -1 \
    --max_steps 1000 \
    --output_dir smolvlm2-2.2b-instruct-sft-jobs \
    --push_to_hub \
    --hub_model_id your-username/smolvlm2-2.2b-instruct-sft-jobs \
    --report_to trackio