MiniMind

Training and inference features

• Read the architecture: Dense and MoE implementations expose embeddings, grouped-query attention, RoPE, RMSNorm and feed-forward layers. Follow the tensor path and compare it with the architecture diagram. The current mainline aligns with the Qwen3 / Qwen3-MoE ecosystem.
• Pretraining: next-token prediction: The pretraining script reads JSONL text examples, computes next-token loss and saves weights. The mini dataset is useful for an initial reproduction; a larger version supports further training. Text length, cleaning and learning rate affect the result.
• SFT: conversation and instruction format: Full-parameter SFT continues from pretrained weights using conversations. Current mainline SFT data includes basic tool-call examples, and full_sft weights include corresponding basic behavior. Tokenizer and chat template must match training and inference.
• LoRA adaptation and weight merging: A native LoRA implementation adjusts the model with a small set of additional trainable parameters. A merge-and-export workflow is included. Use it to study task or style changes while keeping the adapter matched to its base model.
• MoE expert routing: MiniMind-3-MoE has about 198 million total parameters and 64 million active parameters per token. Compare routing and auxiliary loss with Dense. Fewer active parameters do not make the full set of weights occupy the same memory or storage as a 64M Dense model.
• DPO preference optimization: DPO adjusts behavior using paired preferred and rejected answers. The repository includes a script and data loading. Preference criteria shape the output; training loss alone does not establish correctness or reliability.
• PPO, GRPO and CISPO experiments: These methods introduce rewards and include generation, scoring and parameter updates. Study them after the basic path works. They require extra generation compute, and poorly designed rewards can teach the model to satisfy the scorer rather than solve the task.
• Multi-turn tool use and Agentic RL: train_agent.py supports GRPO / CISPO for multi-turn tool-use scenarios. Tool interaction trajectories become part of training, making it possible to study when calls are made, how results are used and what happens in later turns.
• Reasoning content and chat templates: Current code uses the chat template, think markers and open_thinking to control reasoning content. It no longer follows older tutorials built around a separate train_reason.py. Displaying reasoning and producing correct answers need separate evaluation.
• Distillation and data preparation: The project includes distillation, cleaning, deduplication and format preparation. Inspect how teacher outputs or training signals reach a smaller model. Teacher outputs can contain mistakes, and source data permissions need their own review.
• Training charts and resuming: Scripts record loss, learning rate and progress, and save resume state including model and optimizer. Use --from_resume 1 to recover an existing checkpoint. A training-resume checkpoint serves a different purpose from inference-only weights.
• Command line and Streamlit WebUI: Use eval_llm.py for command-line chat or the Streamlit interface. WebUI scans model subdirectories under scripts, so Transformers-format models must be placed there first. It displays reasoning, tool selection and multi-turn calls.
• API serving and inference tools: A minimal OpenAI-compatible server supports reasoning_content, tool_calls and open_thinking. Documentation also describes Ollama, vLLM and llama.cpp paths. Check the model format and engine version before integration.
• Evaluation, licensing and runtime costs: The project includes evaluation paths for C-Eval, C-MMLU and OpenBookQA and uses Apache 2.0 for code. Downloading code requires no subscription. GPU, storage or cloud services have their own costs. Use a separate fixed set of business prompts to check facts, refusals and repeatability.