{"rewrite":{"id":"r_4d8396718b5453e7d7c6b1ff","clusterId":"c_5dca0f3588fe1552bc41d6fd","slug":"nanochat-jp-bundles-japanese-training-and-evaluation-into-one-repository","model":"deepseek-v4-1-flash","headline":"nanochat-jp Bundles Japanese Training And Evaluation Into One Repository","summary":"nanochat-jp, a Japanese fork of Andrej Karpathy's nanochat, carries tokenizer training, pretraining, supervised fine-tuning, reinforcement learning, evaluation and inference in a single repository. It adds a SentencePiece Unigram tokenizer, a Japanese pretraining corpus, Japanese SFT and RL data, Japanese CORE and ChatCORE benchmarks, and a converter to Hugging Face Transformers format. Training runs from speedrun.sh; base_eval.sh and chat_eval.sh handle evaluation.","whyItMatters":"A complete Japanese pipeline in one repository lowers the setup cost for anyone testing how a small model handles Japanese, and the released v4 and v3 checkpoints let others check the developer's benchmark claims without training from scratch.","webCardHtml":"\u003cp\u003eThe fork swaps in a SentencePiece Unigram tokenizer and publishes its Japanese text data on Hugging Face, including a pretraining corpus plus SFT and RL sets. The standard script runs tokenizer training and evaluation, pretraining, SFT and report generation in that order; reinforcement learning is optional in it.\u003c/p\u003e\u003cp\u003eIn the developer\u0026#39;s benchmarks, the model trained with standard settings scored comparably to Qwen3.5-0.8B on Japanese ability, and nanochat-jp-v4 improved on four of five metrics against v3. On YOMI-Gen it rose from 41.4 to 52.3, and on YOMI-Gen and PFGen it passed Qwen3.5-2B; the Qwen3.5 models were evaluated with thinking disabled. Checkpoints for v4 and v3, an SFT model in Hugging Face format, and a vLLM plugin are released.\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cstrong\u003eSentencePiece Unigram tokenizer\u003c/strong\u003e: processes Japanese text\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eJapanese pretraining corpus\u003c/strong\u003e: published on Hugging Face\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eJapanese SFT data\u003c/strong\u003e: question and answer examples for conversational ability\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eJapanese RL data\u003c/strong\u003e: for improving responses\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eCORE and ChatCORE benchmarks\u003c/strong\u003e: measure Japanese ability\u003c/li\u003e\u003cli\u003e\u003cstrong\u003eHugging Face Transformers format converter\u003c/strong\u003e\u003c/li\u003e\u003c/ul\u003e","blueskyPost":"nanochat-jp's Hugging Face converter is the quiet load-bearing piece. Training and eval in one repo only pays off if the weights leave the repo, and that script is the exit.","twitterPost":"nanochat-jp keeps training and eval in one repo, but its Hugging Face converter decides whether the weights ever leave it.","threadsPost":"nanochat-jp packs tokenizer, pretraining, SFT, RL and eval into one repo, which makes the Hugging Face converter the piece that matters. A self-contained Japanese pipeline is only useful to outsiders if its weights can be exported to a format the rest of the tooling reads.","newsletterBlurb":"nanochat-jp is a Japanese fork of Andrej Karpathy's nanochat that keeps the whole pipeline, from tokenizer training to evaluation and inference, in one repository. It adds a SentencePiece Unigram tokenizer, a Japanese pretraining corpus, Japanese SFT and RL data, Japanese CORE and ChatCORE benchmarks and a Hugging Face format converter. The developer reports v4 improved on four of five metrics against v3 and released v4 and v3 checkpoints plus a vLLM plugin.","attributionJson":"[{\"source\":\"GIGAZINE\",\"url\":\"https://gigazine.net/news/20261102-nanochat-jp/\",\"title\":\"nanochat-jp, a Japanese fork of nanochat, completes tokenizer training, pretraining, SFT, RL, evaluation, and inference in a single repository, with a SentencePiece Unigram tokenizer, Japanese corpus and Japanese SFT/RL data, Japanese benchmarks (CORE / ChatCORE), and a converter to HuggingFace format\"}]","lintFlagsJson":null,"lintHits":0,"costUsd":0,"inputTokens":5654,"outputTokens":1093,"status":"published","repairAttempts":0,"nextRepairAt":null,"factsAttemptedAt":1793630648,"createdAt":"2026-11-02T14:39:18.000Z","publishedAt":"2026-11-02T14:43:01.000Z","updatedAt":"2026-11-02T14:43:01.000Z"},"cluster":{"id":"c_5dca0f3588fe1552bc41d6fd","canonicalTitle":"nanochatの日本語版フォーク「nanochat-jp」、トークナイザ学習・事前学習・SFT・RL・評価・推論を単一リポジトリで完結させ、SentencePiece Unigram トークナイザ・日本語コーパスと日本語 SFT/RL データ・日本語ベンチマーク（CORE / ChatCORE）・HuggingFace形式への変換器あり","representativeArticleId":"a_b5217f9d765a4fb4c1a7f1bb","sourceCount":1,"writtenSourceCount":1,"writeAttempts":0,"isSolo":true,"entitiesJson":"{\"anime_titles\":[],\"manga_titles\":[],\"work_titles\":[\"nanochat-jp\",\"nanochat\"],\"studios\":[],\"people\":[],\"type\":\"release\",\"domain\":\"other\",\"is_roundup\":false}","contentType":"news","status":"published","firstSeenAt":"2026-11-02T14:00:00.000Z","lastSeenAt":"2026-11-02T14:00:00.000Z","updatedAt":"2026-11-02T14:43:00.000Z"},"attribution":[{"source":"GIGAZINE","url":"https://gigazine.net/news/20261102-nanochat-jp/","title":"nanochatの日本語版フォーク「nanochat-jp」、トークナイザ学習・事前学習・SFT・RL・評価・推論を単一リポジトリで完結させ、SentencePiece Unigram トークナイザ・日本語コーパスと日本語 SFT/RL データ・日本語ベンチマーク（CORE / ChatCORE）・HuggingFace形式への変換器あり"}],"entities":{"anime_titles":[],"manga_titles":[],"work_titles":["nanochat-jp","nanochat"],"studios":[],"people":[],"type":"release","domain":"other","is_roundup":false},"keyFacts":["nanochat-jp is a Japanese fork of nanochat, the project released by Andrej Karpathy in October 2025.","nanochat-jp adds a SentencePiece Unigram tokenizer, a Japanese pretraining corpus, Japanese SFT and RL data, Japanese CORE and ChatCORE benchmarks, and a Hugging Face Transformers converter.","Training runs from speedrun.sh in the order of tokenizer training and evaluation, pretraining, SFT and training report generation.","In developer benchmarks, nanochat-jp-v4 rose from 41.4 to 52.3 on YOMI-Gen and passed Qwen3.5-2B on YOMI-Gen and PFGen.","Checkpoints for nanochat-jp-v4 and nanochat-jp-v3, an SFT model in Hugging Face format, and a vLLM plugin are released."],"pageLinks":{"healthSlugs":["nanochat-jp","nanochat"],"storySlugs":[],"entities":[{"slug":"nanochat-jp","supported":0,"disputed":0,"contradicted":0,"unverified":1,"overall":"unverified","story_count":1,"has_dossier":false},{"slug":"nanochat","supported":0,"disputed":0,"contradicted":0,"unverified":0,"overall":"unverified","story_count":1,"has_dossier":false}],"stories":[],"entitiesOk":true,"storiesOk":true}}