> For the complete documentation index, see [llms.txt](https://inwt233.gitbook.io/ai-learning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://inwt233.gitbook.io/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01.md).

# 第 1 周：文本、分词与 Tokenizer

本周目标是弄清文本如何一步步变成模型能够处理的数字。

## 每日安排

1. [Unicode、字符、字节和 Token](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-001.md)
2. [文本规范化与中英文预处理](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-002.md)
3. [词表、未知词和 OOV](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-003.md)
4. [字符、单词、子词与字节级分词](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-004.md)
5. [深入：BPE 的合并过程](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-005.md)
6. [深入：特殊 Token、Attention Mask 与模型输入](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-006.md)
7. [实验：比较 WordPiece、BPE、Unigram 与字节级 Tokenizer](/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01/day-007.md)

## 本周完成标准

* 能区分字符、码点、字节、Token 和 Token ID。
* 能解释词表和 OOV 问题。
* 能手动演示一次简单的 BPE 合并。
* 能比较两个 Tokenizer 的切分结果，并解释差异来源。


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://inwt233.gitbook.io/ai-learning/di-yi-bu-fen-nlp-ji-chu/week-01.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.