feat:生文
This commit is contained in:
@@ -22,76 +22,85 @@
|
||||
- 优势:文档型数据库,灵活性强
|
||||
- 劣势:关联查询复杂,不适合强关联关系场景
|
||||
|
||||
## 2. 数据表设计
|
||||
## 2. 数据库分离设计
|
||||
|
||||
### 2.1 核心设计思路
|
||||
|
||||
**是否分离英语和汉字?**
|
||||
**字和词分离为两个独立的数据库**
|
||||
|
||||
**建议:统一表设计,通过类型字段区分**
|
||||
**设计原则:**
|
||||
1. ✅ **汉字数据库**:专门存储汉字相关数据,包含汉字的特有属性(笔画、结构、部首等)
|
||||
2. ✅ **词语数据库**:专门存储词语相关数据,包含词语的组成和关联关系
|
||||
3. ✅ **数据隔离**:字和词的数据结构差异较大,分离后更便于管理和优化
|
||||
4. ✅ **扩展性好**:后续如需支持英语单词,可单独设计英语数据库
|
||||
|
||||
**理由:**
|
||||
1. ✅ **数据结构相似**:汉字、词语、英语单词、字母的基础字段基本相同
|
||||
2. ✅ **关联关系统一**:都需要关联图片、句子、关联字词等
|
||||
3. ✅ **查询便利**:统一表结构便于统一查询和筛选
|
||||
4. ✅ **维护简单**:减少表数量,降低维护成本
|
||||
5. ✅ **扩展性好**:后续如需新增类型,只需扩展类型枚举
|
||||
**注意:**
|
||||
- 英语展示不考虑和汉字放在一个库
|
||||
- 当前版本先完成汉字数据库的设计和实现
|
||||
- 词语数据库表暂时设计,后续实现
|
||||
|
||||
**如果数据量特别大,可以考虑:**
|
||||
- 按类型分表(水平分表)
|
||||
- 使用数据库分区功能
|
||||
## 3. 汉字数据库设计
|
||||
|
||||
## 3. 数据表详细设计
|
||||
### 3.1 汉字主表 (chars)
|
||||
|
||||
### 3.1 字词主表 (words)
|
||||
|
||||
存储所有字词的基础信息。
|
||||
存储汉字的基础信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE words (
|
||||
CREATE TABLE chars (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
content VARCHAR(50) NOT NULL COMMENT '字词内容',
|
||||
type VARCHAR(20) NOT NULL COMMENT '类型: chinese_char(汉字), chinese_word(词语), english_word(英语单词), english_letter(英语字母)',
|
||||
grade INT COMMENT '年级: 1-9,NULL表示未设置',
|
||||
pinyins TEXT[] COMMENT '拼音数组,支持多音字',
|
||||
pronunciations TEXT[] COMMENT '读音数组,存储音频文件路径或读音标注',
|
||||
svg_data JSONB COMMENT 'SVG笔画数据,JSON格式',
|
||||
char VARCHAR(10) NOT NULL UNIQUE COMMENT '汉字内容,如"中"',
|
||||
strokes INT NOT NULL COMMENT '笔画数',
|
||||
pinyin TEXT[] NOT NULL COMMENT '拼音数组,支持多音字,如["zhōng", "zhòng"]',
|
||||
radicals VARCHAR(10) COMMENT '偏旁部首',
|
||||
frequency INT COMMENT '使用频率: 0(最常用), 1(较常用), 2(次常用), 3(二级字), 4(三级字), 5(生僻字)',
|
||||
structure VARCHAR(10) COMMENT '汉字结构代码,如"D0"(独体结构), "H2"(左窄右宽)等',
|
||||
traditional VARCHAR(50) COMMENT '繁体字写法,可能有多个,用逗号分隔',
|
||||
stroke_data JSONB COMMENT '笔画数据,JSON格式,存储笔画顺序和路径信息',
|
||||
grade INT COMMENT '年级: 0-9,NULL表示未设置',
|
||||
audios TEXT[] COMMENT '音频文件路径数组',
|
||||
description TEXT COMMENT '描述信息',
|
||||
status VARCHAR(20) DEFAULT 'active' COMMENT '状态: active(启用), inactive(禁用)',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
created_by BIGINT COMMENT '创建人ID',
|
||||
updated_by BIGINT COMMENT '更新人ID',
|
||||
|
||||
-- 索引
|
||||
INDEX idx_content (content),
|
||||
INDEX idx_type (type),
|
||||
INDEX idx_char (char),
|
||||
INDEX idx_strokes (strokes),
|
||||
INDEX idx_pinyin (pinyin),
|
||||
INDEX idx_radicals (radicals),
|
||||
INDEX idx_frequency (frequency),
|
||||
INDEX idx_structure (structure),
|
||||
INDEX idx_grade (grade),
|
||||
INDEX idx_status (status),
|
||||
INDEX idx_created_at (created_at)
|
||||
INDEX idx_status (status)
|
||||
);
|
||||
|
||||
-- 全文搜索索引
|
||||
CREATE INDEX idx_content_fulltext ON words USING GIN (to_tsvector('simple', content));
|
||||
CREATE INDEX idx_char_fulltext ON chars USING GIN (to_tsvector('simple', char));
|
||||
```
|
||||
|
||||
**字段说明:**
|
||||
- `content`: 字词内容,如"中"、"中国"、"hello"
|
||||
- `type`: 类型枚举,便于后续扩展
|
||||
- `grade`: 年级,1-9年级,NULL 表示未设置
|
||||
- `pinyins`: PostgreSQL 数组类型,存储多个拼音
|
||||
- `pronunciations`: 读音数组,可以是音频文件路径或文本标注
|
||||
- `svg_data`: JSONB 类型,存储 SVG 配置数据
|
||||
- `char`: 汉字内容,唯一不重复,如"中"
|
||||
- `strokes`: 笔画数,整数类型
|
||||
- `pinyin`: PostgreSQL 数组类型,存储多个拼音,支持多音字
|
||||
- `radicals`: 偏旁部首,如"中"的部首是"丨"
|
||||
- `frequency`: 使用频率,0为最常用,1为较常用,2为次常用,3为二级字,4为三级字,5为不在《通用规范汉字》的生僻字
|
||||
- `structure`: 汉字结构代码,参考结构代码表(见 `structure_codes.json`),如"D0"(独体结构)、"H2"(左窄右宽)等
|
||||
- `traditional`: 繁体字写法,可能有多个,用逗号分隔,如"乾幹"
|
||||
- `stroke_data`: JSONB 类型,存储笔画数据,包含笔画顺序和路径信息(原 svg_data 字段)
|
||||
- `grade`: 年级,0-9年级,NULL 表示未设置
|
||||
- `audios`: 音频文件路径数组,存储音频文件路径
|
||||
- `status`: 启用/禁用状态,支持软删除
|
||||
|
||||
### 3.2 图片表 (word_images)
|
||||
**结构代码说明:**
|
||||
- 结构代码定义见 `structure_codes.json` 文件
|
||||
- 常见结构代码:D0(独体结构)、D1(镶嵌结构)、B1-B4(上下结构)、H1-H3(左右结构)、R1-R6(半包围结构)等
|
||||
|
||||
存储字词关联的图片信息。
|
||||
### 3.2 汉字图片表 (char_images)
|
||||
|
||||
存储汉字关联的图片信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_images (
|
||||
CREATE TABLE char_images (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word_id BIGINT NOT NULL COMMENT '关联的字词ID',
|
||||
char_id BIGINT NOT NULL COMMENT '关联的汉字ID',
|
||||
image_type VARCHAR(20) NOT NULL COMMENT '图片类型: original(原图), standard(标准图)',
|
||||
file_path VARCHAR(500) NOT NULL COMMENT '图片文件路径',
|
||||
file_name VARCHAR(200) NOT NULL COMMENT '原始文件名',
|
||||
@@ -101,50 +110,47 @@ CREATE TABLE word_images (
|
||||
mime_type VARCHAR(50) COMMENT 'MIME类型',
|
||||
is_primary BOOLEAN DEFAULT FALSE COMMENT '是否为主图',
|
||||
sort_order INT DEFAULT 0 COMMENT '排序顺序',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
FOREIGN KEY (word_id) REFERENCES words(id) ON DELETE CASCADE,
|
||||
INDEX idx_word_id (word_id),
|
||||
FOREIGN KEY (char_id) REFERENCES chars(id) ON DELETE CASCADE,
|
||||
INDEX idx_char_id (char_id),
|
||||
INDEX idx_image_type (image_type),
|
||||
INDEX idx_is_primary (is_primary)
|
||||
);
|
||||
```
|
||||
|
||||
**字段说明:**
|
||||
- `word_id`: 关联的字词ID
|
||||
- `char_id`: 关联的汉字ID
|
||||
- `image_type`: 区分原图和标准图
|
||||
- `file_path`: 图片存储路径
|
||||
- `is_primary`: 标记主图,用于列表展示
|
||||
- `sort_order`: 排序字段,支持多图排序
|
||||
|
||||
### 3.3 字词关联表 (word_relations)
|
||||
### 3.3 汉字关联表 (char_relations)
|
||||
|
||||
存储字词之间的关联关系。
|
||||
存储汉字之间的关联关系(如形近字、同音字等)。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_relations (
|
||||
CREATE TABLE char_relations (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source_word_id BIGINT NOT NULL COMMENT '源字词ID',
|
||||
target_word_id BIGINT NOT NULL COMMENT '目标字词ID',
|
||||
relation_type VARCHAR(20) NOT NULL COMMENT '关联类型: word_to_char(词包含字), char_to_word(字的组词), synonym(同义词), antonym(反义词)',
|
||||
source_char_id BIGINT NOT NULL COMMENT '源汉字ID',
|
||||
target_char_id BIGINT NOT NULL COMMENT '目标汉字ID',
|
||||
relation_type VARCHAR(20) NOT NULL COMMENT '关联类型: similar_shape(形近字), same_pinyin(同音字), same_radical(同部首), similar_structure(同结构)',
|
||||
sort_order INT DEFAULT 0 COMMENT '排序顺序',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
FOREIGN KEY (source_word_id) REFERENCES words(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (target_word_id) REFERENCES words(id) ON DELETE CASCADE,
|
||||
UNIQUE (source_word_id, target_word_id, relation_type),
|
||||
INDEX idx_source_word (source_word_id),
|
||||
INDEX idx_target_word (target_word_id),
|
||||
FOREIGN KEY (source_char_id) REFERENCES chars(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (target_char_id) REFERENCES chars(id) ON DELETE CASCADE,
|
||||
UNIQUE (source_char_id, target_char_id, relation_type),
|
||||
INDEX idx_source_char (source_char_id),
|
||||
INDEX idx_target_char (target_char_id),
|
||||
INDEX idx_relation_type (relation_type)
|
||||
);
|
||||
```
|
||||
|
||||
**关联类型说明:**
|
||||
- `word_to_char`: 词包含字,如"中国"关联"中"和"国"
|
||||
- `char_to_word`: 字的组词,如"中"关联"中国"、"中间"等
|
||||
- `synonym`: 同义词(后续扩展)
|
||||
- `antonym`: 反义词(后续扩展)
|
||||
- `similar_shape`: 形近字,如"人"和"入"
|
||||
- `same_pinyin`: 同音字,如"中"和"钟"
|
||||
- `same_radical`: 同部首,如"中"和"串"
|
||||
- `similar_structure`: 同结构,如都是左右结构
|
||||
|
||||
### 3.4 句子表 (sentences)
|
||||
|
||||
@@ -164,22 +170,21 @@ CREATE TABLE sentences (
|
||||
);
|
||||
```
|
||||
|
||||
### 3.5 字词-句子关联表 (word_sentences)
|
||||
### 3.5 汉字-句子关联表 (char_sentences)
|
||||
|
||||
存储字词和句子的关联关系。
|
||||
存储汉字和句子的关联关系。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_sentences (
|
||||
CREATE TABLE char_sentences (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word_id BIGINT NOT NULL COMMENT '字词ID',
|
||||
char_id BIGINT NOT NULL COMMENT '汉字ID',
|
||||
sentence_id BIGINT NOT NULL COMMENT '句子ID',
|
||||
sort_order INT DEFAULT 0 COMMENT '排序顺序',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
FOREIGN KEY (word_id) REFERENCES words(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (char_id) REFERENCES chars(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (sentence_id) REFERENCES sentences(id) ON DELETE CASCADE,
|
||||
UNIQUE (word_id, sentence_id),
|
||||
INDEX idx_word_id (word_id),
|
||||
UNIQUE (char_id, sentence_id),
|
||||
INDEX idx_char_id (char_id),
|
||||
INDEX idx_sentence_id (sentence_id)
|
||||
);
|
||||
```
|
||||
@@ -234,65 +239,200 @@ CREATE TABLE operation_logs (
|
||||
);
|
||||
```
|
||||
|
||||
## 4. 数据库关系图
|
||||
## 4. 词语数据库设计(暂时设计)
|
||||
|
||||
### 4.1 词语主表 (chinese_words)
|
||||
|
||||
存储词语的基础信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE chinese_words (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word VARCHAR(100) NOT NULL UNIQUE COMMENT '词语内容,如"中国"',
|
||||
pinyin TEXT[] COMMENT '拼音数组,如["zhōng", "guó"]',
|
||||
grade INT COMMENT '年级: 0-9,NULL表示未设置',
|
||||
description TEXT COMMENT '词语解释',
|
||||
audios TEXT[] COMMENT '音频文件路径数组',
|
||||
status VARCHAR(20) DEFAULT 'active' COMMENT '状态: active(启用), inactive(禁用)',
|
||||
|
||||
-- 索引
|
||||
INDEX idx_word (word),
|
||||
INDEX idx_grade (grade),
|
||||
INDEX idx_status (status)
|
||||
);
|
||||
|
||||
-- 全文搜索索引
|
||||
CREATE INDEX idx_word_fulltext ON chinese_words USING GIN (to_tsvector('simple', word));
|
||||
```
|
||||
|
||||
### 4.2 词语-汉字关联表 (word_characters)
|
||||
|
||||
存储词语和组成汉字的关联关系。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_characters (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word_id BIGINT NOT NULL COMMENT '词语ID',
|
||||
char_id BIGINT NOT NULL COMMENT '汉字ID(关联到汉字数据库)',
|
||||
position INT NOT NULL COMMENT '汉字在词语中的位置,从1开始',
|
||||
|
||||
FOREIGN KEY (word_id) REFERENCES chinese_words(id) ON DELETE CASCADE,
|
||||
INDEX idx_word_id (word_id),
|
||||
INDEX idx_char_id (char_id),
|
||||
INDEX idx_position (position)
|
||||
);
|
||||
```
|
||||
|
||||
**注意:** 词语数据库和汉字数据库是分离的,`char_id` 字段存储的是汉字数据库中的汉字ID,需要通过跨库查询或应用层关联。
|
||||
|
||||
### 4.3 词语图片表 (word_images)
|
||||
|
||||
存储词语关联的图片信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_images (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word_id BIGINT NOT NULL COMMENT '关联的词语ID',
|
||||
image_type VARCHAR(20) NOT NULL COMMENT '图片类型: original(原图), standard(标准图)',
|
||||
file_path VARCHAR(500) NOT NULL COMMENT '图片文件路径',
|
||||
file_name VARCHAR(200) NOT NULL COMMENT '原始文件名',
|
||||
file_size BIGINT COMMENT '文件大小(字节)',
|
||||
width INT COMMENT '图片宽度',
|
||||
height INT COMMENT '图片高度',
|
||||
mime_type VARCHAR(50) COMMENT 'MIME类型',
|
||||
is_primary BOOLEAN DEFAULT FALSE COMMENT '是否为主图',
|
||||
sort_order INT DEFAULT 0 COMMENT '排序顺序',
|
||||
|
||||
FOREIGN KEY (word_id) REFERENCES chinese_words(id) ON DELETE CASCADE,
|
||||
INDEX idx_word_id (word_id),
|
||||
INDEX idx_image_type (image_type),
|
||||
INDEX idx_is_primary (is_primary)
|
||||
);
|
||||
```
|
||||
|
||||
### 4.4 词语-句子关联表 (word_sentences)
|
||||
|
||||
存储词语和句子的关联关系。
|
||||
|
||||
```sql
|
||||
CREATE TABLE word_sentences (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
word_id BIGINT NOT NULL COMMENT '词语ID',
|
||||
sentence_id BIGINT NOT NULL COMMENT '句子ID',
|
||||
sort_order INT DEFAULT 0 COMMENT '排序顺序',
|
||||
|
||||
FOREIGN KEY (word_id) REFERENCES chinese_words(id) ON DELETE CASCADE,
|
||||
FOREIGN KEY (sentence_id) REFERENCES sentences(id) ON DELETE CASCADE,
|
||||
UNIQUE (word_id, sentence_id),
|
||||
INDEX idx_word_id (word_id),
|
||||
INDEX idx_sentence_id (sentence_id)
|
||||
);
|
||||
```
|
||||
|
||||
## 5. 数据库关系图
|
||||
|
||||
### 5.1 汉字数据库关系图
|
||||
|
||||
```
|
||||
words (字词主表)
|
||||
├── word_images (图片表) - 一对多
|
||||
├── word_relations (字词关联) - 自关联(多对多)
|
||||
├── word_sentences (字词-句子关联) - 多对多
|
||||
chars (汉字主表)
|
||||
├── char_images (图片表) - 一对多
|
||||
├── char_relations (汉字关联) - 自关联(多对多)
|
||||
├── char_sentences (汉字-句子关联) - 多对多
|
||||
└── operation_logs (操作日志) - 关联资源
|
||||
|
||||
sentences (句子表)
|
||||
└── word_sentences (字词-句子关联) - 多对多
|
||||
|
||||
users (用户表)
|
||||
└── operation_logs (操作日志) - 一对多
|
||||
└── char_sentences (汉字-句子关联) - 多对多
|
||||
```
|
||||
|
||||
## 5. 索引优化建议
|
||||
### 5.2 词语数据库关系图
|
||||
|
||||
### 5.1 查询优化索引
|
||||
- 字词内容索引:支持快速查找
|
||||
- 类型+年级组合索引:支持类型和年级筛选
|
||||
```
|
||||
chinese_words (词语主表)
|
||||
├── word_characters (词语-汉字关联) - 多对多(跨库关联)
|
||||
├── word_images (图片表) - 一对多
|
||||
├── word_sentences (词语-句子关联) - 多对多
|
||||
└── operation_logs (操作日志) - 关联资源
|
||||
|
||||
sentences (句子表)
|
||||
└── word_sentences (词语-句子关联) - 多对多
|
||||
```
|
||||
|
||||
### 5.3 公共表
|
||||
|
||||
```
|
||||
users (用户表)
|
||||
└── operation_logs (操作日志) - 一对多
|
||||
|
||||
sentences (句子表)
|
||||
├── character_sentences (汉字-句子关联) - 多对多
|
||||
└── word_sentences (词语-句子关联) - 多对多
|
||||
```
|
||||
|
||||
## 6. 索引优化建议
|
||||
|
||||
### 6.1 汉字数据库查询优化索引
|
||||
- 汉字内容索引:支持快速查找
|
||||
- 笔画数索引:支持按笔画数筛选
|
||||
- 拼音索引:支持按拼音查找
|
||||
- 部首索引:支持按部首查找
|
||||
- 结构代码索引:支持按结构筛选
|
||||
- 频率索引:支持按使用频率筛选
|
||||
- 全文搜索索引:支持模糊搜索
|
||||
|
||||
### 5.2 关联查询优化
|
||||
### 6.2 词语数据库查询优化索引
|
||||
- 词语内容索引:支持快速查找
|
||||
- 年级索引:支持按年级筛选
|
||||
- 全文搜索索引:支持模糊搜索
|
||||
|
||||
### 6.3 关联查询优化
|
||||
- 外键索引:所有外键字段建立索引
|
||||
- 关联表索引:word_relations、word_sentences 的关联字段索引
|
||||
- 关联表索引:character_relations、character_sentences、word_characters、word_sentences 的关联字段索引
|
||||
|
||||
## 6. 数据迁移策略
|
||||
## 7. 数据迁移策略
|
||||
|
||||
### 6.1 初始化数据
|
||||
### 7.1 初始化数据
|
||||
- 创建默认超级管理员账户
|
||||
- 初始化年级数据字典(可选)
|
||||
- 导入汉字结构代码数据(structure_codes.json)
|
||||
|
||||
### 6.2 数据导入
|
||||
- 支持批量导入字词数据
|
||||
### 7.2 数据导入
|
||||
- 支持批量导入汉字数据(参考 char_common_base.json 格式)
|
||||
- 支持批量导入词语数据
|
||||
- 支持 CSV/Excel 格式导入
|
||||
|
||||
## 7. 数据备份策略
|
||||
### 7.3 数据迁移注意事项
|
||||
- 从旧的 words 表迁移到 chinese_characters 表时,需要:
|
||||
- 将 `content` 字段映射到 `char` 字段
|
||||
- 将 `svg_data` 字段重命名为 `stroke_data`
|
||||
- 补充 `strokes`、`pinyin`、`radicals`、`frequency`、`structure`、`traditional` 字段
|
||||
- 过滤出 `type = 'chinese_char'` 的数据
|
||||
|
||||
### 7.1 备份方案
|
||||
## 8. 数据备份策略
|
||||
|
||||
### 8.1 备份方案
|
||||
- 每日全量备份
|
||||
- 实时增量备份(可选)
|
||||
- 汉字数据库和词语数据库分别备份
|
||||
|
||||
### 7.2 恢复方案
|
||||
### 8.2 恢复方案
|
||||
- 支持时间点恢复
|
||||
- 定期恢复演练
|
||||
|
||||
## 8. 性能优化建议
|
||||
## 9. 性能优化建议
|
||||
|
||||
### 8.1 查询优化
|
||||
### 9.1 查询优化
|
||||
- 使用连接池管理数据库连接
|
||||
- 复杂查询使用视图或物化视图
|
||||
- 合理使用缓存(Redis)
|
||||
- 跨库查询(词语-汉字关联)建议在应用层实现,避免跨库JOIN
|
||||
|
||||
### 8.2 数据量预估
|
||||
- 字词数据:预计 10,000+ 条
|
||||
### 9.2 数据量预估
|
||||
- 汉字数据:预计 8,000+ 条(通用规范汉字表)
|
||||
- 词语数据:预计 10,000+ 条
|
||||
- 图片数据:预计 50,000+ 条
|
||||
- 关联关系:预计 100,000+ 条
|
||||
|
||||
### 8.3 分表策略(如需要)
|
||||
- 当 words 表数据量超过 100 万时,考虑按类型分表
|
||||
### 9.3 分表策略(如需要)
|
||||
- 当 chinese_characters 表数据量超过 100 万时,考虑按频率分表
|
||||
- 当 chinese_words 表数据量超过 100 万时,考虑按年级分表
|
||||
- 使用 PostgreSQL 分区功能
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"3500": "一乙二十丁厂七天"
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "voice-production",
|
||||
"version": "1.0.0",
|
||||
"description": "",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
"test": "echo \"Error: no test specified\" && exit 1",
|
||||
"generate:char-introduce": "node scripts/generate_char_introduce.js"
|
||||
},
|
||||
"keywords": [],
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"packageManager": "pnpm@10.28.0"
|
||||
}
|
||||
@@ -0,0 +1,253 @@
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const https = require("https");
|
||||
const crypto = require("crypto");
|
||||
|
||||
// 腾讯云混元大模型配置
|
||||
// 请设置环境变量 TENCENT_SECRET_ID 和 TENCENT_SECRET_KEY
|
||||
const SECRET_ID = process.env.TENCENT_SECRET_ID;
|
||||
const SECRET_KEY = process.env.TENCENT_SECRET_KEY;
|
||||
|
||||
// API配置
|
||||
const HOST = "hunyuan.tencentcloudapi.com";
|
||||
const SERVICE = "hunyuan";
|
||||
const REGION = "ap-guangzhou";
|
||||
const ACTION = "ChatCompletions";
|
||||
const VERSION = "2023-09-01";
|
||||
const MODEL = "hunyuan-lite"; // 使用混元lite模型,可根据需要更换
|
||||
|
||||
// 签名相关函数
|
||||
function sha256(message, secret = "", encoding = "hex") {
|
||||
const hmac = secret
|
||||
? crypto.createHmac("sha256", secret)
|
||||
: crypto.createHash("sha256");
|
||||
return hmac.update(message).digest(encoding);
|
||||
}
|
||||
|
||||
function getDate(timestamp) {
|
||||
const date = new Date(timestamp * 1000);
|
||||
const year = date.getUTCFullYear();
|
||||
const month = ("0" + (date.getUTCMonth() + 1)).slice(-2);
|
||||
const day = ("0" + date.getUTCDate()).slice(-2);
|
||||
return `${year}-${month}-${day}`;
|
||||
}
|
||||
|
||||
function generateSignature(secretId, secretKey, host, payload, timestamp) {
|
||||
const date = getDate(timestamp);
|
||||
|
||||
// 步骤1:拼接规范请求串
|
||||
const httpRequestMethod = "POST";
|
||||
const canonicalUri = "/";
|
||||
const canonicalQueryString = "";
|
||||
const contentType = "application/json";
|
||||
const canonicalHeaders = `content-type:${contentType}\nhost:${host}\nx-tc-action:${ACTION.toLowerCase()}\n`;
|
||||
const signedHeaders = "content-type;host;x-tc-action";
|
||||
const hashedRequestPayload = sha256(payload);
|
||||
const canonicalRequest = `${httpRequestMethod}\n${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeaders}\n${hashedRequestPayload}`;
|
||||
|
||||
// 步骤2:拼接待签名字符串
|
||||
const algorithm = "TC3-HMAC-SHA256";
|
||||
const credentialScope = `${date}/${SERVICE}/tc3_request`;
|
||||
const hashedCanonicalRequest = sha256(canonicalRequest);
|
||||
const stringToSign = `${algorithm}\n${timestamp}\n${credentialScope}\n${hashedCanonicalRequest}`;
|
||||
|
||||
// 步骤3:计算签名
|
||||
const secretDate = sha256(date, "TC3" + secretKey, "buffer");
|
||||
const secretService = sha256(SERVICE, secretDate, "buffer");
|
||||
const secretSigning = sha256("tc3_request", secretService, "buffer");
|
||||
const signature = sha256(stringToSign, secretSigning);
|
||||
|
||||
// 步骤4:拼接Authorization
|
||||
const authorization = `${algorithm} Credential=${secretId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}`;
|
||||
|
||||
return authorization;
|
||||
}
|
||||
|
||||
// 调用混元API
|
||||
async function callHunyuan(prompt) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const timestamp = Math.floor(Date.now() / 1000);
|
||||
|
||||
const payload = JSON.stringify({
|
||||
Model: MODEL,
|
||||
Messages: [
|
||||
{
|
||||
Role: "user",
|
||||
Content: prompt,
|
||||
},
|
||||
],
|
||||
Stream: false,
|
||||
});
|
||||
|
||||
const authorization = generateSignature(
|
||||
SECRET_ID,
|
||||
SECRET_KEY,
|
||||
HOST,
|
||||
payload,
|
||||
timestamp
|
||||
);
|
||||
|
||||
const options = {
|
||||
hostname: HOST,
|
||||
port: 443,
|
||||
path: "/",
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Host: HOST,
|
||||
"X-TC-Action": ACTION,
|
||||
"X-TC-Version": VERSION,
|
||||
"X-TC-Timestamp": timestamp.toString(),
|
||||
"X-TC-Region": REGION,
|
||||
Authorization: authorization,
|
||||
},
|
||||
};
|
||||
|
||||
const req = https.request(options, (res) => {
|
||||
let data = "";
|
||||
res.on("data", (chunk) => (data += chunk));
|
||||
res.on("end", () => {
|
||||
try {
|
||||
const result = JSON.parse(data);
|
||||
if (result.Response && result.Response.Choices) {
|
||||
resolve(result.Response.Choices[0].Message.Content.trim());
|
||||
} else if (result.Response && result.Response.Error) {
|
||||
reject(new Error(result.Response.Error.Message));
|
||||
} else {
|
||||
reject(new Error("Unknown API response: " + data));
|
||||
}
|
||||
} catch (e) {
|
||||
reject(e);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
req.on("error", reject);
|
||||
req.write(payload);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
// 为单个汉字生成介绍
|
||||
async function generateIntroduce(char) {
|
||||
const prompt = `请为汉字"${char}"生成一个简短的词组介绍,格式为"${char},XXX的${char}",其中XXX是一个包含该汉字的常见词语。
|
||||
例如:
|
||||
- 汉字"天"的介绍是"天,天空的天"
|
||||
- 汉字"地"的介绍是"地,大地的地"
|
||||
- 汉字"人"的介绍是"人,人民的人"
|
||||
|
||||
请只输出介绍内容,不要包含其他解释。比如对于"天",只输出"天,天空的天"。`;
|
||||
|
||||
try {
|
||||
const response = await callHunyuan(prompt);
|
||||
// 清理响应,只保留核心内容
|
||||
let introduce = response.replace(/["""]/g, "").trim();
|
||||
// 如果响应包含多余内容,尝试提取核心部分
|
||||
const match = introduce.match(/.,.+的./);
|
||||
if (match) {
|
||||
introduce = match[0];
|
||||
}
|
||||
// 确保格式正确:如果没有逗号,添加逗号
|
||||
if (!introduce.startsWith(char + ",")) {
|
||||
// 尝试提取词组部分
|
||||
const wordMatch = introduce.match(/(.+的.)/);
|
||||
if (wordMatch) {
|
||||
introduce = `${char},${wordMatch[1]}`;
|
||||
} else {
|
||||
introduce = `${char},${char}字的${char}`;
|
||||
}
|
||||
}
|
||||
return introduce;
|
||||
} catch (error) {
|
||||
console.error(`生成 "${char}" 介绍失败:`, error.message);
|
||||
return `${char},${char}字的${char}`;
|
||||
}
|
||||
}
|
||||
|
||||
// 延时函数
|
||||
function delay(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
// 主函数
|
||||
async function main() {
|
||||
// 检查环境变量
|
||||
if (!SECRET_ID || !SECRET_KEY) {
|
||||
console.error("请设置环境变量 TENCENT_SECRET_ID 和 TENCENT_SECRET_KEY");
|
||||
console.error("例如:");
|
||||
console.error(
|
||||
" export TENCENT_SECRET_ID=your_secret_id"
|
||||
);
|
||||
console.error(
|
||||
" export TENCENT_SECRET_KEY=your_secret_key"
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// 读取汉字数据
|
||||
const charStringPath = path.join(
|
||||
__dirname,
|
||||
"../data/char/char_string.json"
|
||||
);
|
||||
const charString = JSON.parse(fs.readFileSync(charStringPath, "utf-8"));
|
||||
|
||||
// 获取所有汉字
|
||||
const chars = charString["3500"].split("");
|
||||
console.log(`共有 ${chars.length} 个汉字需要处理`);
|
||||
|
||||
// 检查是否有已存在的进度文件
|
||||
const outputPath = path.join(
|
||||
__dirname,
|
||||
"../data/char/char_introduce.json"
|
||||
);
|
||||
let results = [];
|
||||
let startIndex = 0;
|
||||
|
||||
if (fs.existsSync(outputPath)) {
|
||||
try {
|
||||
results = JSON.parse(fs.readFileSync(outputPath, "utf-8"));
|
||||
startIndex = results.length;
|
||||
console.log(`发现已有进度,从第 ${startIndex + 1} 个汉字继续`);
|
||||
} catch (e) {
|
||||
console.log("已有文件解析失败,从头开始");
|
||||
}
|
||||
}
|
||||
|
||||
// 批量处理
|
||||
const BATCH_SIZE = 10; // 每批处理10个
|
||||
const DELAY_BETWEEN_BATCHES = 1000; // 批次间延迟1秒
|
||||
|
||||
for (let i = startIndex; i < chars.length; i += BATCH_SIZE) {
|
||||
const batch = chars.slice(i, Math.min(i + BATCH_SIZE, chars.length));
|
||||
console.log(
|
||||
`处理第 ${i + 1} - ${Math.min(i + BATCH_SIZE, chars.length)} 个汉字...`
|
||||
);
|
||||
|
||||
// 并行处理当前批次
|
||||
const batchResults = await Promise.all(
|
||||
batch.map(async (char, idx) => {
|
||||
// 每个请求之间稍微错开
|
||||
await delay(idx * 100);
|
||||
const introduce = await generateIntroduce(char);
|
||||
console.log(` ${char}: ${introduce}`);
|
||||
return { char, introduce };
|
||||
})
|
||||
);
|
||||
|
||||
results.push(...batchResults);
|
||||
|
||||
// 每批处理完后保存进度
|
||||
fs.writeFileSync(outputPath, JSON.stringify(results, null, 2), "utf-8");
|
||||
console.log(`已保存进度:${results.length}/${chars.length}`);
|
||||
|
||||
// 批次间延迟,避免API限流
|
||||
if (i + BATCH_SIZE < chars.length) {
|
||||
await delay(DELAY_BETWEEN_BATCHES);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n完成!共生成 ${results.length} 个汉字介绍`);
|
||||
console.log(`结果已保存到: ${outputPath}`);
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
Reference in New Issue
Block a user