feat: 完成分类、worksheet 接口开发，资源迁移到七牛
This commit is contained in:
R524809
@@ -0,0 +1,17 @@
|
||||
# 相对路径以 packages/asset-sync 为基准，也可填写绝对路径。
|
||||
SOURCE_DIR=../../../doodle-assets/cloud_backup
|
||||
# 七牛服务端凭据，仅在 --apply 时要求；不能使用微信 AppSecret。
|
||||
QINIU_AK=
|
||||
QINIU_SK=
|
||||
# 所有资源共用的公开空间，包含 user-avatars 头像。
|
||||
QINIU_BUCKET=
|
||||
# 自动查询区域；需要固定时填 z0、z1、z2、na0、as0 等官方区域 ID。
|
||||
QINIU_REGION=
|
||||
# 对象 key = 前缀 + 备份相对路径；留空表示保留原始路径。
|
||||
QINIU_KEY_PREFIX=doodle/
|
||||
# 可选：用于报告中的公共 URL，只填 HTTPS origin，不含路径；包含头像 URL。
|
||||
QINIU_PUBLIC_DOMAIN=
|
||||
SYNC_CONCURRENCY=3
|
||||
SYNC_RETRIES=2
|
||||
# 内存上传上限，默认 64 MiB；每个并发任务最多读取一份文件。
|
||||
SYNC_MAX_FILE_BYTES=67108864
|
||||
@@ -0,0 +1 @@
|
||||
reports/
|
||||
@@ -0,0 +1,107 @@
|
||||
# 本地备份同步七牛
|
||||
|
||||
独立 Node.js 脚本项目，使用官方 qiniu SDK 7.15.2。入口命令在 doodle-store 根目录执行。只复制本地文件到已存在的七牛空间，不创建空间、不删除/覆盖远端对象、不修改数据库或业务 URL。
|
||||
|
||||
## 配置
|
||||
|
||||
已生成本机 `packages/asset-sync/.env`，源目录配置为：
|
||||
|
||||
```dotenv
|
||||
SOURCE_DIR=/Users/joey-xd/sites/doodle/doodle-assets/cloud_backup
|
||||
QINIU_AK=
|
||||
QINIU_SK=
|
||||
QINIU_BUCKET=
|
||||
QINIU_REGION=
|
||||
QINIU_KEY_PREFIX=doodle/
|
||||
QINIU_PUBLIC_DOMAIN=
|
||||
SYNC_CONCURRENCY=3
|
||||
SYNC_RETRIES=2
|
||||
SYNC_MAX_FILE_BYTES=67108864
|
||||
```
|
||||
|
||||
填写本地 `.env` 即可，不要将 AK/SK 写到小程序、Git、命令参数或聊天中。该文件被 Git 忽略，并以 0600 权限保存。新机器可从 `.env.example` 复制；相对 SOURCE_DIR 以配置文件目录为基准，运行目录不影响源路径。进程环境变量优先于文件配置。
|
||||
|
||||
| 配置 | 说明 |
|
||||
| ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
||||
| SOURCE_DIR | 本地备份根目录，工具只读 |
|
||||
| QINIU_AK / QINIU_SK | 七牛服务端 AccessKey / SecretKey，仅正式同步要求 |
|
||||
| QINIU_BUCKET | 所有资源共用的公开空间：图片、头像、字体、JSON 等 |
|
||||
| QINIU_REGION | 可留空自动查询；固定区域时使用七牛 region ID，如 z0、z1、z2、na0、as0 |
|
||||
| QINIU_KEY_PREFIX | 对象前缀；默认 doodle/，空串为原相对路径 |
|
||||
| QINIU_PUBLIC_DOMAIN | 可选 HTTPS origin，例如 https://cdn.example.com；不含 /doodle 等路径，仅用于生成报告中的公共 URL，不证明域名已经绑定该空间 |
|
||||
| SYNC_CONCURRENCY | 并发文件数，1～10，默认 3 |
|
||||
| SYNC_RETRIES | 瞬时错误额外重试次数，0～5，默认 2 |
|
||||
| SYNC_MAX_FILE_BYTES | 每个文件读取内存的大小限制，默认 64 MiB |
|
||||
|
||||
空间必须已存在且有权查询和上传。执行前读取空间设置，核对目标空间实际 private=0，工具不会自动调整 ACL。头像与其他资源使用同一个 QINIU_BUCKET，无需配置私有空间。
|
||||
|
||||
## 使用
|
||||
|
||||
```bash
|
||||
# 查看用法
|
||||
pnpm assets:sync --help
|
||||
|
||||
# 默认预演全部文件，不调用七牛接口，也不要求 AK/SK
|
||||
pnpm assets:sync
|
||||
|
||||
# 只预演题型预览图
|
||||
pnpm assets:sync --only assets/previews
|
||||
|
||||
# 填写 .env 后，先同步 51 张预览
|
||||
pnpm assets:sync --apply --only assets/previews
|
||||
|
||||
# 可选择多个目录
|
||||
pnpm assets:sync --apply --only assets/fonts --only assets/materials
|
||||
|
||||
# 全部同步，含头像，均使用 QINIU_BUCKET 公开空间
|
||||
pnpm assets:sync --apply
|
||||
|
||||
# 自定义配置文件/报告；报告文件不能已存在，也不能位于源备份内
|
||||
pnpm assets:sync --env /绝对路径/.env.sync --report /绝对路径/report.json
|
||||
```
|
||||
|
||||
`--only` 按目录前缀或精确文件路径筛选，不是 glob；`--dry-run` 可显式声明预演，与 `--apply` 互斥。通过默认预演确认清单后，再添加 --apply 上传。文件缺失、损坏 JSON、超限文件、软链接及无法识别的预览/头像图片会停止预检。
|
||||
|
||||
## 对象路径和同步行为
|
||||
|
||||
默认映射：
|
||||
|
||||
```text
|
||||
assets/previews/math/number-find.jpg
|
||||
→ 公共 bucket : doodle/assets/previews/math/number-find.jpg
|
||||
user-avatars/<文件名>.png
|
||||
→ 同一公共 bucket : doodle/user-avatars/<文件名>.png
|
||||
```
|
||||
|
||||
头像保留 `user-avatars/` 相对路径，与预览等资源共用公开空间。配置 QINIU_PUBLIC_DOMAIN 后，报告也会生成头像的公开地址。工具不根据文件名推断用户身份，用户归属仍需在后续绑定时核实。
|
||||
|
||||
1. 扫描源目录，忽略隐藏文件，记录实际 MIME、字节数、SHA-256 与七牛 ETag；不修改源文件。
|
||||
2. 正式执行时重新读取并核对本地摘要，使用该次读取的 Buffer 上传，避免核对后源文件被修改。
|
||||
3. 远端不存在时，用指定 bucket:key 的 insertOnly=1 凭证上传，仅允许新增，启用 SDK CRC 校验和 HTTPS。
|
||||
4. 远端同 key 且 ETag/大小一致：跳过并标记已核对。不同内容：报告 conflict，保留原远端文件；需要新版本时选择新的前缀/key。
|
||||
5. 上传完成再次 stat，对比远端 ETag/大小；不是仅凭“请求成功”就记完成。
|
||||
6. 瞬时网络错误、429、5xx 有限重试，鉴权错误和冲突不盲目重试。上传已成功但响应丢失时，重试遇到 614 后回查摘要恢复。
|
||||
7. 任一文件失败或冲突时命令退出码为 1，其余文件结果仍写入报告。中断后重新运行同一命令，会重新检查远端并跳过已完成对象；未提供分片级断点续传。
|
||||
|
||||
校验方式是七牛内容 ETag 和大小，不等同于从 CDN 下载全部文件后比较 SHA-256。SHA-256 是本地备份摘要，报告明确记录 verifiedBy=qiniu-etag-and-size。不会自动刷新 CDN 或替换同名对象。
|
||||
|
||||
## 报告与后续接入
|
||||
|
||||
默认报告写 `packages/asset-sync/reports/<时间>-<随机ID>.json`，目录被 Git 和格式化工具忽略，文件权限 0600；含源相对路径、目标 bucket/key、MIME、大小、摘要、状态和有限错误码。不包含 AK、SK、上传凭证、签名链接或 SDK 原始异常。中断时报告可能未写完，以重新运行后的远端校验为准。
|
||||
|
||||
主要状态：planned（预演）、uploaded（上传并核对）、skipped（已有同内容）、conflict（同名异内容）、failed、not_attempted（全批配置/权限预检未通过）。常见全批错误：CREDENTIALS_REQUIRED、BUCKET_REQUIRED、BUCKET_VISIBILITY_MISMATCH。上传用户需有 bucketInfo、stat、upload 所需权限。
|
||||
|
||||
这次只做文件同步。同步成功后仍需按报告映射更新新后端 assets（保持现有 ID 并回填 bucket/key/MIME/摘要、status=ready）、字体配置、用户头像绑定及首页 JSON 内的 cloud 引用。content/page-config.json 当前会原样复制，不会自动转换嵌套链接；原云上传和定时任务也不会自动切换到七牛。
|
||||
|
||||
## 验证与当前预演
|
||||
|
||||
```bash
|
||||
pnpm assets:test
|
||||
pnpm check
|
||||
```
|
||||
|
||||
测试使用临时本地文件和模拟七牛接口，覆盖配置、ETag 边界、头像同空间映射、预演无网络、首次上传、重复跳过、冲突保护、重试、响应丢失恢复、源文件变化、校验失败、凭据脱敏及 CLI 报告权限。没有使用真实 AK/SK 或执行真实上传。
|
||||
|
||||
本地备份预演：62 个文件，共 3,568,778 字节；其中题型预览 51、头像 7、字体 1、素材 1、首页配置 1、demo 1。该数量为本地预演结果，不代表已上传。
|
||||
|
||||
实现依据：[七牛官方 Node.js SDK](https://developer.qiniu.com/kodo/sdk/nodejs)、[七牛上传策略](https://developer.qiniu.com/kodo/manual/put-policy)。
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "@doodle/asset-sync",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"description": "本地云存储备份同步七牛工具",
|
||||
"scripts": {
|
||||
"sync": "node src/cli.mjs",
|
||||
"test": "node --test test/*.test.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"qiniu": "7.15.2"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { mkdir, open, realpath } from 'node:fs/promises';
|
||||
import {
|
||||
basename,
|
||||
dirname,
|
||||
isAbsolute,
|
||||
relative,
|
||||
resolve,
|
||||
sep,
|
||||
} from 'node:path';
|
||||
import { parseArgs } from 'node:util';
|
||||
import { loadConfig, packageRoot, SyncError } from './config.mjs';
|
||||
import { buildPlan } from './files.mjs';
|
||||
import { safeFailure, syncPlan } from './sync.mjs';
|
||||
|
||||
/** 报告以独占、仅当前用户可读写方式创建；拒绝写入源备份目录，避免污染下次清单。 */
|
||||
async function reserveReport(path, sourceDir) {
|
||||
const check = (target) => {
|
||||
const inside = relative(sourceDir, target);
|
||||
if (
|
||||
!inside ||
|
||||
(!isAbsolute(inside) &&
|
||||
inside !== '..' &&
|
||||
!inside.startsWith(`..${sep}`))
|
||||
)
|
||||
throw new SyncError('REPORT_INSIDE_SOURCE');
|
||||
};
|
||||
check(path);
|
||||
await mkdir(dirname(path), { recursive: true, mode: 0o700 });
|
||||
const canonicalPath = resolve(
|
||||
await realpath(dirname(path)),
|
||||
basename(path),
|
||||
);
|
||||
check(canonicalPath);
|
||||
return open(canonicalPath, 'wx', 0o600);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
apply: { type: 'boolean' },
|
||||
'dry-run': { type: 'boolean' },
|
||||
only: { type: 'string', multiple: true },
|
||||
report: { type: 'string' },
|
||||
env: { type: 'string' },
|
||||
help: { type: 'boolean' },
|
||||
},
|
||||
});
|
||||
if (values.help) {
|
||||
console.log(
|
||||
'资源同步七牛：默认预演，不修改云端或数据库。\n\npnpm assets:sync [--only assets/previews] [--report /绝对路径/report.json]\npnpm assets:sync --apply [--only assets/previews]\n\n配置：packages/asset-sync/.env；可用 --env 指定其他配置文件。AK/SK 只通过配置/环境变量提供。',
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (values.apply && values['dry-run'])
|
||||
throw new SyncError('CONFLICTING_OPTIONS');
|
||||
const config = await loadConfig(
|
||||
values.env ? { envFile: resolve(values.env) } : {},
|
||||
);
|
||||
const plan = await buildPlan(config, values.only ?? []);
|
||||
const reportPath = values.report
|
||||
? resolve(values.report)
|
||||
: resolve(packageRoot, 'reports', `${Date.now()}-${randomUUID()}.json`);
|
||||
const output = await reserveReport(reportPath, plan.sourceDir);
|
||||
try {
|
||||
// 预演不加载 SDK，不要求七牛凭据，也不调用任何远程接口。
|
||||
const client = values.apply
|
||||
? (await import('./qiniu-client.mjs')).createQiniuClient(config)
|
||||
: undefined;
|
||||
const report = await syncPlan(config, plan, {
|
||||
apply: Boolean(values.apply),
|
||||
client,
|
||||
progress: (value) => console.log(JSON.stringify(value)),
|
||||
});
|
||||
await output.writeFile(`${JSON.stringify(report, null, 4)}\n`, 'utf8');
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
mode: report.mode,
|
||||
ok: report.ok,
|
||||
summary: report.summary,
|
||||
...(report.error ? { error: report.error } : {}),
|
||||
report: reportPath,
|
||||
}),
|
||||
);
|
||||
if (!report.ok) process.exitCode = 1;
|
||||
} finally {
|
||||
await output.close();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
// 不打印异常对象或 stack；配置文件及 SDK 异常可能包含凭据或签名地址。
|
||||
console.error(
|
||||
JSON.stringify({
|
||||
ok: false,
|
||||
error: safeFailure(error),
|
||||
message:
|
||||
error instanceof SyncError
|
||||
? error.message
|
||||
: '同步未完成，请检查配置、源目录和报告路径',
|
||||
}),
|
||||
);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,126 @@
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { dirname, isAbsolute, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { parseEnv } from 'node:util';
|
||||
|
||||
export const packageRoot = fileURLToPath(new URL('../', import.meta.url));
|
||||
|
||||
/** 有限错误码与固定提示；禁止把 SDK 原始异常、签名或配置值写入日志。 */
|
||||
export class SyncError extends Error {
|
||||
constructor(code, message = code, statusCode = 0, retryable = false) {
|
||||
super(message);
|
||||
this.code = code;
|
||||
this.statusCode = statusCode;
|
||||
this.retryable = retryable;
|
||||
}
|
||||
}
|
||||
|
||||
/** 本地文件配置与环境变量合并，环境变量优先；相对源路径以配置文件所在目录为准。 */
|
||||
export async function loadConfig({
|
||||
env = process.env,
|
||||
envFile = resolve(packageRoot, '.env'),
|
||||
} = {}) {
|
||||
let values = {};
|
||||
try {
|
||||
values = parseEnv(await readFile(envFile, 'utf8'));
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT')
|
||||
throw new SyncError('CONFIG_READ_FAILED', '无法读取配置文件');
|
||||
}
|
||||
const merged = { ...values, ...env };
|
||||
const number = (key, fallback, min, max) => {
|
||||
const raw = merged[key] ?? String(fallback);
|
||||
if (
|
||||
!/^\d+$/.test(raw) ||
|
||||
!Number.isSafeInteger(Number(raw)) ||
|
||||
Number(raw) < min ||
|
||||
Number(raw) > max
|
||||
)
|
||||
throw new SyncError('INVALID_CONFIG', `${key} 配置不合法`);
|
||||
return Number(raw);
|
||||
};
|
||||
const source = merged.SOURCE_DIR ?? '../../../doodle-assets/cloud_backup';
|
||||
if (!source.trim())
|
||||
throw new SyncError('INVALID_CONFIG', 'SOURCE_DIR 不能为空');
|
||||
const prefix = normalizeRelativePath(
|
||||
merged.QINIU_KEY_PREFIX ?? 'doodle/',
|
||||
true,
|
||||
);
|
||||
const bucket = (key) => {
|
||||
const value = (merged[key] ?? '').trim();
|
||||
if (value && !/^[a-z0-9][a-z0-9-]{1,62}$/.test(value))
|
||||
throw new SyncError('INVALID_CONFIG', `${key} 配置不合法`);
|
||||
return value;
|
||||
};
|
||||
const domain = (merged.QINIU_PUBLIC_DOMAIN ?? '').trim();
|
||||
if (domain) {
|
||||
try {
|
||||
const url = new URL(domain);
|
||||
if (
|
||||
url.protocol !== 'https:' ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
url.pathname !== '/' ||
|
||||
url.search ||
|
||||
url.hash
|
||||
)
|
||||
throw new Error();
|
||||
} catch {
|
||||
throw new SyncError(
|
||||
'INVALID_CONFIG',
|
||||
'QINIU_PUBLIC_DOMAIN 必须为 HTTPS origin',
|
||||
);
|
||||
}
|
||||
}
|
||||
const region = (merged.QINIU_REGION ?? '').trim();
|
||||
if (region && !/^[a-z0-9][a-z0-9-]{0,30}$/.test(region))
|
||||
throw new SyncError('INVALID_CONFIG', 'QINIU_REGION 配置不合法');
|
||||
return {
|
||||
sourceDir: isAbsolute(source)
|
||||
? resolve(source)
|
||||
: resolve(dirname(envFile), source),
|
||||
ak: (merged.QINIU_AK ?? '').trim(),
|
||||
sk: (merged.QINIU_SK ?? '').trim(),
|
||||
publicBucket: bucket('QINIU_BUCKET'),
|
||||
region,
|
||||
prefix: prefix ? `${prefix}/` : '',
|
||||
publicDomain: domain.replace(/\/$/, ''),
|
||||
concurrency: number('SYNC_CONCURRENCY', 3, 1, 10),
|
||||
retries: number('SYNC_RETRIES', 2, 0, 5),
|
||||
maxFileBytes: number('SYNC_MAX_FILE_BYTES', 67108864, 1, 1073741824),
|
||||
};
|
||||
}
|
||||
|
||||
/** 文件 key 与筛选前缀使用 POSIX 相对路径，拒绝越界、反斜线或空路径段。 */
|
||||
export function normalizeRelativePath(value, allowEmpty = false) {
|
||||
if (
|
||||
typeof value !== 'string' ||
|
||||
value.includes('\\') ||
|
||||
[...value].some(
|
||||
(char) => char.charCodeAt(0) < 32 || char.charCodeAt(0) === 127,
|
||||
)
|
||||
)
|
||||
throw new SyncError('INVALID_PATH', '相对路径配置不合法');
|
||||
const text = value.replace(/\/+$/, '');
|
||||
if (!text && allowEmpty && !value.startsWith('/')) return '';
|
||||
if (
|
||||
!text ||
|
||||
text.startsWith('/') ||
|
||||
text.split('/').some((part) => !part || part === '.' || part === '..')
|
||||
)
|
||||
throw new SyncError('INVALID_PATH', '路径必须位于备份目录内');
|
||||
return text;
|
||||
}
|
||||
|
||||
/** 执行前一次检查全部目标；头像与其他资源共用 QINIU_BUCKET 公开空间。 */
|
||||
export function validateApplyConfig(config, files) {
|
||||
if (!config.ak || !config.sk || /\s/.test(config.ak + config.sk))
|
||||
throw new SyncError(
|
||||
'CREDENTIALS_REQUIRED',
|
||||
'请配置 QINIU_AK 和 QINIU_SK',
|
||||
);
|
||||
for (const file of files) {
|
||||
if (!file.bucket)
|
||||
throw new SyncError('BUCKET_REQUIRED', '请配置 QINIU_BUCKET');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { constants } from 'node:fs';
|
||||
import { lstat, open, readdir, realpath } from 'node:fs/promises';
|
||||
import { join, relative, sep } from 'node:path';
|
||||
import { normalizeRelativePath, SyncError } from './config.mjs';
|
||||
|
||||
/** 七牛 ETag：4 MiB 分块 SHA-1，单块前缀 0x16，多块摘要前缀 0x96。 */
|
||||
export function qiniuEtag(buffer) {
|
||||
const blockSize = 4 * 1024 * 1024;
|
||||
const sha1 = (data) => createHash('sha1').update(data).digest();
|
||||
if (buffer.length <= blockSize)
|
||||
return Buffer.concat([Buffer.from([0x16]), sha1(buffer)]).toString(
|
||||
'base64url',
|
||||
);
|
||||
const blocks = [];
|
||||
for (let offset = 0; offset < buffer.length; offset += blockSize)
|
||||
blocks.push(sha1(buffer.subarray(offset, offset + blockSize)));
|
||||
return Buffer.concat([
|
||||
Buffer.from([0x96]),
|
||||
sha1(Buffer.concat(blocks)),
|
||||
]).toString('base64url');
|
||||
}
|
||||
|
||||
/** 只读普通文件，不跟随文件或父目录软链接；限制单文件内存占用。 */
|
||||
export async function readSource(config, relativePath) {
|
||||
relativePath = normalizeRelativePath(relativePath);
|
||||
let parent = config.sourceDir;
|
||||
for (const part of relativePath.split('/').slice(0, -1)) {
|
||||
parent = join(parent, part);
|
||||
if ((await lstat(parent)).isSymbolicLink())
|
||||
throw new SyncError('SYMLINK_NOT_ALLOWED', '源目录包含软链接');
|
||||
}
|
||||
const path = join(config.sourceDir, relativePath);
|
||||
const actual = await realpath(path);
|
||||
if (!actual.startsWith(`${config.sourceDir}${sep}`))
|
||||
throw new SyncError('SOURCE_OUTSIDE_ROOT');
|
||||
const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
try {
|
||||
const stat = await handle.stat();
|
||||
if (!stat.isFile() || stat.size > config.maxFileBytes)
|
||||
throw new SyncError(
|
||||
'FILE_TOO_LARGE_OR_INVALID',
|
||||
'文件超限或不是普通文件',
|
||||
);
|
||||
const buffer = await handle.readFile();
|
||||
if (buffer.length > config.maxFileBytes)
|
||||
throw new SyncError('FILE_TOO_LARGE_OR_INVALID');
|
||||
return {
|
||||
buffer,
|
||||
size: buffer.length,
|
||||
sha256: createHash('sha256').update(buffer).digest('hex'),
|
||||
etag: qiniuEtag(buffer),
|
||||
};
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
function mimeType(path, buffer) {
|
||||
const head = buffer.subarray(0, 16);
|
||||
if (
|
||||
head
|
||||
.subarray(0, 8)
|
||||
.equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]))
|
||||
)
|
||||
return 'image/png';
|
||||
if (head[0] === 255 && head[1] === 216 && head[2] === 255)
|
||||
return 'image/jpeg';
|
||||
if (
|
||||
head.toString('ascii', 0, 4) === 'RIFF' &&
|
||||
head.toString('ascii', 8, 12) === 'WEBP'
|
||||
)
|
||||
return 'image/webp';
|
||||
if (/^GIF8[79]a/.test(head.toString('ascii'))) return 'image/gif';
|
||||
if (head.toString('ascii', 0, 4) === 'wOF2') return 'font/woff2';
|
||||
if (head.toString('ascii', 0, 4) === 'wOFF') return 'font/woff';
|
||||
if (head.length >= 4 && head.readUInt32BE(0) === 0x00010000)
|
||||
return 'font/ttf';
|
||||
if (head.toString('ascii', 0, 4) === 'OTTO') return 'font/otf';
|
||||
if (path.toLowerCase().endsWith('.json')) {
|
||||
try {
|
||||
JSON.parse(buffer.toString('utf8'));
|
||||
return 'application/json';
|
||||
} catch {
|
||||
throw new SyncError('INVALID_JSON', 'JSON 文件损坏');
|
||||
}
|
||||
}
|
||||
if (
|
||||
path.toLowerCase().endsWith('.svg') &&
|
||||
/^\s*(?:<\?xml[^>]*>\s*)?<svg[\s>]/.test(buffer.toString('utf8'))
|
||||
)
|
||||
return 'image/svg+xml';
|
||||
return 'application/octet-stream';
|
||||
}
|
||||
|
||||
/** 预演同样读取文件并生成摘要；头像与其他资源统一上传到公开空间。 */
|
||||
export async function buildPlan(config, only = []) {
|
||||
const root = await lstat(config.sourceDir);
|
||||
if (!root.isDirectory() || root.isSymbolicLink())
|
||||
throw new SyncError('INVALID_SOURCE_DIR', '备份路径必须为真实目录');
|
||||
config = { ...config, sourceDir: await realpath(config.sourceDir) };
|
||||
const filters = only.map((value) => normalizeRelativePath(value));
|
||||
const selected = (path) =>
|
||||
!filters.length ||
|
||||
filters.some(
|
||||
(prefix) => path === prefix || path.startsWith(`${prefix}/`),
|
||||
);
|
||||
const paths = [];
|
||||
async function walk(dir) {
|
||||
for (const entry of (await readdir(dir, { withFileTypes: true })).sort(
|
||||
(a, b) => a.name.localeCompare(b.name, 'en'),
|
||||
)) {
|
||||
if (entry.name.startsWith('.')) continue;
|
||||
const path = join(dir, entry.name);
|
||||
const rel = relative(config.sourceDir, path).split(sep).join('/');
|
||||
if (entry.isSymbolicLink())
|
||||
throw new SyncError(
|
||||
'SYMLINK_NOT_ALLOWED',
|
||||
'备份目录包含软链接，请移除后重试',
|
||||
);
|
||||
if (entry.isDirectory()) await walk(path);
|
||||
else if (entry.isFile() && selected(rel))
|
||||
paths.push(normalizeRelativePath(rel));
|
||||
else if (!entry.isFile())
|
||||
throw new SyncError('UNSUPPORTED_FILE', '备份包含非普通文件');
|
||||
}
|
||||
}
|
||||
await walk(config.sourceDir);
|
||||
if (!paths.length) throw new SyncError('NO_FILES', '没有找到匹配文件');
|
||||
const files = [];
|
||||
for (const path of paths) {
|
||||
const { buffer, ...hashes } = await readSource(config, path);
|
||||
const visibility = 'public';
|
||||
const key = `${config.prefix}${path}`;
|
||||
if (Buffer.byteLength(key) > 750)
|
||||
throw new SyncError('KEY_TOO_LONG', '七牛对象路径超过 750 字节');
|
||||
const mime = mimeType(path, buffer);
|
||||
if (
|
||||
(path.startsWith('assets/previews/') ||
|
||||
path.startsWith('user-avatars/')) &&
|
||||
!mime.startsWith('image/')
|
||||
)
|
||||
throw new SyncError(
|
||||
'INVALID_IMAGE',
|
||||
'预览或头像文件不是支持的图片格式',
|
||||
);
|
||||
files.push({
|
||||
relativePath: path,
|
||||
key,
|
||||
bucket: config.publicBucket || null,
|
||||
visibility,
|
||||
mimeType: mime,
|
||||
...hashes,
|
||||
publicUrl: config.publicDomain
|
||||
? `${config.publicDomain}/${key.split('/').map(encodeURIComponent).join('/')}`
|
||||
: null,
|
||||
});
|
||||
}
|
||||
return { sourceDir: config.sourceDir, files };
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import qiniu from 'qiniu';
|
||||
import { SyncError } from './config.mjs';
|
||||
|
||||
const networkCodes = new Set([
|
||||
'ECONNRESET',
|
||||
'ETIMEDOUT',
|
||||
'ECONNREFUSED',
|
||||
'ENOTFOUND',
|
||||
'EAI_AGAIN',
|
||||
'UND_ERR_CONNECT_TIMEOUT',
|
||||
]);
|
||||
|
||||
/** 仅以状态码和有限网络错误识别重试；不保留 SDK 返回体或异常信息。 */
|
||||
async function sdkCall(operation, allowed = []) {
|
||||
let result;
|
||||
try {
|
||||
result = await operation();
|
||||
} catch (error) {
|
||||
const status =
|
||||
Number(error?.statusCode ?? error?.resp?.statusCode) || 0;
|
||||
if (allowed.includes(status)) return { status, data: null };
|
||||
throw new SyncError(
|
||||
'QINIU_REQUEST_FAILED',
|
||||
'七牛请求失败',
|
||||
status,
|
||||
status === 429 ||
|
||||
(status >= 500 && status <= 599) ||
|
||||
networkCodes.has(error?.code),
|
||||
);
|
||||
}
|
||||
const status = result?.resp?.statusCode ?? 0;
|
||||
if (status < 200 || status >= 300) {
|
||||
if (allowed.includes(status)) return { status, data: result.data };
|
||||
throw new SyncError(
|
||||
'QINIU_REQUEST_FAILED',
|
||||
'七牛请求失败',
|
||||
status,
|
||||
status === 429 || (status >= 500 && status <= 599),
|
||||
);
|
||||
}
|
||||
return { status, data: result.data };
|
||||
}
|
||||
|
||||
/** 七牛官方 SDK 适配：强制 HTTPS、逐对象上传凭证、仅新增；不创建或修改空间权限。 */
|
||||
export function createQiniuClient(config, sdk = qiniu) {
|
||||
const sdkConfig = new sdk.conf.Config({ useHttpsDomain: true });
|
||||
if (config.region)
|
||||
sdkConfig.regionsProvider = sdk.httpc.Region.fromRegionId(
|
||||
config.region,
|
||||
);
|
||||
const mac = new sdk.auth.digest.Mac(config.ak, config.sk);
|
||||
const manager = new sdk.rs.BucketManager(mac, sdkConfig);
|
||||
const uploader = new sdk.form_up.FormUploader(sdkConfig);
|
||||
return {
|
||||
async bucketInfo(bucket) {
|
||||
return (await sdkCall(() => manager.getBucketInfo(bucket))).data;
|
||||
},
|
||||
async stat(bucket, key) {
|
||||
const result = await sdkCall(
|
||||
() => manager.stat(bucket, key),
|
||||
[612],
|
||||
);
|
||||
return result.status === 612 ? null : result.data;
|
||||
},
|
||||
async upload(file, buffer) {
|
||||
const policy = new sdk.rs.PutPolicy({
|
||||
scope: `${file.bucket}:${file.key}`,
|
||||
insertOnly: 1,
|
||||
expires: 3600,
|
||||
fsizeLimit: file.size,
|
||||
});
|
||||
const token = policy.uploadToken(mac);
|
||||
const extra = new sdk.form_up.PutExtra();
|
||||
extra.fname = file.relativePath.split('/').at(-1);
|
||||
extra.mimeType = file.mimeType;
|
||||
extra.checkCrc = true;
|
||||
const result = await sdkCall(() =>
|
||||
uploader.put(token, file.key, buffer, extra),
|
||||
);
|
||||
if (
|
||||
result.data?.key !== file.key ||
|
||||
result.data?.hash !== file.etag
|
||||
)
|
||||
throw new SyncError('UPLOAD_RESPONSE_MISMATCH');
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
import { readSource } from './files.mjs';
|
||||
import { SyncError, validateApplyConfig } from './config.mjs';
|
||||
|
||||
/** SDK 错误不透传；本地源码可用固定的错误码定位失败环节。 */
|
||||
export function safeFailure(error) {
|
||||
return error instanceof SyncError
|
||||
? { code: error.code, statusCode: error.statusCode || null }
|
||||
: { code: 'LOCAL_OPERATION_FAILED', statusCode: null };
|
||||
}
|
||||
|
||||
/** 只重试瞬时网络/限流/服务异常；同名冲突、授权失败不做盲目覆盖或重试。 */
|
||||
export async function retry(
|
||||
operation,
|
||||
retries,
|
||||
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
|
||||
) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try {
|
||||
return await operation();
|
||||
} catch (error) {
|
||||
if (
|
||||
!(error instanceof SyncError) ||
|
||||
!error.retryable ||
|
||||
attempt >= retries
|
||||
)
|
||||
throw error;
|
||||
await sleep(Math.min(200 * 2 ** attempt, 3000));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function matches(remote, file) {
|
||||
return (
|
||||
remote &&
|
||||
remote.hash === file.etag &&
|
||||
Number.isSafeInteger(remote.fsize) &&
|
||||
remote.fsize === file.size
|
||||
);
|
||||
}
|
||||
|
||||
/** 默认只预演；正式同步先检查所有空间权限，之后有限并发逐项校验，失败不影响报告落盘。 */
|
||||
export async function syncPlan(
|
||||
config,
|
||||
plan,
|
||||
{ apply = false, client, progress = () => {}, sleep } = {},
|
||||
) {
|
||||
const report = {
|
||||
schemaVersion: 1,
|
||||
mode: apply ? 'apply' : 'dry-run',
|
||||
startedAt: new Date().toISOString(),
|
||||
sourceDir: plan.sourceDir,
|
||||
files: plan.files.map((file) => ({
|
||||
...file,
|
||||
status: apply ? 'not_attempted' : 'planned',
|
||||
})),
|
||||
};
|
||||
const run = (operation) => retry(operation, config.retries, sleep);
|
||||
if (apply) {
|
||||
try {
|
||||
validateApplyConfig(config, plan.files);
|
||||
const destinations = new Set(plan.files.map((file) => file.bucket));
|
||||
// 包含头像在内的资源共用公开空间，执行前核对实际访问权限。
|
||||
for (const bucket of destinations) {
|
||||
const info = await run(() => client.bucketInfo(bucket));
|
||||
if (info.private !== 0)
|
||||
throw new SyncError('BUCKET_VISIBILITY_MISMATCH');
|
||||
}
|
||||
let cursor = 0;
|
||||
const worker = async () => {
|
||||
for (;;) {
|
||||
const index = cursor++;
|
||||
const file = report.files[index];
|
||||
if (!file) return;
|
||||
try {
|
||||
const source = await readSource(
|
||||
{ ...config, sourceDir: plan.sourceDir },
|
||||
file.relativePath,
|
||||
);
|
||||
if (
|
||||
source.sha256 !== file.sha256 ||
|
||||
source.size !== file.size
|
||||
)
|
||||
throw new SyncError('SOURCE_CHANGED');
|
||||
const remote = await run(() =>
|
||||
client.stat(file.bucket, file.key),
|
||||
);
|
||||
if (remote) {
|
||||
if (!matches(remote, file))
|
||||
throw new SyncError('REMOTE_CONFLICT');
|
||||
file.status = 'skipped';
|
||||
} else {
|
||||
try {
|
||||
await run(() =>
|
||||
client.upload(file, source.buffer),
|
||||
);
|
||||
} catch (error) {
|
||||
// 614 表示另一任务已创建对象；按摘要复核，不覆盖同名文件。
|
||||
if (
|
||||
!(error instanceof SyncError) ||
|
||||
error.statusCode !== 614
|
||||
)
|
||||
throw error;
|
||||
}
|
||||
const uploaded = await run(() =>
|
||||
client.stat(file.bucket, file.key),
|
||||
);
|
||||
if (!matches(uploaded, file))
|
||||
throw new SyncError('VERIFY_FAILED');
|
||||
file.status = 'uploaded';
|
||||
}
|
||||
file.verifiedBy = 'qiniu-etag-and-size';
|
||||
} catch (error) {
|
||||
file.status =
|
||||
error instanceof SyncError &&
|
||||
error.code === 'REMOTE_CONFLICT'
|
||||
? 'conflict'
|
||||
: 'failed';
|
||||
file.error = safeFailure(error);
|
||||
}
|
||||
progress({
|
||||
completed: report.files.filter(
|
||||
(entry) => entry.status !== 'not_attempted',
|
||||
).length,
|
||||
total: report.files.length,
|
||||
status: file.status,
|
||||
});
|
||||
}
|
||||
};
|
||||
await Promise.all(
|
||||
Array.from(
|
||||
{ length: Math.min(config.concurrency, plan.files.length) },
|
||||
worker,
|
||||
),
|
||||
);
|
||||
} catch (error) {
|
||||
report.error = safeFailure(error);
|
||||
}
|
||||
}
|
||||
report.finishedAt = new Date().toISOString();
|
||||
report.summary = {
|
||||
total: report.files.length,
|
||||
bytes: report.files.reduce((sum, file) => sum + file.size, 0),
|
||||
};
|
||||
for (const file of report.files)
|
||||
report.summary[file.status] = (report.summary[file.status] ?? 0) + 1;
|
||||
report.ok =
|
||||
!report.error &&
|
||||
!report.summary.failed &&
|
||||
!report.summary.conflict &&
|
||||
!report.summary.not_attempted;
|
||||
return report;
|
||||
}
|
||||
@@ -0,0 +1,391 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import {
|
||||
mkdtemp,
|
||||
mkdir,
|
||||
readFile,
|
||||
rm,
|
||||
stat,
|
||||
symlink,
|
||||
writeFile,
|
||||
} from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { test } from 'node:test';
|
||||
import qiniu from 'qiniu';
|
||||
import {
|
||||
loadConfig,
|
||||
normalizeRelativePath,
|
||||
packageRoot,
|
||||
SyncError,
|
||||
} from '../src/config.mjs';
|
||||
import { buildPlan, qiniuEtag } from '../src/files.mjs';
|
||||
import { safeFailure, syncPlan } from '../src/sync.mjs';
|
||||
import { createQiniuClient } from '../src/qiniu-client.mjs';
|
||||
|
||||
const jpeg = Buffer.from([
|
||||
255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 0, 0, 0, 0,
|
||||
]);
|
||||
async function fixture(context) {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'doodle-sync-'));
|
||||
context.after(() => rm(dir, { recursive: true, force: true }));
|
||||
const source = join(dir, 'source');
|
||||
await mkdir(join(source, 'assets/previews/math'), { recursive: true });
|
||||
await mkdir(join(source, 'user-avatars'));
|
||||
await mkdir(join(source, 'content'));
|
||||
await writeFile(join(source, 'assets/previews/math/中文 图.jpg'), jpeg);
|
||||
await writeFile(
|
||||
join(source, 'user-avatars/u.png'),
|
||||
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 0]),
|
||||
);
|
||||
await writeFile(join(source, 'content/page-config.json'), '{"home":null}');
|
||||
await writeFile(join(source, '.DS_Store'), 'ignored');
|
||||
const envFile = join(dir, '.env');
|
||||
await writeFile(
|
||||
envFile,
|
||||
`SOURCE_DIR=./source\nQINIU_AK=test-ak-never-log\nQINIU_SK=test-sk-never-log\nQINIU_BUCKET=test-public\nQINIU_PUBLIC_DOMAIN=https://cdn.example.test\n`,
|
||||
);
|
||||
const config = await loadConfig({ env: {}, envFile });
|
||||
const plan = await buildPlan(config);
|
||||
return { dir, source, envFile, config, plan };
|
||||
}
|
||||
function fakeClient() {
|
||||
const objects = new Map();
|
||||
const calls = [];
|
||||
return {
|
||||
calls,
|
||||
objects,
|
||||
async bucketInfo(bucket) {
|
||||
calls.push(['info', bucket]);
|
||||
return { private: bucket === 'test-private' ? 1 : 0 };
|
||||
},
|
||||
async stat(bucket, key) {
|
||||
calls.push(['stat', bucket, key]);
|
||||
return objects.get(`${bucket}/${key}`) ?? null;
|
||||
},
|
||||
async upload(file, buffer) {
|
||||
calls.push(['upload', file.key]);
|
||||
const id = `${file.bucket}/${file.key}`;
|
||||
if (objects.has(id))
|
||||
throw new SyncError('QINIU_REQUEST_FAILED', 'conflict', 614);
|
||||
objects.set(id, { hash: qiniuEtag(buffer), fsize: buffer.length });
|
||||
},
|
||||
};
|
||||
}
|
||||
const noSleep = async () => {};
|
||||
|
||||
test('配置：相对源路径按 env 文件定位，环境变量覆盖，拒绝非法前缀/域名/数值', async (context) => {
|
||||
const fixtureData = await fixture(context);
|
||||
assert.equal(fixtureData.config.sourceDir, fixtureData.source);
|
||||
assert.equal(fixtureData.config.prefix, 'doodle/');
|
||||
const override = await loadConfig({
|
||||
env: { QINIU_KEY_PREFIX: '' },
|
||||
envFile: fixtureData.envFile,
|
||||
});
|
||||
assert.equal(override.prefix, '');
|
||||
for (const env of [
|
||||
{ SYNC_CONCURRENCY: '0' },
|
||||
{ SYNC_RETRIES: '-1' },
|
||||
{ QINIU_PUBLIC_DOMAIN: 'https://user:secret@host.test' },
|
||||
{ QINIU_KEY_PREFIX: '../escape' },
|
||||
{ QINIU_KEY_PREFIX: '/absolute' },
|
||||
])
|
||||
await assert.rejects(
|
||||
loadConfig({ env, envFile: fixtureData.envFile }),
|
||||
SyncError,
|
||||
);
|
||||
for (const path of ['../x', 'a/../../b', '/x', 'a\\b', 'a//b', 'x\0'])
|
||||
assert.throws(() => normalizeRelativePath(path), SyncError);
|
||||
});
|
||||
|
||||
test('ETag：固定向量覆盖空文件、普通内容与 4 MiB 分块边界', () => {
|
||||
assert.equal(qiniuEtag(Buffer.alloc(0)), 'Fto5o-5ea0sNMlW_75VgGJCv2AcJ');
|
||||
assert.equal(
|
||||
qiniuEtag(Buffer.from('hello')),
|
||||
'Fqr0xh3cxeii2r7eDztILNmuqUNN',
|
||||
);
|
||||
// 固定多块期望值由独立 SHA-1 实现生成，避免只用同一函数自我验证。
|
||||
assert.equal(
|
||||
qiniuEtag(Buffer.alloc(4194304)),
|
||||
'FivMvS848VwT631aif2dhfWV4jvD',
|
||||
);
|
||||
assert.equal(
|
||||
qiniuEtag(Buffer.alloc(4194305)),
|
||||
'lhCFgki5yzon0rjN9uJusf6qtsF6',
|
||||
);
|
||||
});
|
||||
|
||||
test('预演：不调用七牛，不包含 AK/SK，保留目录/摘要头像与预览共用公开空间', async (context) => {
|
||||
const { config, plan } = await fixture(context);
|
||||
const report = await syncPlan(config, plan, {
|
||||
client: new Proxy(
|
||||
{},
|
||||
{
|
||||
get() {
|
||||
throw new Error('禁止网络');
|
||||
},
|
||||
},
|
||||
),
|
||||
});
|
||||
assert.equal(report.ok, true);
|
||||
assert.equal(report.summary.planned, 3);
|
||||
const avatar = report.files.find((file) =>
|
||||
file.relativePath.startsWith('user-avatars/'),
|
||||
);
|
||||
assert.equal(avatar.visibility, 'public');
|
||||
assert.equal(avatar.bucket, 'test-public');
|
||||
assert.equal(
|
||||
avatar.publicUrl,
|
||||
'https://cdn.example.test/doodle/user-avatars/u.png',
|
||||
);
|
||||
assert.ok(report.files.every((file) => file.bucket === avatar.bucket));
|
||||
const preview = report.files.find((file) =>
|
||||
file.relativePath.startsWith('assets/'),
|
||||
);
|
||||
assert.equal(preview.key, 'doodle/assets/previews/math/中文 图.jpg');
|
||||
assert.match(preview.publicUrl, /%E4%B8%AD%E6%96%87%20%E5%9B%BE/);
|
||||
assert.equal(preview.mimeType, 'image/jpeg');
|
||||
assert.equal(preview.sha256.length, 64);
|
||||
assert.ok(!JSON.stringify(report).includes(config.ak));
|
||||
assert.ok(!JSON.stringify(report).includes(config.sk));
|
||||
assert.equal(
|
||||
(await buildPlan(config, ['assets/previews'])).files.length,
|
||||
1,
|
||||
);
|
||||
});
|
||||
|
||||
test('同步：上传后逐对象校验，重跑跳过，相同 key 的不同内容绝不覆盖', async (context) => {
|
||||
const { config, plan } = await fixture(context);
|
||||
const client = fakeClient();
|
||||
const first = await syncPlan(config, plan, { apply: true, client });
|
||||
assert.equal(first.summary.uploaded, 3);
|
||||
assert.equal(first.ok, true);
|
||||
assert.equal(client.objects.size, 3);
|
||||
const second = await syncPlan(config, plan, { apply: true, client });
|
||||
assert.equal(second.summary.skipped, 3);
|
||||
const item = plan.files[0];
|
||||
client.objects.set(`${item.bucket}/${item.key}`, {
|
||||
hash: 'different',
|
||||
fsize: item.size,
|
||||
});
|
||||
const third = await syncPlan(config, plan, { apply: true, client });
|
||||
assert.equal(third.summary.conflict, 1);
|
||||
assert.equal(third.summary.skipped, 2);
|
||||
assert.equal(third.ok, false);
|
||||
assert.equal(
|
||||
client.objects.get(`${item.bucket}/${item.key}`).hash,
|
||||
'different',
|
||||
);
|
||||
assert.equal(client.calls.filter(([op]) => op === 'upload').length, 3);
|
||||
});
|
||||
|
||||
test('全批预检查：缺少凭据/空间、目标空间实际私有时不上传任何文件', async (context) => {
|
||||
const { config, plan } = await fixture(context);
|
||||
for (const changed of [
|
||||
{ ...config, ak: '' },
|
||||
{ ...config, publicBucket: '' },
|
||||
]) {
|
||||
const client = fakeClient();
|
||||
const changedPlan = await buildPlan(changed);
|
||||
const result = await syncPlan(changed, changedPlan, {
|
||||
apply: true,
|
||||
client,
|
||||
});
|
||||
assert.equal(result.ok, false);
|
||||
assert.equal(client.objects.size, 0);
|
||||
}
|
||||
const missing = {
|
||||
...plan,
|
||||
files: plan.files.map((file) =>
|
||||
file.relativePath.startsWith('user-avatars/')
|
||||
? { ...file, bucket: null }
|
||||
: file,
|
||||
),
|
||||
};
|
||||
const client = fakeClient();
|
||||
assert.equal(
|
||||
(await syncPlan(config, missing, { apply: true, client })).error.code,
|
||||
'BUCKET_REQUIRED',
|
||||
);
|
||||
client.bucketInfo = async () => ({ private: 1 });
|
||||
assert.equal(
|
||||
(await syncPlan(config, plan, { apply: true, client })).error.code,
|
||||
'BUCKET_VISIBILITY_MISMATCH',
|
||||
);
|
||||
assert.equal(client.objects.size, 0);
|
||||
});
|
||||
|
||||
test('瞬时错误重试：上传成功但响应丢失时按 614 和远端摘要恢复，不覆盖', async (context) => {
|
||||
const { config, plan } = await fixture(context);
|
||||
const client = fakeClient();
|
||||
const normalUpload = client.upload;
|
||||
let lost = false;
|
||||
client.upload = async (file, buffer) => {
|
||||
await normalUpload(file, buffer);
|
||||
if (!lost) {
|
||||
lost = true;
|
||||
throw new SyncError('QINIU_REQUEST_FAILED', 'lost', 0, true);
|
||||
}
|
||||
};
|
||||
const result = await syncPlan(config, plan, {
|
||||
apply: true,
|
||||
client,
|
||||
sleep: noSleep,
|
||||
});
|
||||
assert.equal(result.ok, true);
|
||||
assert.equal(result.summary.uploaded, 3);
|
||||
assert.equal(client.objects.size, 3);
|
||||
});
|
||||
|
||||
test('源文件变化、远端摘要错误会失败；普通异常和 SDK 凭据不会写进报告', async (context) => {
|
||||
const { config, plan, source } = await fixture(context);
|
||||
await writeFile(
|
||||
join(source, plan.files[0].relativePath),
|
||||
Buffer.concat([jpeg, Buffer.from('changed')]),
|
||||
);
|
||||
const client = fakeClient();
|
||||
const result = await syncPlan(config, plan, { apply: true, client });
|
||||
assert.equal(result.summary.failed, 1);
|
||||
assert.equal(result.files[0].error.code, 'SOURCE_CHANGED');
|
||||
const other = fakeClient();
|
||||
other.stat = async () => null;
|
||||
const verify = await syncPlan(
|
||||
config,
|
||||
{ ...plan, files: [plan.files[1]] },
|
||||
{ apply: true, client: other },
|
||||
);
|
||||
assert.equal(verify.files[0].error.code, 'VERIFY_FAILED');
|
||||
assert.deepEqual(safeFailure(new Error(`${config.ak} ${config.sk}`)), {
|
||||
code: 'LOCAL_OPERATION_FAILED',
|
||||
statusCode: null,
|
||||
});
|
||||
});
|
||||
|
||||
test('扫描：拒绝软链接、超限和损坏图片，保留空普通文件', async (context) => {
|
||||
const { config, source } = await fixture(context);
|
||||
await assert.rejects(buildPlan({ ...config, maxFileBytes: 1 }), SyncError);
|
||||
await writeFile(join(source, 'empty.txt'), '');
|
||||
assert.ok(
|
||||
(await buildPlan(config)).files.some(
|
||||
(file) => file.relativePath === 'empty.txt' && file.size === 0,
|
||||
),
|
||||
);
|
||||
await writeFile(
|
||||
join(source, 'assets/previews/math/bad.jpg'),
|
||||
'<html>denied</html>',
|
||||
);
|
||||
await assert.rejects(buildPlan(config), { code: 'INVALID_IMAGE' });
|
||||
await rm(join(source, 'assets/previews/math/bad.jpg'));
|
||||
await symlink(join(source, 'content'), join(source, 'link'));
|
||||
await assert.rejects(buildPlan(config), { code: 'SYMLINK_NOT_ALLOWED' });
|
||||
});
|
||||
|
||||
test('官方 SDK 适配：HTTPS、明确 key 的 insertOnly token、MIME/CRC 和安全错误映射', async (context) => {
|
||||
const { config, plan } = await fixture(context);
|
||||
const file = plan.files[0];
|
||||
let token, policy, extraValue;
|
||||
const sdk = {
|
||||
...qiniu,
|
||||
rs: {
|
||||
...qiniu.rs,
|
||||
BucketManager: class {
|
||||
constructor(_mac, conf) {
|
||||
assert.equal(conf.useHttpsDomain, true);
|
||||
}
|
||||
async getBucketInfo() {
|
||||
return { resp: { statusCode: 200 }, data: { private: 0 } };
|
||||
}
|
||||
async stat() {
|
||||
return {
|
||||
resp: { statusCode: 612 },
|
||||
data: { error: 'ignored' },
|
||||
};
|
||||
}
|
||||
},
|
||||
},
|
||||
form_up: {
|
||||
...qiniu.form_up,
|
||||
FormUploader: class {
|
||||
async put(value, key, _buffer, extra) {
|
||||
token = value;
|
||||
extraValue = extra;
|
||||
return {
|
||||
resp: { statusCode: 200 },
|
||||
data: { key, hash: file.etag },
|
||||
};
|
||||
}
|
||||
},
|
||||
},
|
||||
};
|
||||
const client = createQiniuClient(config, sdk);
|
||||
assert.equal(await client.stat(file.bucket, file.key), null);
|
||||
assert.equal((await client.bucketInfo(file.bucket)).private, 0);
|
||||
await client.upload(file, jpeg);
|
||||
policy = JSON.parse(
|
||||
Buffer.from(token.split(':')[2], 'base64url').toString(),
|
||||
);
|
||||
assert.equal(policy.insertOnly, 1);
|
||||
assert.equal(policy.scope, `${file.bucket}:${file.key}`);
|
||||
assert.equal(extraValue.mimeType, file.mimeType);
|
||||
assert.equal(extraValue.checkCrc, true);
|
||||
sdk.rs.BucketManager = class {
|
||||
async stat() {
|
||||
throw Object.assign(new Error(config.sk), {
|
||||
resp: { statusCode: 401 },
|
||||
});
|
||||
}
|
||||
};
|
||||
await assert.rejects(
|
||||
createQiniuClient(config, sdk).stat(file.bucket, file.key),
|
||||
(error) =>
|
||||
error.statusCode === 401 &&
|
||||
!error.retryable &&
|
||||
!error.message.includes(config.sk),
|
||||
);
|
||||
});
|
||||
|
||||
test('CLI：默认只预演，报告独占且权限 0600；禁止报告写入源目录', async (context) => {
|
||||
const { dir, envFile, source } = await fixture(context);
|
||||
const reportPath = join(dir, 'report.json');
|
||||
const cli = resolve(packageRoot, 'src/cli.mjs');
|
||||
const result = execFileSync(
|
||||
process.execPath,
|
||||
[cli, '--env', envFile, '--report', reportPath],
|
||||
{ encoding: 'utf8', env: { PATH: process.env.PATH } },
|
||||
);
|
||||
assert.equal(JSON.parse(result).mode, 'dry-run');
|
||||
assert.equal(
|
||||
JSON.parse(await readFile(reportPath, 'utf8')).summary.planned,
|
||||
3,
|
||||
);
|
||||
assert.equal((await stat(reportPath)).mode & 0o777, 0o600);
|
||||
const deniedReport = join(dir, 'no-credentials.json');
|
||||
assert.throws(() =>
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[cli, '--env', envFile, '--apply', '--report', deniedReport],
|
||||
{
|
||||
stdio: 'pipe',
|
||||
env: { PATH: process.env.PATH, QINIU_AK: '', QINIU_SK: '' },
|
||||
},
|
||||
),
|
||||
);
|
||||
const denied = JSON.parse(await readFile(deniedReport, 'utf8'));
|
||||
assert.equal(denied.error.code, 'CREDENTIALS_REQUIRED');
|
||||
assert.equal(denied.summary.not_attempted, 3);
|
||||
|
||||
assert.throws(() =>
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[cli, '--env', envFile, '--report', reportPath],
|
||||
{ stdio: 'pipe' },
|
||||
),
|
||||
);
|
||||
assert.throws(() =>
|
||||
execFileSync(
|
||||
process.execPath,
|
||||
[cli, '--env', envFile, '--report', join(source, 'report.json')],
|
||||
{ stdio: 'pipe' },
|
||||
),
|
||||
);
|
||||
});
|
||||
@@ -1,6 +1,6 @@
|
||||
# 公共接口契约
|
||||
|
||||
仅包含可公开的基础响应和健康检查类型。服务端通过 `workspace:*` 引用，使用 `import type` 避免引入无必要的运行时逻辑。
|
||||
包含可公开的基础响应、健康检查、微信登录请求/结果、本人摘要、分类查询/创建/更新、题型查询/公开 DTO/分页及有限错误原因类型。题型计数为含 seed 的十进制展示字符串，previewImg 可空。服务端通过 `workspace:*` 引用，使用 `import type` 避免引入无必要的运行时逻辑。
|
||||
|
||||
后续可以加入公开请求/响应类型及枚举，但禁止包含 Prisma Client、数据库模型、密钥、管理权限实现或私有生成器。
|
||||
|
||||
|
||||
@@ -10,3 +10,165 @@ export interface HealthStatus {
|
||||
status: 'ok';
|
||||
service: 'doodle-store-server';
|
||||
}
|
||||
|
||||
/** 安全、有限的失败原因；HTTP code 仍承担错误大类，不暴露内部异常。 */
|
||||
export type ApiErrorReason =
|
||||
| 'AUTH_REQUIRED'
|
||||
| 'AUTH_INVALID'
|
||||
| 'AUTH_EXPIRED'
|
||||
| 'AUTH_REVOKED'
|
||||
| 'ACCOUNT_NOT_FOUND'
|
||||
| 'ACCOUNT_DISABLED'
|
||||
| 'WECHAT_CODE_INVALID'
|
||||
| 'WECHAT_LOGIN_BLOCKED'
|
||||
| 'RATE_LIMITED'
|
||||
| 'AUTH_UPSTREAM_UNAVAILABLE'
|
||||
| 'DATABASE_UNAVAILABLE';
|
||||
|
||||
export interface ApiError {
|
||||
code: number;
|
||||
message: string;
|
||||
requestId?: string;
|
||||
reason?: ApiErrorReason;
|
||||
}
|
||||
|
||||
export type WechatLoginMode = 'start' | 'resume';
|
||||
export interface WechatLoginRequest {
|
||||
code: string;
|
||||
mode: WechatLoginMode;
|
||||
}
|
||||
|
||||
/** 显式白名单摘要，不含微信身份或凭证版本。头像签名由后续资产模块补齐。 */
|
||||
export interface UserSummary {
|
||||
id: string;
|
||||
nickName: string | null;
|
||||
avatarUrl: string | null;
|
||||
avatarUrlExpiresAt: string | null;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface WechatLoginResult {
|
||||
accessToken: string;
|
||||
tokenType: 'Bearer';
|
||||
expiresIn: number;
|
||||
expiresAt: string;
|
||||
user: UserSummary;
|
||||
}
|
||||
|
||||
export interface CurrentUserProfile extends UserSummary {
|
||||
role: 'user' | 'admin';
|
||||
}
|
||||
|
||||
export type ContentStatus = 'draft' | 'active' | 'hidden';
|
||||
|
||||
/** 分类 ID 沿用云数据库原值；列表为扁平结构，parentId=null 表示根分类。 */
|
||||
export interface Category {
|
||||
status: ContentStatus;
|
||||
id: string;
|
||||
name: string;
|
||||
icon: string;
|
||||
color: string;
|
||||
sortOrder: number;
|
||||
parentId: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface CreateCategoryRequest {
|
||||
status?: ContentStatus;
|
||||
id: string;
|
||||
name: string;
|
||||
icon: string;
|
||||
color: string;
|
||||
sortOrder?: number;
|
||||
parentId?: string | null;
|
||||
}
|
||||
|
||||
/** PATCH 不允许修改 ID；省略父级保持原值，null 或空串移到根级。 */
|
||||
export type UpdateCategoryRequest = Partial<Omit<CreateCategoryRequest, 'id'>>;
|
||||
|
||||
export type WorksheetAgeBand = '3-4' | '4-5' | '5-6' | '6-7' | '7-8';
|
||||
export type WorksheetSort = 'default' | 'latest' | 'hot';
|
||||
|
||||
/** 筛选条件可组合；grade 与 grades、age 与 ageBand 分别互斥。 */
|
||||
export interface WorksheetQuery {
|
||||
page?: number;
|
||||
pageSize?: number;
|
||||
categoryId?: string;
|
||||
grade?: number;
|
||||
grades?: number[];
|
||||
ageBand?: WorksheetAgeBand;
|
||||
age?: number;
|
||||
tag?: string;
|
||||
sort?: WorksheetSort;
|
||||
isHot?: boolean;
|
||||
isNew?: boolean;
|
||||
}
|
||||
|
||||
/** 公开题型白名单；64 位展示计数使用十进制字符串，避免 JavaScript 精度丢失。 */
|
||||
export interface PublicWorksheet {
|
||||
id: string;
|
||||
title: string;
|
||||
subtitle: string;
|
||||
category: string;
|
||||
subcategory: string | null;
|
||||
path: string;
|
||||
ageMin: number;
|
||||
ageMax: number;
|
||||
grades: number[];
|
||||
difficulty: number;
|
||||
previewImg: string | null;
|
||||
tags: string[];
|
||||
isNew: boolean;
|
||||
isHot: boolean;
|
||||
sortOrder: number;
|
||||
downloads: string;
|
||||
likes: string;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
contentUpdatedAt: string | null;
|
||||
firstPublishedAt: string | null;
|
||||
}
|
||||
|
||||
export interface WorksheetPage {
|
||||
items: PublicWorksheet[];
|
||||
total: number;
|
||||
page: number;
|
||||
pageSize: number;
|
||||
hasNext: boolean;
|
||||
}
|
||||
|
||||
/** 管理预览沿用公开字段白名单，额外标明自身状态和整条所属关系是否可见。 */
|
||||
export interface AdminWorksheet extends PublicWorksheet {
|
||||
status: ContentStatus;
|
||||
visible: boolean;
|
||||
}
|
||||
export interface AdminWorksheetPage extends Omit<WorksheetPage, 'items'> {
|
||||
items: AdminWorksheet[];
|
||||
}
|
||||
export interface UpdateContentStatusRequest {
|
||||
status: ContentStatus;
|
||||
}
|
||||
export interface ContentPublication {
|
||||
id: string;
|
||||
status: ContentStatus;
|
||||
firstPublishedAt: string | null;
|
||||
updatedAt: string;
|
||||
}
|
||||
/** 资源管理预览不返回配置覆盖、源文件地址、私有对象或签名 URL。 */
|
||||
export interface AdminWorksheetResource extends ContentPublication {
|
||||
worksheetId: string;
|
||||
title: string;
|
||||
previewImg: string | null;
|
||||
sortOrder: number;
|
||||
downloads: string;
|
||||
createdAt: string;
|
||||
visible: boolean;
|
||||
}
|
||||
export interface AdminResourcePage {
|
||||
items: AdminWorksheetResource[];
|
||||
total: number;
|
||||
page: number;
|
||||
pageSize: number;
|
||||
hasNext: boolean;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user