数据中心同步

This commit is contained in:
乘风
2026-01-05 10:16:20 +08:00
parent 0457528dd0
commit ba0ebcf273
98 changed files with 28583 additions and 0 deletions

View File

@@ -0,0 +1,233 @@
# Moncter MCP 服务器使用说明
## 概述
Moncter MCP Server 是一个基于 Model Context Protocol (MCP) 的服务器,允许通过 MCP 协议来管理 Moncter 系统的数据采集任务和标签任务。
## 安装步骤
### 1. 安装依赖
```bash
cd MCP/moncter-mcp-server
npm install
```
### 2. 编译 TypeScript
```bash
npm run build
```
### 3. 配置 MCP
编辑 `MCP/mcp.json` 文件,确保 Moncter MCP 服务器配置正确:
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["./MCP/moncter-mcp-server/dist/index.js"],
"cwd": "E:/Cunkebao/Cunkebao02/Moncter",
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
**注意**`cwd` 路径需要根据实际项目路径修改。
## 可用的 MCP 工具
### 1. create_data_collection_task
创建数据采集任务。
**参数**
- `name` (string, 必需): 任务名称
- `description` (string, 可选): 任务描述
- `data_source_id` (string, 必需): 数据源ID
- `database` (string, 必需): 数据库名称
- `collection` (string, 可选): 集合名称(单集合模式)
- `collections` (array, 可选): 集合列表(多集合模式)
- `mode` (string, 必需): 采集模式batch/realtime
- `field_mappings` (array, 可选): 字段映射配置
- `schedule` (object, 可选): 调度配置
**示例**
```json
{
"name": "create_data_collection_task",
"arguments": {
"name": "订单数据采集",
"data_source_id": "data_source_id_123",
"database": "KR_商城",
"collection": "21年贝蒂喜订单整合",
"mode": "realtime"
}
}
```
### 2. create_tag_task
创建标签计算任务。
**参数**
- `name` (string, 必需): 任务名称
- `task_type` (string, 必需): 任务类型full/incremental/specific
- `target_tag_ids` (array, 必需): 要计算的标签ID列表
- `user_scope` (object, 可选): 用户范围配置
- `schedule` (object, 可选): 调度配置
- `config` (object, 可选): 高级配置
**示例**
```json
{
"name": "create_tag_task",
"arguments": {
"name": "高价值用户标签计算",
"task_type": "full",
"target_tag_ids": ["tag_id_1", "tag_id_2"],
"user_scope": {
"type": "all"
}
}
}
```
### 3. list_data_collection_tasks
获取数据采集任务列表。
**参数**
- `page` (number, 可选): 页码
- `page_size` (number, 可选): 每页数量
### 4. list_tag_tasks
获取标签任务列表。
**参数**
- `page` (number, 可选): 页码
- `page_size` (number, 可选): 每页数量
### 5. get_data_sources
获取数据源列表。
**参数**
- `type` (string, 可选): 数据源类型筛选
- `status` (number, 可选): 状态筛选1=启用0=禁用)
### 6. get_tag_definitions
获取标签定义列表。
**参数**
- `status` (number, 可选): 状态筛选1=启用0=禁用)
### 7. start_data_collection_task
启动数据采集任务。
**参数**
- `task_id` (string, 必需): 任务ID
### 8. start_tag_task
启动标签任务。
**参数**
- `task_id` (string, 必需): 任务ID
## 环境变量
- `MONCTER_API_URL`: 后端API基础URL默认: http://127.0.0.1:8787
## 使用场景
### 场景1通过 AI 助手创建数据采集任务
你可以通过支持 MCP 的 AI 助手(如 Claude Desktop来创建任务
1. 告诉 AI"创建一个实时监听的数据采集任务,从数据源 X 的数据库 Y 的集合 Z 采集数据"
2. AI 会调用 `create_data_collection_task` 工具
3. 任务创建成功后AI 会告诉你任务ID和状态
### 场景2批量创建标签任务
通过 MCP 工具批量创建多个标签计算任务:
1. 列出所有标签定义:`get_tag_definitions`
2. 为每个标签创建计算任务:`create_tag_task`
3. 启动所有任务:`start_tag_task`
### 场景3任务管理
通过 MCP 工具查询和管理任务:
1. 列出所有任务:`list_data_collection_tasks` / `list_tag_tasks`
2. 查看任务详情和状态
3. 启动/暂停/停止任务
## 开发调试
### 开发模式
```bash
cd MCP/moncter-mcp-server
npm run dev
```
### 测试 MCP 服务器
可以使用 MCP Inspector 或其他 MCP 客户端工具来测试服务器:
```bash
# 如果安装了 @modelcontextprotocol/inspector
npx @modelcontextprotocol/inspector node dist/index.js
```
## 故障排除
### 问题1服务器无法启动
- 检查 Node.js 版本(需要 >= 18
- 检查是否已安装依赖:`npm install`
- 检查是否已编译:`npm run build`
### 问题2无法连接到后端API
- 检查 `MONCTER_API_URL` 环境变量是否正确
- 检查后端服务是否运行在指定端口
- 检查防火墙和网络连接
### 问题3工具调用失败
- 检查后端API接口是否正常
- 检查参数是否正确
- 查看服务器日志输出
## 扩展开发
要添加新的 MCP 工具:
1.`src/index.ts``ListToolsRequestSchema` handler 中添加新工具定义
2.`CallToolRequestSchema` handler 中添加工具处理逻辑
3. 重新编译:`npm run build`
4. 重启 MCP 服务器
## 注意事项
1. **API URL 配置**:确保 `MONCTER_API_URL` 指向正确的后端服务地址
2. **路径配置**`mcp.json` 中的 `cwd``args` 路径需要根据实际项目路径调整
3. **权限**MCP 工具调用会直接操作后端API请确保权限控制
4. **错误处理**:工具调用失败时会返回错误信息,请检查返回内容
---
**更新时间**2025-01-24

70
Moncter/MCP/README.md Normal file
View File

@@ -0,0 +1,70 @@
# Moncter MCP 集成
这个目录包含 Moncter 系统的 MCP (Model Context Protocol) 服务器实现。
## 目录结构
```
MCP/
├── mcp.json # MCP 服务器配置文件(需要配置路径)
├── mcp.json.example # MCP 配置文件示例
├── moncter-mcp-server/ # Moncter MCP 服务器源代码
│ ├── src/
│ │ └── index.ts # 服务器主文件
│ ├── package.json # Node.js 依赖配置
│ ├── tsconfig.json # TypeScript 配置
│ ├── install.sh # Linux/Mac 安装脚本
│ ├── install.bat # Windows 安装脚本
│ └── README.md # 服务器详细文档
├── 快速开始.md # 快速开始指南
└── MCP服务器使用说明.md # 详细使用说明
```
## 快速开始
1. **安装 MCP Server**
```bash
cd MCP/moncter-mcp-server
npm install
npm run build
```
2. **配置 MCP 客户端**
编辑 `MCP/mcp.json`,将 `YOUR_PROJECT_PATH` 替换为实际的项目路径。
3. **启动后端服务**
```bash
php start.php start
```
4. **使用 MCP 工具**
在支持 MCP 的 AI 客户端(如 Claude Desktop中使用 Moncter MCP 服务器提供的工具。
## 可用的 MCP 工具
- `create_data_collection_task` - 创建数据采集任务
- `create_tag_task` - 创建标签计算任务
- `list_data_collection_tasks` - 获取数据采集任务列表
- `list_tag_tasks` - 获取标签任务列表
- `get_data_sources` - 获取数据源列表
- `get_tag_definitions` - 获取标签定义列表
- `start_data_collection_task` - 启动数据采集任务
- `start_tag_task` - 启动标签任务
## 文档
- [快速开始指南](./快速开始.md) - 安装和配置步骤
- [详细使用说明](./MCP服务器使用说明.md) - 完整的工具说明和使用示例
- [服务器 README](./moncter-mcp-server/README.md) - 服务器开发文档
## 注意事项
1. 确保 Node.js 版本 >= 18
2. 确保后端服务运行在配置的端口(默认 8787
3. 配置文件中的路径需要根据实际情况修改
4. MCP 工具调用会直接操作后端API请确保权限控制

View File

@@ -0,0 +1,27 @@
{
"mcpServers": {
"MongoDB_ckb": {
"command": "npx",
"args": ["-y", "mongodb-mcp-server@1.2.0", "--readOnly"],
"env": {
"MDB_MCP_CONNECTION_STRING": "mongodb://ckb:123456@192.168.1.106:27017/ckb"
}
},
"MongoDB_KR": {
"command": "npx",
"args": ["-y", "mongodb-mcp-server@1.2.0", "--readOnly"],
"env": {
"MDB_MCP_CONNECTION_STRING": "mongodb://admin:key123456@192.168.2.16:27017/admin"
}
},
"Moncter": {
"command": "node",
"args": ["./MCP/moncter-mcp-server/dist/index.js"],
"cwd": "YOUR_PROJECT_PATH",
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}

View File

@@ -0,0 +1,7 @@
node_modules/
dist/
*.log
.DS_Store
.env
*.tsbuildinfo

View File

@@ -0,0 +1,194 @@
# MCP服务器接口对比分析
## 问题分析
对比标签引擎的MCP服务器和实际的采集任务接口发现以下差异
---
## 一、后端接口要求DataCollectionTaskController
### 必填字段从Controller验证逻辑看
```php
$requiredFields = ['name', 'data_source_id', 'database', 'target_data_source_id', 'target_database', 'target_collection'];
```
### 可选但支持的字段:
- `target_type` - 目标类型consumption_record 或 generic
- `mode` - 采集模式batch 或 realtime
- `collection` - 单集合模式
- `collections` - 多集合模式与collection二选一
- `multi_collection` - 是否多集合模式
- `field_mappings` - 字段映射(单集合模式)
- `collection_field_mappings` - 字段映射(多集合模式)
- `lookups` - 连表查询配置(单集合模式)
- `collection_lookups` - 连表查询配置(多集合模式)
- `filter_conditions` - 过滤条件
- `schedule` - 调度配置
- `description` - 任务描述
---
## 二、前端TaskForm实际使用的字段
根据 `TaskShow/src/views/DataCollection/TaskForm.vue`,前端表单包含:
```typescript
{
name: string
description: string
data_source_id: string
database: string
collection?: string // 单集合模式
collections?: string[] // 多集合模式
multi_collection: boolean
target_type: string // 'consumption_record' | 'generic'
mode: 'batch' | 'realtime'
field_mappings: FieldMapping[]
collection_field_mappings?: Record<string, FieldMapping[]>
lookups?: LookupConfig[]
collection_lookups?: Record<string, LookupConfig[]>
filter_conditions?: FilterCondition[]
schedule: {
enabled: boolean
cron?: string
}
}
```
**注意**:前端**没有** `target_data_source_id`, `target_database`, `target_collection` 字段!
---
## 三、MCP服务器当前支持的字段
根据 `MCP/moncter-mcp-server/src/index.ts`,当前支持:
```typescript
{
name: string
description: string
data_source_id: string
database: string
collection?: string
collections?: string[]
mode: 'batch' | 'realtime'
field_mappings: FieldMapping[]
schedule: {
enabled: boolean
cron?: string
}
}
```
---
## 四、字段差异对比
### ❌ MCP服务器缺少的字段
1. **`target_type`** - 目标类型(重要!)
- 前端使用:`'consumption_record'``'generic'`
- 用途决定使用哪个Handler处理数据
- 优先级:**高**
2. **`multi_collection`** - 多集合模式标识
- 前端使用:`boolean`
- 用途:区分单集合和多集合模式
- 优先级:**中**
3. **`filter_conditions`** - 过滤条件
- 前端使用:`FilterCondition[]`
- 用途:数据采集的过滤条件
- 优先级:**中**
4. **`lookups`** - 连表查询配置(单集合模式)
- 前端使用:`LookupConfig[]`
- 用途MongoDB $lookup 连表查询
- 优先级:**低**
5. **`collection_lookups`** - 连表查询配置(多集合模式)
- 前端使用:`Record<string, LookupConfig[]>`
- 用途:多集合模式下的连表查询
- 优先级:**低**
6. **`collection_field_mappings`** - 字段映射(多集合模式)
- 前端使用:`Record<string, FieldMapping[]>`
- 用途:多集合模式下每个集合的字段映射
- 优先级:**中**
### ⚠️ 后端Controller要求的字段但前端和Service都没使用
1. **`target_data_source_id`** - 目标数据源ID
2. **`target_database`** - 目标数据库
3. **`target_collection`** - 目标集合
**分析**
- Controller的验证逻辑要求这些字段必填
- 但Service的`createTask`方法并没有使用这些字段
- 前端TaskForm也没有这些字段
- 对于`consumption_record`类型Handler会自动处理存储不需要指定目标
- 对于`generic`类型,可能需要这些字段
**结论**:这些字段可能是历史遗留,或者仅对`generic`类型必需,对`consumption_record`类型不是必需的。需要确认后端逻辑。
---
## 五、建议的修复方案
### 方案1更新MCP服务器添加缺失字段
**优先添加的字段**
1.`target_type` - **必需**
2.`multi_collection` - **推荐**
3.`filter_conditions` - **推荐**
4.`collection_field_mappings` - **推荐**(支持多集合模式)
5. ⚠️ `lookups` - 可选
6. ⚠️ `collection_lookups` - 可选
**暂不处理**
- `target_data_source_id`, `target_database`, `target_collection` - 需要先确认后端逻辑
### 方案2确认后端接口的实际要求
需要确认:
1. `target_data_source_id`, `target_database`, `target_collection` 是否真的必需?
2. 对于`consumption_record`类型,这些字段是否可选?
3. 如果必需MCP服务器需要添加这些字段
---
## 六、当前状态总结
### ✅ MCP服务器已支持的字段
- name
- description
- data_source_id
- database
- collection / collections
- mode
- field_mappings
- schedule
### ❌ MCP服务器缺少的字段按优先级
1. **target_type** - ⚠️ 重要决定Handler类型
2. **multi_collection** - 区分单/多集合模式
3. **filter_conditions** - 数据过滤条件
4. **collection_field_mappings** - 多集合模式的字段映射
5. **lookups** - 连表查询(可选)
6. **collection_lookups** - 多集合连表查询(可选)
### ⚠️ 需要确认的字段:
- target_data_source_id
- target_database
- target_collection
---
## 七、下一步行动
1. ✅ 对比分析完成
2. ⏳ 更新MCP服务器添加缺失字段
3. ⏳ 确认后端接口对target相关字段的要求
4. ⏳ 测试更新后的MCP服务器

View File

@@ -0,0 +1,342 @@
# MCP服务器同步更新说明
## 更新日期
2025年12月
## 更新背景
根据最新的 `TaskForm.vue` 界面变更MCP服务器需要同步更新以匹配最新的界面逻辑。
---
## 主要变更
### ✅ 移除的字段
根据最新的TaskForm.vue以下字段已经从界面中移除MCP服务器也已移除
1. **`target_data_source_id`** - 目标数据源ID
2. **`target_database`** - 目标数据库
3. **`target_collection`** - 目标集合
**原因**
- 对于 `consumption_record` 类型Handler会自动处理存储到标签引擎数据库不需要指定目标
- 对于 `generic` 类型如果需要指定目标应该在Handler配置或业务逻辑中处理
- 界面已经简化,不再要求用户配置这些字段
### ✅ 保留和优化的字段
#### 核心字段(必填)
1. **`name`** - 任务名称
2. **`data_source_id`** - 源数据源ID
3. **`database`** - 源数据库名称
4. **`mode`** - 采集模式(`batch``realtime`
5. **`target_type`** - 目标类型(`consumption_record``generic`
#### 集合配置字段
6. **`collection`** - 源集合名称(单集合模式)
7. **`collections`** - 源集合列表(多集合模式)
8. **`multi_collection`** - 是否启用多集合模式
**说明**`collection``collections` 二选一,由 `multi_collection` 字段决定使用哪个。
#### 字段映射字段
9. **`field_mappings`** - 字段映射配置(单集合模式)
- 格式:`[{ source_field, target_field, transform? }]`
- 转换函数:`parse_amount`, `parse_datetime`, `parse_phone`
10. **`collection_field_mappings`** - 字段映射配置(多集合模式)
- 格式:`{ "collection_name": [FieldMapping] }`
- 每个集合可配置独立的字段映射
#### 查询配置字段
11. **`lookups`** - MongoDB $lookup连表查询配置单集合模式
- 格式:`[{ from, local_field, foreign_field, as, unwrap?, preserve_null? }]`
12. **`collection_lookups`** - MongoDB $lookup连表查询配置多集合模式
- 格式:`{ "collection_name": [LookupConfig] }`
13. **`filter_conditions`** - 过滤条件
- 格式:`[{ field, operator, value }]`
- 运算符:`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `nin`
#### 调度配置字段
14. **`schedule`** - 调度配置(批量模式使用)
- `enabled`: 是否启用调度
- `cron`: Cron表达式
---
## 字段使用指南
### 1. 消费记录采集任务consumption_record
**适用场景**:采集订单、交易等消费记录数据
**特点**
- Handler会自动转换数据格式
- 通过手机号/身份证自动解析用户ID
- 自动时间分片存储(按月分表)
- 存储到标签引擎数据库
**示例**
```json
{
"name": "订单数据采集",
"description": "从KR商城采集订单数据",
"data_source_id": "source_123",
"database": "KR_商城",
"collection": "21年贝蒂喜订单整合",
"multi_collection": false,
"target_type": "consumption_record",
"mode": "batch",
"field_mappings": [
{
"source_field": "联系手机",
"target_field": "phone_number",
"transform": "parse_phone"
},
{
"source_field": "买家实际支付金额",
"target_field": "actual_amount",
"transform": "parse_amount"
},
{
"source_field": "订单付款时间",
"target_field": "consume_time",
"transform": "parse_datetime"
},
{
"source_field": "店铺名称",
"target_field": "store_name"
}
],
"filter_conditions": [
{
"field": "买家实际支付金额",
"operator": "ne",
"value": "0"
},
{
"field": "订单付款时间",
"operator": "ne",
"value": null
}
],
"schedule": {
"enabled": true,
"cron": "0 2 * * *"
}
}
```
### 2. 通用集合采集任务generic
**适用场景**:采集任意数据并存储到指定集合
**特点**
- 需要自定义字段映射
- 需要指定目标存储在Handler配置中
**示例**
```json
{
"name": "通用数据采集",
"description": "采集用户数据",
"data_source_id": "source_123",
"database": "user_db",
"collection": "users",
"multi_collection": false,
"target_type": "generic",
"mode": "realtime",
"field_mappings": [
{
"source_field": "user_name",
"target_field": "name"
},
{
"source_field": "user_email",
"target_field": "email"
}
],
"schedule": {
"enabled": false
}
}
```
### 3. 多集合模式
**适用场景**:同时从多个集合采集数据
**示例**
```json
{
"name": "多集合数据采集",
"data_source_id": "source_123",
"database": "multi_db",
"multi_collection": true,
"collections": ["collection1", "collection2"],
"target_type": "generic",
"mode": "batch",
"collection_field_mappings": {
"collection1": [
{
"source_field": "field1",
"target_field": "target1"
}
],
"collection2": [
{
"source_field": "field2",
"target_field": "target2"
}
]
},
"collection_lookups": {
"collection1": [
{
"from": "related_collection",
"local_field": "related_id",
"foreign_field": "_id",
"as": "related_data",
"unwrap": false,
"preserve_null": true
}
]
},
"schedule": {
"enabled": true,
"cron": "0 3 * * *"
}
}
```
### 4. 连表查询配置
**适用场景**:需要从其他集合关联数据
**示例**
```json
{
"lookups": [
{
"from": "user_info",
"local_field": "user_id",
"foreign_field": "_id",
"as": "user_info",
"unwrap": true,
"preserve_null": false
}
]
}
```
**说明**
- `unwrap: true` - 解构后可直接使用 `user_info.mobile`
- `unwrap: false` - 返回数组 `user_info[0].mobile`
---
## 与界面的对应关系
| MCP字段 | TaskForm字段 | 界面位置 | 说明 |
|---------|-------------|---------|------|
| name | form.name | 步骤1基本信息 | 任务名称 |
| description | form.description | 步骤1基本信息 | 任务描述 |
| mode | form.mode | 步骤1基本信息 | 采集模式 |
| target_type | form.target_type | 步骤2Handler配置 | 数据处理方式 |
| data_source_id | form.data_source_id | 步骤3源数据配置 | 数据源 |
| database | form.database | 步骤3源数据配置 | 数据库 |
| collection | form.collection | 步骤3源数据配置 | 集合(单集合) |
| collections | form.collections | 步骤3源数据配置 | 集合列表(多集合) |
| multi_collection | form.multi_collection | 步骤3源数据配置 | 多集合模式开关 |
| lookups | form.lookups | 步骤3连表查询 | 连表查询配置 |
| filter_conditions | form.filter_conditions | 步骤3过滤条件 | 过滤条件 |
| field_mappings | form.field_mappings | 步骤4字段映射 | 字段映射(单集合) |
| collection_field_mappings | form.collection_field_mappings | 步骤4字段映射 | 字段映射(多集合) |
| schedule | form.schedule | 步骤5调度配置 | 调度配置 |
---
## 验证清单
✅ MCP服务器已更新完全匹配最新的TaskForm.vue界面逻辑
- [x] 移除了 `target_data_source_id`, `target_database`, `target_collection` 字段
- [x] 保留了所有界面使用的字段
- [x] 更新了字段描述,使其更清晰准确
- [x] 明确了字段的使用场景和条件要求
- [x] 支持单集合和多集合模式
- [x] 支持连表查询和过滤条件
- [x] 支持字段映射和转换函数
---
## 注意事项
1. **target_type是必填的**:必须明确指定是 `consumption_record` 还是 `generic`
2. **collection和collections二选一**:根据 `multi_collection` 决定使用哪个
3. **字段映射方式**
- 单集合模式使用 `field_mappings`
- 多集合模式使用 `collection_field_mappings`
4. **连表查询方式**
- 单集合模式使用 `lookups`
- 多集合模式使用 `collection_lookups`
5. **调度配置**:仅在 `mode=batch` 时使用
---
## 测试建议
1. **测试消费记录采集任务创建**
```json
{
"name": "测试任务",
"data_source_id": "test_source",
"database": "test_db",
"collection": "test_collection",
"target_type": "consumption_record",
"mode": "batch",
"field_mappings": [...],
"schedule": {"enabled": true, "cron": "0 2 * * *"}
}
```
2. **测试多集合模式**
```json
{
"name": "多集合测试",
"data_source_id": "test_source",
"database": "test_db",
"multi_collection": true,
"collections": ["coll1", "coll2"],
"target_type": "generic",
"mode": "batch",
"collection_field_mappings": {...}
}
```
3. **测试连表查询**
```json
{
"lookups": [{
"from": "related",
"local_field": "id",
"foreign_field": "_id",
"as": "related_data"
}]
}
```
---
## 总结
MCP服务器已成功同步更新完全匹配最新的TaskForm.vue界面逻辑。所有字段定义、使用方式和验证规则都与界面保持一致。

View File

@@ -0,0 +1,246 @@
# MCP服务器更新说明
## 更新日期
2025年12月
## 更新内容
### ✅ 已添加的字段
更新了 `create_data_collection_task` 工具,添加了以下缺失的字段:
1. **`target_type`** ⭐ **重要**
- 类型:`'consumption_record' | 'generic'`
- 必填:是
- 说明决定使用哪个Handler处理数据
- 用途:
- `consumption_record`: 使用ConsumptionCollectionHandler自动处理消费记录格式转换和存储
- `generic`: 使用GenericCollectionHandler支持自定义字段映射和目标存储
2. **`multi_collection`**
- 类型:`boolean`
- 必填:否
- 说明:是否启用多集合模式
3. **`target_data_source_id`**
- 类型:`string`
- 必填但后端Controller验证要求必填见注意事项
- 说明目标数据源ID通用Handler需要
4. **`target_database`**
- 类型:`string`
- 必填但后端Controller验证要求必填见注意事项
- 说明目标数据库通用Handler需要
5. **`target_collection`**
- 类型:`string`
- 必填但后端Controller验证要求必填见注意事项
- 说明目标集合通用Handler需要
6. **`collection_field_mappings`**
- 类型:`object`
- 必填:否
- 说明:多集合模式下的字段映射,格式:`{ "collection_name": [FieldMapping] }`
7. **`lookups`**
- 类型:`array`
- 必填:否
- 说明单集合模式下的MongoDB $lookup连表查询配置
- 结构:
```typescript
{
from: string, // 关联集合名
local_field: string, // 主集合字段
foreign_field: string, // 关联集合字段
as: string, // 结果字段名
unwrap?: boolean, // 是否解构
preserve_null?: boolean // 是否保留空值
}
```
8. **`collection_lookups`**
- 类型:`object`
- 必填:否
- 说明:多集合模式下的连表查询配置,格式:`{ "collection_name": [LookupConfig] }`
9. **`filter_conditions`**
- 类型:`array`
- 必填:否
- 说明:数据采集的过滤条件
- 结构:
```typescript
{
field: string,
operator: 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte' | 'in' | 'nin',
value: any
}
```
### 📝 更新的字段说明
- **`field_mappings`**: 添加了更详细的说明,明确是单集合模式的字段映射
### ✅ 更新的必填字段
- 新增 `target_type` 为必填字段这是最重要的字段决定Handler类型
---
## ⚠️ 注意事项
### 1. 后端Controller验证问题
**问题**
后端 `DataCollectionTaskController::create()` 方法要求以下字段必填:
- `target_data_source_id`
- `target_database`
- `target_collection`
**但实际情况**
- 前端 `TaskForm.vue` 没有这些字段
- Service的 `createTask()` 方法也没有使用这些字段
- 对于 `consumption_record` 类型Handler会自动处理存储不需要指定目标
**可能的原因**
1. Controller的验证逻辑过于严格
2. 这些字段仅对 `generic` 类型必需
3. 后端代码不一致Controller和Service不同步
**建议**
1. 如果使用 `consumption_record` 类型,可能需要传递空值或默认值
2. 如果使用 `generic` 类型,必须提供这些字段
3. 建议后端修改验证逻辑,根据 `target_type` 动态验证必填字段
### 2. 字段使用建议
**对于 consumption_record 类型**
```json
{
"name": "订单采集任务",
"data_source_id": "source_123",
"database": "KR_商城",
"collection": "21年贝蒂喜订单整合",
"target_type": "consumption_record",
"mode": "batch",
"field_mappings": [
{
"source_field": "联系手机",
"target_field": "phone_number",
"transform": "parse_phone"
},
{
"source_field": "买家实际支付金额",
"target_field": "actual_amount",
"transform": "parse_amount"
}
],
"filter_conditions": [
{
"field": "买家实际支付金额",
"operator": "ne",
"value": "0"
}
],
"schedule": {
"enabled": true,
"cron": "0 2 * * *"
}
}
```
**对于 generic 类型**
```json
{
"name": "通用数据采集",
"data_source_id": "source_123",
"database": "KR_商城",
"collection": "some_collection",
"target_type": "generic",
"target_data_source_id": "target_source_123",
"target_database": "target_db",
"target_collection": "target_collection",
"mode": "batch",
"field_mappings": [
{
"source_field": "source_field1",
"target_field": "target_field1"
}
],
"schedule": {
"enabled": false
}
}
```
---
## 📊 字段对比表
| 字段 | MCP服务器更新前 | MCP服务器更新后 | 前端 | 后端Controller | 优先级 |
|------|-------------------|-------------------|------|---------------|--------|
| name | ✅ | ✅ | ✅ | ✅ 必填 | 高 |
| data_source_id | ✅ | ✅ | ✅ | ✅ 必填 | 高 |
| database | ✅ | ✅ | ✅ | ✅ 必填 | 高 |
| collection/collections | ✅ | ✅ | ✅ | ✅ 必填 | 高 |
| mode | ✅ | ✅ | ✅ | ✅ | 高 |
| field_mappings | ✅ | ✅ | ✅ | ✅ | 中 |
| schedule | ✅ | ✅ | ✅ | ✅ | 中 |
| target_type | ❌ | ✅ **新增** | ✅ | ✅ | **高** ⭐ |
| multi_collection | ❌ | ✅ **新增** | ✅ | ✅ | 中 |
| filter_conditions | ❌ | ✅ **新增** | ✅ | ✅ | 中 |
| collection_field_mappings | ❌ | ✅ **新增** | ✅ | ✅ | 中 |
| lookups | ❌ | ✅ **新增** | ✅ | ✅ | 低 |
| collection_lookups | ❌ | ✅ **新增** | ✅ | ✅ | 低 |
| target_data_source_id | ❌ | ✅ **新增** | ❌ | ✅ 必填 | ⚠️ |
| target_database | ❌ | ✅ **新增** | ❌ | ✅ 必填 | ⚠️ |
| target_collection | ❌ | ✅ **新增** | ❌ | ✅ 必填 | ⚠️ |
---
## ✅ 更新后的状态
### 完全支持的字段:
- ✅ 基本字段name, description, data_source_id, database, collection/collections
- ✅ 模式配置mode, multi_collection
- ✅ 目标配置target_type, target_data_source_id, target_database, target_collection
- ✅ 字段映射field_mappings, collection_field_mappings
- ✅ 查询配置lookups, collection_lookups, filter_conditions
- ✅ 调度配置schedule
### ⚠️ 需要注意的问题:
- 后端Controller要求 `target_data_source_id`, `target_database`, `target_collection` 必填但前端和Service都没有使用
- 建议在使用MCP创建任务时对于 `consumption_record` 类型,可能需要传递这些字段的空值或默认值,或者后端需要修改验证逻辑
---
## 🔄 下一步建议
1. **后端验证逻辑优化**
- 根据 `target_type` 动态验证必填字段
- 对于 `consumption_record` 类型,不需要 `target_*` 字段
- 对于 `generic` 类型,需要 `target_*` 字段
2. **测试验证**
- 测试使用MCP创建 `consumption_record` 类型的任务
- 测试使用MCP创建 `generic` 类型的任务
- 验证所有新增字段是否正确传递
3. **文档更新**
- 更新MCP使用文档说明不同 `target_type` 的字段要求
- 添加使用示例
---
## 📝 总结
MCP服务器已经更新**基本符合**当前的采集任务接口要求。主要添加了:
1.**target_type** - 最重要的字段决定Handler类型
2.**multi_collection** - 支持多集合模式
3.**filter_conditions** - 支持数据过滤
4.**lookups/collection_lookups** - 支持连表查询
5.**collection_field_mappings** - 支持多集合字段映射
6.**target_* 字段** - 虽然前端没有但后端Controller要求已添加
**剩余问题**后端Controller的验证逻辑可能与实际使用不一致需要确认或调整。

View File

@@ -0,0 +1,122 @@
# Moncter MCP Server
Moncter MCP Server 是一个 Model Context Protocol (MCP) 服务器,用于通过 MCP 协议管理 Moncter 系统的数据采集任务和标签任务。
## 功能
提供以下 MCP 工具:
1. **create_data_collection_task** - 创建数据采集任务
2. **create_tag_task** - 创建标签计算任务
3. **list_data_collection_tasks** - 获取数据采集任务列表
4. **list_tag_tasks** - 获取标签任务列表
5. **get_data_sources** - 获取数据源列表
6. **get_tag_definitions** - 获取标签定义列表
7. **start_data_collection_task** - 启动数据采集任务
8. **start_tag_task** - 启动标签任务
## 安装
```bash
cd MCP/moncter-mcp-server
npm install
npm run build
```
## 配置
`mcp.json` 中配置服务器:
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["E:/Cunkebao/Cunkebao02/Moncter/MCP/moncter-mcp-server/dist/index.js"],
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
或者使用 npm 方式(如果全局安装):
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["./MCP/moncter-mcp-server/dist/index.js"],
"cwd": "E:/Cunkebao/Cunkebao02/Moncter",
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
## 环境变量
- `MONCTER_API_URL`: 后端API基础URL默认: http://127.0.0.1:8787
## 使用示例
### 创建数据采集任务
```json
{
"name": "create_data_collection_task",
"arguments": {
"name": "订单数据采集",
"description": "从KR商城采集订单数据",
"data_source_id": "your_data_source_id",
"database": "KR_商城",
"collection": "21年贝蒂喜订单整合",
"mode": "realtime",
"field_mappings": [
{
"source_field": "订单号",
"target_field": "order_no"
}
]
}
}
```
### 创建标签任务
```json
{
"name": "create_tag_task",
"arguments": {
"name": "高价值用户标签计算",
"description": "计算高价值用户标签",
"task_type": "full",
"target_tag_ids": ["tag_id_1", "tag_id_2"],
"user_scope": {
"type": "all"
},
"schedule": {
"enabled": true,
"cron": "0 2 * * *"
}
}
}
```
## 开发
```bash
# 开发模式(使用 tsx
npm run dev
# 编译
npm run build
# 运行
npm start
```

View File

@@ -0,0 +1,39 @@
@echo off
REM Moncter MCP Server 安装脚本 (Windows)
echo 正在安装 Moncter MCP Server...
REM 检查 Node.js
where node >nul 2>nul
if %ERRORLEVEL% NEQ 0 (
echo 错误: 未找到 Node.js请先安装 Node.js (^>= 18^)
exit /b 1
)
REM 安装依赖
echo 安装依赖...
call npm install
if %ERRORLEVEL% NEQ 0 (
echo 错误: npm install 失败
exit /b 1
)
REM 编译 TypeScript
echo 编译 TypeScript...
call npm run build
if %ERRORLEVEL% NEQ 0 (
echo 错误: 编译失败
exit /b 1
)
echo.
echo ✅ Moncter MCP Server 安装成功!
echo.
echo 使用说明:
echo 1. 确保后端服务运行在 http://127.0.0.1:8787
echo 2. 配置 MCP 客户端,添加 Moncter MCP 服务器
echo 3. 查看 MCP/MCP服务器使用说明.md 了解详细用法
echo.
pause

View File

@@ -0,0 +1,38 @@
#!/bin/bash
# Moncter MCP Server 安装脚本
echo "正在安装 Moncter MCP Server..."
# 检查 Node.js
if ! command -v node &> /dev/null; then
echo "错误: 未找到 Node.js请先安装 Node.js (>= 18)"
exit 1
fi
NODE_VERSION=$(node -v | cut -d'v' -f2 | cut -d'.' -f1)
if [ "$NODE_VERSION" -lt 18 ]; then
echo "错误: Node.js 版本过低,需要 >= 18"
exit 1
fi
# 安装依赖
echo "安装依赖..."
npm install
# 编译 TypeScript
echo "编译 TypeScript..."
npm run build
if [ $? -eq 0 ]; then
echo "✅ Moncter MCP Server 安装成功!"
echo ""
echo "使用说明:"
echo "1. 确保后端服务运行在 http://127.0.0.1:8787"
echo "2. 配置 MCP 客户端,添加 Moncter MCP 服务器"
echo "3. 查看 MCP/MCP服务器使用说明.md 了解详细用法"
else
echo "❌ 编译失败,请检查错误信息"
exit 1
fi

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,26 @@
{
"name": "moncter-mcp-server",
"version": "1.0.0",
"description": "MCP Server for Moncter - Data Collection and Tag Task Management",
"main": "dist/index.js",
"type": "module",
"scripts": {
"build": "tsc",
"start": "node dist/index.js",
"dev": "tsx src/index.ts"
},
"keywords": ["mcp", "moncter", "data-collection", "tag-task"],
"author": "",
"license": "MIT",
"dependencies": {
"@modelcontextprotocol/sdk": "^0.5.0",
"node-fetch": "^3.3.2"
},
"devDependencies": {
"@types/node": "^20.10.0",
"@types/node-fetch": "^2.6.11",
"tsx": "^4.7.0",
"typescript": "^5.3.3"
}
}

View File

@@ -0,0 +1,595 @@
#!/usr/bin/env node
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import {
CallToolRequestSchema,
ListToolsRequestSchema,
} from '@modelcontextprotocol/sdk/types.js';
// 获取后端API基础URL从环境变量或默认值
const API_BASE_URL = process.env.MONCTER_API_URL || 'http://127.0.0.1:8787';
/**
* HTTP请求辅助函数
*/
async function apiRequest(
method: string,
endpoint: string,
data?: any
): Promise<any> {
const url = `${API_BASE_URL}${endpoint}`;
const options: RequestInit = {
method,
headers: {
'Content-Type': 'application/json',
},
};
if (data && (method === 'POST' || method === 'PUT')) {
options.body = JSON.stringify(data);
}
try {
const response = await fetch(url, options);
const result = await response.json() as any;
if (result.code !== 0 && result.code !== undefined) {
throw new Error(result.message || 'API请求失败');
}
return result.data || result;
} catch (error: any) {
throw new Error(`API请求错误: ${error.message}`);
}
}
/**
* 创建MCP服务器
*/
const server = new Server(
{
name: 'moncter-mcp-server',
version: '1.0.0',
},
{
capabilities: {
tools: {},
},
}
);
// 列出可用工具
server.setRequestHandler(ListToolsRequestSchema, async () => {
return {
tools: [
{
name: 'create_data_collection_task',
description: '创建数据采集任务。用于从数据源采集数据并转换为消费记录或其他格式。支持单集合和多集合模式,支持连表查询和过滤条件。',
inputSchema: {
type: 'object',
properties: {
name: {
type: 'string',
description: '任务名称(必填)',
},
description: {
type: 'string',
description: '任务描述(可选)',
},
data_source_id: {
type: 'string',
description: '源数据源ID必填',
},
database: {
type: 'string',
description: '源数据库名称(必填)',
},
collection: {
type: 'string',
description: '源集合名称单集合模式与collections二选一',
},
collections: {
type: 'array',
items: { type: 'string' },
description: '源集合列表多集合模式与collection二选一',
},
multi_collection: {
type: 'boolean',
description: '是否启用多集合模式。true=使用collections字段false=使用collection字段',
},
target_type: {
type: 'string',
enum: ['consumption_record', 'generic'],
description: '目标类型必填consumption_record=消费记录处理自动转换格式通过手机号解析用户ID时间分片存储到标签引擎数据库generic=通用集合处理(需要自定义字段映射和目标存储配置)',
},
mode: {
type: 'string',
enum: ['batch', 'realtime'],
description: '采集模式必填batch=批量采集定时执行realtime=实时监听(持续监听数据变化)',
},
field_mappings: {
type: 'array',
description: '字段映射配置(单集合模式使用),将源字段映射到目标字段',
items: {
type: 'object',
properties: {
source_field: {
type: 'string',
description: '源字段名(查询结果中的字段)'
},
target_field: {
type: 'string',
description: '目标字段名Handler需要的字段名'
},
transform: {
type: 'string',
enum: ['parse_amount', 'parse_datetime', 'parse_phone'],
description: '转换函数可选parse_amount=解析金额parse_datetime=解析日期时间parse_phone=解析手机号'
},
},
required: ['source_field', 'target_field'],
},
},
collection_field_mappings: {
type: 'object',
description: '字段映射配置(多集合模式使用),格式:{ "collection_name": [FieldMapping] },每个集合可配置独立的字段映射',
additionalProperties: {
type: 'array',
items: {
type: 'object',
properties: {
source_field: { type: 'string' },
target_field: { type: 'string' },
transform: { type: 'string' },
},
},
},
},
lookups: {
type: 'array',
description: 'MongoDB $lookup连表查询配置单集合模式使用可选可以从其他集合关联数据',
items: {
type: 'object',
properties: {
from: {
type: 'string',
description: '关联集合名'
},
local_field: {
type: 'string',
description: '主集合字段(用于关联的字段)'
},
foreign_field: {
type: 'string',
description: '关联集合字段被关联集合的字段通常是_id'
},
as: {
type: 'string',
description: '结果字段名(关联结果存储的字段名)'
},
unwrap: {
type: 'boolean',
description: '是否解构true=解构后可直接使用user_info.mobilefalse=返回数组)'
},
preserve_null: {
type: 'boolean',
description: '是否保留空值(当关联不到数据时是否保留)'
},
},
required: ['from', 'local_field', 'foreign_field', 'as'],
},
},
collection_lookups: {
type: 'object',
description: 'MongoDB $lookup连表查询配置多集合模式使用可选格式{ "collection_name": [LookupConfig] },每个集合可配置独立的连表查询',
additionalProperties: {
type: 'array',
items: {
type: 'object',
properties: {
from: { type: 'string' },
local_field: { type: 'string' },
foreign_field: { type: 'string' },
as: { type: 'string' },
unwrap: { type: 'boolean' },
preserve_null: { type: 'boolean' },
},
},
},
},
filter_conditions: {
type: 'array',
description: '过滤条件(可选),只采集满足条件的数据',
items: {
type: 'object',
properties: {
field: {
type: 'string',
description: '字段名'
},
operator: {
type: 'string',
enum: ['eq', 'ne', 'gt', 'gte', 'lt', 'lte', 'in', 'nin'],
description: '运算符eq=等于ne=不等于gt=大于gte=大于等于lt=小于lte=小于等于in=在列表中nin=不在列表中'
},
value: {
type: ['string', 'number', 'boolean', 'array'],
description: '值(可以是字符串、数字、布尔值或数组)'
},
},
required: ['field', 'operator', 'value'],
},
},
schedule: {
type: 'object',
description: '调度配置批量模式batch时使用',
properties: {
enabled: {
type: 'boolean',
description: '是否启用调度批量模式时可启用Cron定时执行'
},
cron: {
type: 'string',
description: 'Cron表达式格式分 时 日 月 周例如0 2 * * * 表示每天凌晨2点执行'
},
},
},
},
required: ['name', 'data_source_id', 'database', 'mode', 'target_type'],
},
},
{
name: 'create_tag_task',
description: '创建标签计算任务。用于批量计算用户标签。',
inputSchema: {
type: 'object',
properties: {
name: {
type: 'string',
description: '任务名称',
},
description: {
type: 'string',
description: '任务描述',
},
task_type: {
type: 'string',
enum: ['full', 'incremental', 'specific'],
description: '任务类型full=全量计算incremental=增量计算specific=指定用户',
},
target_tag_ids: {
type: 'array',
items: { type: 'string' },
description: '要计算的标签ID列表',
},
user_scope: {
type: 'object',
properties: {
type: {
type: 'string',
enum: ['all', 'list', 'filter'],
description: '用户范围类型',
},
user_ids: {
type: 'array',
items: { type: 'string' },
description: '用户ID列表当type=list时',
},
filter_conditions: {
type: 'array',
description: '筛选条件当type=filter时',
items: {
type: 'object',
properties: {
field: { type: 'string' },
operator: { type: 'string' },
value: { type: 'string' },
},
},
},
},
},
schedule: {
type: 'object',
properties: {
enabled: { type: 'boolean' },
cron: { type: 'string' },
},
},
config: {
type: 'object',
properties: {
concurrency: { type: 'number' },
batch_size: { type: 'number' },
error_handling: {
type: 'string',
enum: ['skip', 'stop', 'retry'],
},
},
},
},
required: ['name', 'task_type', 'target_tag_ids'],
},
},
{
name: 'list_data_collection_tasks',
description: '获取数据采集任务列表',
inputSchema: {
type: 'object',
properties: {
page: { type: 'number', description: '页码' },
page_size: { type: 'number', description: '每页数量' },
},
},
},
{
name: 'list_tag_tasks',
description: '获取标签任务列表',
inputSchema: {
type: 'object',
properties: {
page: { type: 'number', description: '页码' },
page_size: { type: 'number', description: '每页数量' },
},
},
},
{
name: 'get_data_sources',
description: '获取数据源列表',
inputSchema: {
type: 'object',
properties: {
type: { type: 'string', description: '数据源类型筛选' },
status: { type: 'number', description: '状态筛选1=启用0=禁用' },
},
},
},
{
name: 'get_tag_definitions',
description: '获取标签定义列表',
inputSchema: {
type: 'object',
properties: {
status: { type: 'number', description: '状态筛选1=启用0=禁用' },
},
},
},
{
name: 'start_data_collection_task',
description: '启动数据采集任务',
inputSchema: {
type: 'object',
properties: {
task_id: {
type: 'string',
description: '任务ID',
},
},
required: ['task_id'],
},
},
{
name: 'start_tag_task',
description: '启动标签任务',
inputSchema: {
type: 'object',
properties: {
task_id: {
type: 'string',
description: '任务ID',
},
},
required: ['task_id'],
},
},
],
};
});
// 处理工具调用
server.setRequestHandler(CallToolRequestSchema, async (request) => {
const { name, arguments: args } = request.params;
try {
switch (name) {
case 'create_data_collection_task': {
const result = await apiRequest(
'POST',
'/api/data-collection-tasks',
args
);
return {
content: [
{
type: 'text',
text: JSON.stringify(
{
success: true,
message: '数据采集任务创建成功',
data: result,
},
null,
2
),
},
],
};
}
case 'create_tag_task': {
const result = await apiRequest('POST', '/api/tag-tasks', args);
return {
content: [
{
type: 'text',
text: JSON.stringify(
{
success: true,
message: '标签任务创建成功',
data: result,
},
null,
2
),
},
],
};
}
case 'list_data_collection_tasks': {
const params = new URLSearchParams();
if (args?.page) params.append('page', String(args.page));
if (args?.page_size) params.append('page_size', String(args.page_size));
const query = params.toString();
const endpoint = `/api/data-collection-tasks${query ? `?${query}` : ''}`;
const result = await apiRequest('GET', endpoint);
return {
content: [
{
type: 'text',
text: JSON.stringify(result, null, 2),
},
],
};
}
case 'list_tag_tasks': {
const params = new URLSearchParams();
if (args?.page) params.append('page', String(args.page));
if (args?.page_size) params.append('page_size', String(args.page_size));
const query = params.toString();
const endpoint = `/api/tag-tasks${query ? `?${query}` : ''}`;
const result = await apiRequest('GET', endpoint);
return {
content: [
{
type: 'text',
text: JSON.stringify(result, null, 2),
},
],
};
}
case 'get_data_sources': {
const params = new URLSearchParams();
if (args?.type) params.append('type', String(args.type));
if (args?.status !== undefined) params.append('status', String(args.status));
const query = params.toString();
const endpoint = `/api/data-sources${query ? `?${query}` : ''}`;
const result = await apiRequest('GET', endpoint);
return {
content: [
{
type: 'text',
text: JSON.stringify(result, null, 2),
},
],
};
}
case 'get_tag_definitions': {
const params = new URLSearchParams();
if (args?.status !== undefined) params.append('status', String(args.status));
const query = params.toString();
const endpoint = `/api/tag-definitions${query ? `?${query}` : ''}`;
const result = await apiRequest('GET', endpoint);
return {
content: [
{
type: 'text',
text: JSON.stringify(result, null, 2),
},
],
};
}
case 'start_data_collection_task': {
if (!args?.task_id) {
throw new Error('缺少必需参数: task_id');
}
const result = await apiRequest(
'POST',
`/api/data-collection-tasks/${args.task_id}/start`,
{}
);
return {
content: [
{
type: 'text',
text: JSON.stringify(
{
success: true,
message: '数据采集任务启动成功',
data: result,
},
null,
2
),
},
],
};
}
case 'start_tag_task': {
if (!args?.task_id) {
throw new Error('缺少必需参数: task_id');
}
const result = await apiRequest(
'POST',
`/api/tag-tasks/${args.task_id}/start`,
{}
);
return {
content: [
{
type: 'text',
text: JSON.stringify(
{
success: true,
message: '标签任务启动成功',
data: result,
},
null,
2
),
},
],
};
}
default:
throw new Error(`未知的工具: ${name}`);
}
} catch (error: any) {
return {
content: [
{
type: 'text',
text: JSON.stringify(
{
success: false,
error: error.message,
},
null,
2
),
},
],
isError: true,
};
}
});
// 启动服务器
async function main() {
const transport = new StdioServerTransport();
await server.connect(transport);
console.error('Moncter MCP Server running on stdio');
}
main().catch((error) => {
console.error('服务器启动失败:', error);
process.exit(1);
});

View File

@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"lib": ["ES2022"],
"moduleResolution": "node",
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

164
Moncter/MCP/实现总结.md Normal file
View File

@@ -0,0 +1,164 @@
# Moncter MCP 服务器实现总结
## 一、实现概述
已成功为 Moncter 系统创建了一个 MCP (Model Context Protocol) 服务器,允许通过 MCP 协议来管理数据采集任务和标签任务。
## 二、已创建的文件
### 1. 核心代码
- `MCP/moncter-mcp-server/src/index.ts` - MCP 服务器主文件
- `MCP/moncter-mcp-server/package.json` - Node.js 依赖配置
- `MCP/moncter-mcp-server/tsconfig.json` - TypeScript 配置
### 2. 配置文件
- `MCP/mcp.json` - MCP 服务器配置(已更新,添加了 Moncter 服务器)
- `MCP/mcp.json.example` - 配置示例文件
### 3. 安装脚本
- `MCP/moncter-mcp-server/install.sh` - Linux/Mac 安装脚本
- `MCP/moncter-mcp-server/install.bat` - Windows 安装脚本
### 4. 文档
- `MCP/README.md` - MCP 目录说明
- `MCP/快速开始.md` - 快速开始指南
- `MCP/MCP服务器使用说明.md` - 详细使用说明
- `MCP/moncter-mcp-server/README.md` - 服务器开发文档
## 三、实现的 MCP 工具
### 1. 数据采集任务管理
-`create_data_collection_task` - 创建数据采集任务
-`list_data_collection_tasks` - 获取数据采集任务列表
-`start_data_collection_task` - 启动数据采集任务
### 2. 标签任务管理
-`create_tag_task` - 创建标签计算任务
-`list_tag_tasks` - 获取标签任务列表
-`start_tag_task` - 启动标签任务
### 3. 辅助工具
-`get_data_sources` - 获取数据源列表
-`get_tag_definitions` - 获取标签定义列表
## 四、技术实现
### 架构设计
```
MCP Client (Claude Desktop, etc.)
↓ (stdio/stdin)
Moncter MCP Server (Node.js)
↓ (HTTP REST API)
Moncter Backend (PHP/Webman)
MongoDB / Redis / RabbitMQ
```
### 关键技术
- **MCP SDK**: 使用 `@modelcontextprotocol/sdk` 实现 MCP 协议
- **HTTP 客户端**: 使用原生 `fetch` APINode.js 18+
- **TypeScript**: 类型安全的实现
- **Stdio Transport**: 通过标准输入输出与 MCP 客户端通信
## 五、安装和使用步骤
### 1. 安装
```bash
cd MCP/moncter-mcp-server
npm install
npm run build
```
### 2. 配置
编辑 `MCP/mcp.json`,确保路径正确:
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["./MCP/moncter-mcp-server/dist/index.js"],
"cwd": "YOUR_PROJECT_PATH",
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
### 3. 使用
在支持 MCP 的客户端(如 Claude Desktop
- 配置 MCP 服务器(引用 `mcp.json`
- 重启客户端
- 通过对话使用工具:"创建一个数据采集任务..."
## 六、功能特点
1. **完整的任务管理**: 支持创建、查询、启动数据采集任务和标签任务
2. **参数验证**: 通过 JSON Schema 验证工具参数
3. **错误处理**: 完善的错误处理和错误信息返回
4. **类型安全**: 使用 TypeScript 确保类型安全
5. **易于扩展**: 可以轻松添加新的 MCP 工具
## 七、后续扩展建议
### 可以添加的工具
1. **任务管理工具**:
- `update_data_collection_task` - 更新数据采集任务
- `delete_data_collection_task` - 删除数据采集任务
- `pause_data_collection_task` - 暂停数据采集任务
- `stop_data_collection_task` - 停止数据采集任务
- 类似的标签任务管理工具
2. **数据源管理工具**:
- `create_data_source` - 创建数据源
- `update_data_source` - 更新数据源
- `test_data_source_connection` - 测试数据源连接
3. **标签定义管理工具**:
- `create_tag_definition` - 创建标签定义
- `update_tag_definition` - 更新标签定义
4. **查询工具**:
- `get_task_detail` - 获取任务详情
- `get_task_progress` - 获取任务进度
- `get_task_executions` - 获取任务执行记录
5. **批量操作工具**:
- `batch_create_tasks` - 批量创建任务
- `batch_start_tasks` - 批量启动任务
## 八、注意事项
1. **Node.js 版本**: 需要 Node.js >= 18使用原生 fetch API
2. **后端服务**: 确保后端服务运行在配置的端口
3. **路径配置**: MCP 配置中的路径需要根据实际情况修改
4. **权限控制**: MCP 工具直接调用后端API需要考虑权限控制
5. **错误处理**: 工具调用失败时会返回错误信息,便于调试
## 九、测试建议
1. **单元测试**: 为各个工具函数编写单元测试
2. **集成测试**: 测试与后端API的集成
3. **端到端测试**: 使用 MCP Inspector 进行端到端测试
4. **错误场景测试**: 测试各种错误场景的处理
---
**实现完成时间**: 2025-01-24
**版本**: 1.0.0

212
Moncter/MCP/快速开始.md Normal file
View File

@@ -0,0 +1,212 @@
# Moncter MCP Server 快速开始
## 一、安装 MCP Server
### Windows
```bash
cd MCP/moncter-mcp-server
install.bat
```
### Linux/Mac
```bash
cd MCP/moncter-mcp-server
chmod +x install.sh
./install.sh
```
### 手动安装
```bash
cd MCP/moncter-mcp-server
npm install
npm run build
```
## 二、配置 MCP 客户端
### 方式1使用相对路径推荐
编辑 `MCP/mcp.json`,使用相对于项目根目录的路径:
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["./MCP/moncter-mcp-server/dist/index.js"],
"cwd": "E:/Cunkebao/Cunkebao02/Moncter",
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
**注意**`cwd` 需要修改为你的实际项目路径。
### 方式2使用绝对路径
```json
{
"mcpServers": {
"Moncter": {
"command": "node",
"args": ["E:/Cunkebao/Cunkebao02/Moncter/MCP/moncter-mcp-server/dist/index.js"],
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
### 方式3使用 npx如果发布到npm
```json
{
"mcpServers": {
"Moncter": {
"command": "npx",
"args": ["-y", "moncter-mcp-server"],
"env": {
"MONCTER_API_URL": "http://127.0.0.1:8787"
}
}
}
}
```
## 三、确保后端服务运行
确保 Moncter 后端服务正在运行:
```bash
# 检查服务状态
php start.php status
# 如果未运行,启动服务
php start.php start
```
默认端口:`8787`
## 四、测试 MCP Server
### 在 Claude Desktop 中使用
1. 打开 Claude Desktop
2. 在设置中添加 MCP 服务器配置(引用 `mcp.json`
3. 重启 Claude Desktop
4. 在对话中尝试:"列出所有数据源"
### 使用 MCP Inspector 测试
```bash
# 安装 MCP Inspector
npm install -g @modelcontextprotocol/inspector
# 测试服务器
cd MCP/moncter-mcp-server
npx @modelcontextprotocol/inspector node dist/index.js
```
## 五、使用示例
### 示例1创建数据采集任务
对 AI 说:
> "创建一个实时监听的数据采集任务,名称为'订单采集',从数据源'data_source_123'的数据库'KR_商城'的集合'21年贝蒂喜订单整合'采集数据"
AI 会调用 `create_data_collection_task` 工具来创建任务。
### 示例2创建标签任务
对 AI 说:
> "创建一个全量标签计算任务,名称为'高价值用户标签'计算所有标签每天凌晨2点执行"
AI 会调用 `create_tag_task` 工具来创建任务。
### 示例3查询数据源
对 AI 说:
> "列出所有启用的数据源"
AI 会调用 `get_data_sources` 工具。
## 六、可用的工具列表
| 工具名称 | 功能 | 主要参数 |
|---------|------|---------|
| `create_data_collection_task` | 创建数据采集任务 | name, data_source_id, database, collection, mode |
| `create_tag_task` | 创建标签任务 | name, task_type, target_tag_ids |
| `list_data_collection_tasks` | 列出数据采集任务 | page, page_size |
| `list_tag_tasks` | 列出标签任务 | page, page_size |
| `get_data_sources` | 获取数据源列表 | type, status |
| `get_tag_definitions` | 获取标签定义列表 | status |
| `start_data_collection_task` | 启动数据采集任务 | task_id |
| `start_tag_task` | 启动标签任务 | task_id |
详细参数说明请查看 `MCP/MCP服务器使用说明.md`
## 七、故障排除
### 问题1找不到模块
**错误**`Cannot find module '@modelcontextprotocol/sdk'`
**解决**
```bash
cd MCP/moncter-mcp-server
npm install
```
### 问题2编译失败
**错误**TypeScript 编译错误
**解决**
- 检查 Node.js 版本(需要 >= 18
- 检查 TypeScript 版本
- 运行 `npm install` 重新安装依赖
### 问题3无法连接到后端
**错误**`API请求错误: connect ECONNREFUSED`
**解决**
1. 检查后端服务是否运行:`php start.php status`
2. 检查 `MONCTER_API_URL` 环境变量是否正确
3. 检查端口是否被占用:`netstat -ano | findstr :8787`
### 问题4路径错误
**错误**`Cannot find module` 或路径相关错误
**解决**
- 检查 `mcp.json` 中的路径是否正确
- 使用绝对路径而不是相对路径
- 确保 `cwd` 设置正确
## 八、开发调试
### 查看日志
MCP 服务器的错误日志会输出到 stderr可以在 MCP 客户端中查看。
### 本地测试
```bash
cd MCP/moncter-mcp-server
npm run dev
```
然后使用 MCP Inspector 连接测试。
---
**需要帮助?** 查看 `MCP/MCP服务器使用说明.md` 获取详细文档。

View File

@@ -0,0 +1,13 @@
root = true
[*]
charset = utf-8
indent_style = space
indent_size = 2
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
trim_trailing_whitespace = false

29
Moncter/TaskShow/.gitignore vendored Normal file
View File

@@ -0,0 +1,29 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
# Environment variables
.env.local
.env.*.local

127
Moncter/TaskShow/README.md Normal file
View File

@@ -0,0 +1,127 @@
# Task Show
基于 Vue3 + Element Plus + Pinia + TypeScript + Axios 的前端基础工程
## 技术栈
- **Vue 3** - 渐进式 JavaScript 框架
- **TypeScript** - JavaScript 的超集
- **Vite** - 下一代前端构建工具
- **Element Plus** - 基于 Vue 3 的组件库
- **Pinia** - Vue 的状态管理库
- **Vue Router** - Vue 官方路由管理器
- **Axios** - 基于 Promise 的 HTTP 客户端
## 项目结构
```
TaskShow/
├── src/
│ ├── assets/ # 静态资源
│ ├── components/ # 公共组件
│ ├── router/ # 路由配置
│ │ └── index.ts
│ ├── store/ # Pinia 状态管理
│ │ └── index.ts
│ ├── types/ # TypeScript 类型定义
│ │ └── api.ts
│ ├── utils/ # 工具函数
│ │ └── request.ts # Axios 请求封装
│ ├── views/ # 页面组件
│ │ └── Home.vue
│ ├── App.vue # 根组件
│ └── main.ts # 入口文件
├── index.html # HTML 模板
├── package.json # 项目配置
├── tsconfig.json # TypeScript 配置
├── vite.config.ts # Vite 配置
└── README.md # 项目说明
```
## 安装依赖
```bash
npm install
# 或
yarn install
# 或
pnpm install
```
## 开发
```bash
npm run dev
# 或
yarn dev
# 或
pnpm dev
```
## 构建
```bash
npm run build
# 或
yarn build
# 或
pnpm build
```
## 预览构建结果
```bash
npm run preview
# 或
yarn preview
# 或
pnpm preview
```
## 请求封装说明
### 使用方式
```typescript
import { request } from '@/utils/request'
// GET 请求
const response = await request.get('/api/users', { id: 1 })
// POST 请求
const response = await request.post('/api/users', { name: 'John' })
// PUT 请求
const response = await request.put('/api/users/1', { name: 'Jane' })
// DELETE 请求
const response = await request.delete('/api/users/1')
// 自定义配置
const response = await request.get('/api/users', {}, {
showLoading: false, // 不显示 loading
showError: false, // 不显示错误提示
timeout: 5000 // 自定义超时时间
})
```
### 特性
1. **自动添加 Token**:请求时自动从 store 中获取 token 并添加到请求头
2. **统一错误处理**:自动处理 HTTP 错误和业务错误
3. **Loading 提示**:请求时自动显示 loading可配置
4. **错误提示**:请求失败时自动显示错误消息(可配置)
5. **类型支持**:完整的 TypeScript 类型定义
### 环境变量
`.env``.env.development``.env.production` 文件中配置 API 基础地址:
```
VITE_API_BASE_URL=/api
```
## License
MIT

154
Moncter/TaskShow/USAGE.md Normal file
View File

@@ -0,0 +1,154 @@
# 使用说明
## 快速开始
### 1. 安装依赖
```bash
cd TaskShow
npm install
```
### 2. 启动开发服务器
```bash
npm run dev
```
### 3. 构建生产版本
```bash
npm run build
```
## 核心功能使用
### 1. 使用封装的请求方法
```typescript
import { request } from '@/utils/request'
// GET 请求
const getUserList = async () => {
try {
const response = await request.get('/api/users', { page: 1, pageSize: 10 })
console.log(response.data) // 响应数据
} catch (error) {
console.error('请求失败:', error)
}
}
// POST 请求
const createUser = async () => {
try {
const response = await request.post('/api/users', {
name: 'John',
email: 'john@example.com'
})
console.log(response.data)
} catch (error) {
console.error('创建失败:', error)
}
}
// 自定义配置
const customRequest = async () => {
const response = await request.get('/api/users', {}, {
showLoading: false, // 不显示 loading
showError: false, // 不显示错误提示
timeout: 5000 // 5秒超时
})
}
```
### 2. 使用 Pinia Store
```typescript
import { useUserStore } from '@/store'
// 在组件中使用
const userStore = useUserStore()
// 设置 token
userStore.setToken('your-token-here')
// 设置用户信息
userStore.setUserInfo({ id: 1, name: 'John' })
// 清除用户信息
userStore.clearUser()
// 访问状态
console.log(userStore.token)
console.log(userStore.userInfo)
```
### 3. 使用路由
```typescript
import { useRouter, useRoute } from 'vue-router'
const router = useRouter()
const route = useRoute()
// 编程式导航
router.push('/home')
router.push({ name: 'Home', params: { id: 1 } })
// 获取路由参数
const id = route.params.id
```
### 4. 使用 Element Plus 组件
```vue
<template>
<el-button type="primary" @click="handleClick">点击</el-button>
<el-table :data="tableData">
<el-table-column prop="name" label="姓名" />
<el-table-column prop="email" label="邮箱" />
</el-table>
</template>
<script setup lang="ts">
import { ref } from 'vue'
import { ElMessage } from 'element-plus'
const tableData = ref([
{ name: 'John', email: 'john@example.com' }
])
const handleClick = () => {
ElMessage.success('操作成功')
}
</script>
```
## 项目结构说明
- `src/api/` - API 接口定义
- `src/components/` - 公共组件
- `src/router/` - 路由配置
- `src/store/` - Pinia 状态管理
- `src/types/` - TypeScript 类型定义
- `src/utils/` - 工具函数(包含封装的 request
- `src/views/` - 页面组件
## 环境变量配置
在项目根目录创建 `.env.development``.env.production` 文件:
```bash
# .env.development
VITE_API_BASE_URL=http://localhost:8080/api
# .env.production
VITE_API_BASE_URL=https://api.example.com/api
```
## 注意事项
1. 所有 API 请求会自动添加 token如果存在
2. 请求失败会自动显示错误提示(可通过配置关闭)
3. 请求时会自动显示 loading可通过配置关闭
4. 401 错误会自动清除用户信息并提示登录

View File

@@ -0,0 +1,14 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<link rel="icon" type="image/svg+xml" href="/vite.svg">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Task Show</title>
</head>
<body>
<div id="app"></div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>

2846
Moncter/TaskShow/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,27 @@
{
"name": "task-show",
"version": "1.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vue-tsc && vite build",
"preview": "vite preview",
"lint": "eslint . --ext .vue,.js,.jsx,.cjs,.mjs,.ts,.tsx,.cts,.mts --fix --ignore-path .gitignore"
},
"dependencies": {
"@element-plus/icons-vue": "^2.3.1",
"axios": "^1.6.7",
"element-plus": "^2.5.6",
"pinia": "^2.1.7",
"vue": "^3.4.21",
"vue-router": "^4.3.0"
},
"devDependencies": {
"@types/node": "^20.11.24",
"@vitejs/plugin-vue": "^5.0.4",
"sass-embedded": "^1.97.1",
"typescript": "^5.4.2",
"vite": "^5.1.6",
"vue-tsc": "^1.8.27"
}
}

1794
Moncter/TaskShow/pnpm-lock.yaml generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,32 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"module": "ESNext",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"skipLibCheck": true,
/* Bundler mode */
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "preserve",
/* Linting */
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
/* Path alias */
"baseUrl": ".",
"paths": {
"@/*": ["src/*"]
}
},
"include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"],
"references": [{ "path": "./tsconfig.node.json" }]
}

View File

@@ -0,0 +1,12 @@
{
"compilerOptions": {
"composite": true,
"skipLibCheck": true,
"module": "ESNext",
"moduleResolution": "bundler",
"allowSyntheticDefaultImports": true,
"strict": true
},
"include": ["vite.config.ts"]
}

View File

@@ -0,0 +1,25 @@
import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'
import { resolve } from 'path'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [vue()],
resolve: {
alias: {
'@': resolve(__dirname, 'src')
}
},
server: {
port: 3000,
open: true,
proxy: {
'/api': {
target: 'http://127.0.0.1:8787',
changeOrigin: true,
// 后端路由已经包含 /api 前缀,所以直接转发,不需要 rewrite
}
}
}
})

View File

@@ -0,0 +1,163 @@
<?php
namespace app\command;
use app\repository\TagDefinitionRepository;
use app\repository\UserProfileRepository;
use app\service\TagService;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\LoggerHelper;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
/**
* 批量更新标签命令
*
* 用于定时批量更新非实时标签daily/weekly/monthly
*
* 使用方式:
* php start.php batch-update-tags [--frequency=daily|weekly|monthly] [--limit=1000]
*/
class BatchUpdateTags extends Command
{
protected static $defaultName = 'batch-update-tags';
protected static $defaultDescription = '批量更新标签(定时任务)';
protected function configure(): void
{
$this->addOption(
'frequency',
'f',
InputOption::VALUE_OPTIONAL,
'更新频率daily/weekly/monthly',
'daily'
);
$this->addOption(
'limit',
'l',
InputOption::VALUE_OPTIONAL,
'每次处理的最大用户数',
1000
);
$this->addOption(
'user-ids',
'u',
InputOption::VALUE_OPTIONAL,
'指定用户ID列表逗号分隔如果提供则只更新这些用户',
null
);
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$frequency = $input->getOption('frequency');
$limit = (int)$input->getOption('limit');
$userIdsOption = $input->getOption('user-ids');
$output->writeln("开始批量更新标签...");
$output->writeln("更新频率: {$frequency}");
$output->writeln("处理限制: {$limit}");
try {
// 获取需要更新的标签定义
$tagDefinitionRepo = new TagDefinitionRepository();
$tagDefinitions = $tagDefinitionRepo->newQuery()
->where('status', 0) // 只获取启用的标签
->where('update_frequency', $frequency) // 匹配更新频率
->get();
if ($tagDefinitions->isEmpty()) {
$output->writeln("没有找到需要更新的标签定义(频率: {$frequency}");
return Command::SUCCESS;
}
$tagIds = $tagDefinitions->pluck('tag_id')->toArray();
$output->writeln("找到 " . count($tagIds) . " 个标签需要更新");
// 获取需要更新的用户列表
$userProfileRepo = new UserProfileRepository();
if ($userIdsOption !== null) {
// 指定用户ID列表
$userIds = array_filter(array_map('trim', explode(',', $userIdsOption)));
} else {
// 获取所有有效用户(限制数量)
$users = $userProfileRepo->newQuery()
->where('status', 0)
->limit($limit)
->get();
$userIds = $users->pluck('user_id')->toArray();
}
if (empty($userIds)) {
$output->writeln("没有找到需要更新的用户");
return Command::SUCCESS;
}
$output->writeln("找到 " . count($userIds) . " 个用户需要更新");
// 创建 TagService 实例
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
// 批量更新标签
$successCount = 0;
$errorCount = 0;
$startTime = microtime(true);
foreach ($userIds as $index => $userId) {
try {
$tagService->calculateTags($userId, $tagIds);
$successCount++;
if (($index + 1) % 100 === 0) {
$output->writeln("已处理: " . ($index + 1) . " / " . count($userIds));
}
} catch (\Throwable $e) {
$errorCount++;
LoggerHelper::logError($e, [
'component' => 'BatchUpdateTags',
'user_id' => $userId,
'frequency' => $frequency,
]);
}
}
$duration = microtime(true) - $startTime;
$output->writeln("批量更新完成!");
$output->writeln("成功: {$successCount}");
$output->writeln("失败: {$errorCount}");
$output->writeln("耗时: " . round($duration, 2) . "");
LoggerHelper::logBusiness('batch_tag_update_completed', [
'frequency' => $frequency,
'tag_count' => count($tagIds),
'user_count' => count($userIds),
'success_count' => $successCount,
'error_count' => $errorCount,
'duration' => $duration,
]);
return Command::SUCCESS;
} catch (\Throwable $e) {
$output->writeln("批量更新失败: " . $e->getMessage());
LoggerHelper::logError($e, [
'component' => 'BatchUpdateTags',
'frequency' => $frequency,
]);
return Command::FAILURE;
}
}
}

View File

@@ -0,0 +1,28 @@
<?php
namespace app\command;
use app\service\TagInitService;
/**
* 初始化标签命令
*
* 用于预置基础标签定义
* 使用方法php start.php init:tags
*/
class InitTags
{
public function run(): void
{
echo "开始初始化标签定义...\n";
$initService = new TagInitService(
new \app\repository\TagDefinitionRepository()
);
$initService->initBasicTags();
echo "标签初始化完成!\n";
}
}

View File

@@ -0,0 +1,110 @@
<?php
namespace app\controller;
use app\repository\ConsumptionRecordRepository;
use app\repository\UserProfileRepository;
use app\service\ConsumptionService;
use app\utils\ApiResponseHelper;
use app\utils\LoggerHelper;
use support\Request;
use support\Response;
class ConsumptionController
{
/**
* 创建消费记录
*
* POST /api/consumption/record
*/
public function store(Request $request): Response
{
$startTime = microtime(true);
try {
// 记录请求日志
LoggerHelper::logRequest('POST', '/api/consumption/record', [
'ip' => $request->getRealIp(),
'user_agent' => $request->header('user-agent'),
]);
// 获取 JSON 请求体
$rawBody = $request->rawBody();
if (empty($rawBody)) {
return ApiResponseHelper::error('请求体为空,请确保 Content-Type 为 application/json 并发送有效的 JSON 数据', 400);
}
$payload = json_decode($rawBody, true);
if (json_last_error() !== JSON_ERROR_NONE) {
return ApiResponseHelper::error('JSON 格式错误: ' . json_last_error_msg(), 400);
}
// 验证用户标识:必须提供 user_id、phone_number 或 id_card 之一
// 注意如果手机号和身份证号都为空但提供了user_id仍然可以处理
$phoneNumber = trim($payload['phone_number'] ?? '');
$idCard = trim($payload['id_card'] ?? '');
if (empty($payload['user_id']) && empty($phoneNumber) && empty($idCard)) {
throw new \InvalidArgumentException('缺少用户标识:必须提供 user_id、phone_number 或 id_card 之一');
}
// 简单手动组装 Service 依赖,后续可接入容器配置
$tagService = new \app\service\TagService(
new \app\repository\TagDefinitionRepository(),
new \app\repository\UserProfileRepository(),
new \app\repository\UserTagRepository(),
new \app\repository\TagHistoryRepository(),
new \app\service\TagRuleEngine\SimpleRuleEngine()
);
$identifierService = new \app\service\IdentifierService(
new UserProfileRepository(),
new \app\service\UserPhoneService(
new \app\repository\UserPhoneRelationRepository()
)
);
$service = new ConsumptionService(
new ConsumptionRecordRepository(),
new UserProfileRepository(),
$identifierService,
$tagService
);
$result = $service->createRecord($payload);
// 如果返回 null说明手机号和身份证号都为空跳过该记录
if ($result === null) {
LoggerHelper::logBusiness('consumption_record_skipped_no_identifier', [
'reason' => 'phone_number and id_card are both empty',
'consume_time' => $payload['consume_time'] ?? null,
]);
return ApiResponseHelper::error('记录已跳过:手机号和身份证号都为空', 400, 400);
}
// 记录业务日志
LoggerHelper::logBusiness('consumption_record_created', [
'user_id' => $result['user_id'] ?? null,
'record_id' => $result['record_id'] ?? null,
'amount' => $payload['amount'] ?? null,
'phone_number' => $payload['phone_number'] ?? null,
'id_card_provided' => !empty($payload['id_card']),
]);
$duration = microtime(true) - $startTime;
LoggerHelper::logPerformance('consumption_record_create', $duration, [
'user_id' => $result['user_id'] ?? null,
]);
return ApiResponseHelper::success($result);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
}

View File

@@ -0,0 +1,850 @@
<?php
namespace app\controller;
use app\service\DataCollectionTaskService;
use app\utils\ApiResponseHelper;
use support\Request;
use support\Response;
/**
* 数据采集任务管理控制器
*
* 提供任务创建、管理、进度查询等接口
*/
class DataCollectionTaskController
{
/**
* 获取任务服务实例
*/
private function getService(): DataCollectionTaskService
{
return new DataCollectionTaskService(
new \app\repository\DataCollectionTaskRepository()
);
}
/**
* 创建采集任务
*
* POST /api/data-collection-tasks
*/
public function create(Request $request): Response
{
try {
$data = $request->post();
// 验证必填字段
$requiredFields = ['name', 'data_source_id', 'database', 'target_type'];
foreach ($requiredFields as $field) {
if (empty($data[$field])) {
return ApiResponseHelper::error("缺少必填字段: {$field}", 400);
}
}
// 验证目标类型
if (!in_array($data['target_type'], ['consumption_record', 'generic'])) {
return ApiResponseHelper::error("目标类型必须是 consumption_record 或 generic", 400);
}
// 如果是通用Handler需要目标数据源配置后端会自动处理consumption_record的配置
if ($data['target_type'] === 'generic') {
$genericRequiredFields = ['target_data_source_id', 'target_database', 'target_collection'];
foreach ($genericRequiredFields as $field) {
if (empty($data[$field])) {
return ApiResponseHelper::error("通用Handler缺少必填字段: {$field}", 400);
}
}
}
// 验证模式
if (isset($data['mode']) && !in_array($data['mode'], ['batch', 'realtime'])) {
return ApiResponseHelper::error("模式必须是 batch 或 realtime", 400);
}
// 验证集合配置
if (empty($data['collection']) && empty($data['collections'])) {
return ApiResponseHelper::error("必须指定 collection 或 collections", 400);
}
$service = $this->getService();
$task = $service->createTask($data);
return ApiResponseHelper::success($task, '任务创建成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 更新任务
*
* PUT /api/data-collection-tasks/{task_id}
*/
public function update(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$data = $request->post();
$service = $this->getService();
$result = $service->updateTask($taskId, $data);
if ($result) {
return ApiResponseHelper::success(null, '任务更新成功');
} else {
return ApiResponseHelper::error('任务更新失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除任务
*
* DELETE /api/data-collection-tasks/{task_id}
*/
public function delete(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$result = $service->deleteTask($taskId);
if ($result) {
return ApiResponseHelper::success(null, '任务删除成功');
} else {
return ApiResponseHelper::error('任务删除失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 启动任务
*
* POST /api/data-collection-tasks/{task_id}/start
*/
public function start(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)/start#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$result = $service->startTask($taskId);
if ($result) {
return ApiResponseHelper::success(null, '任务启动成功');
} else {
return ApiResponseHelper::error('任务启动失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 暂停任务
*
* POST /api/data-collection-tasks/{task_id}/pause
*/
public function pause(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)/pause#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$result = $service->pauseTask($taskId);
if ($result) {
return ApiResponseHelper::success(null, '任务暂停成功');
} else {
return ApiResponseHelper::error('任务暂停失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 停止任务
*
* POST /api/data-collection-tasks/{task_id}/stop
*/
public function stop(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)/stop#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$result = $service->stopTask($taskId);
if ($result) {
return ApiResponseHelper::success(null, '任务停止成功');
} else {
return ApiResponseHelper::error('任务停止失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务列表
*
* GET /api/data-collection-tasks
*/
public function list(Request $request): Response
{
try {
// 只收集非空的筛选条件
$filters = [];
if ($request->get('status') !== null && $request->get('status') !== '') {
$filters['status'] = $request->get('status');
}
if ($request->get('data_source_id') !== null && $request->get('data_source_id') !== '') {
$filters['data_source_id'] = $request->get('data_source_id');
}
if ($request->get('name') !== null && $request->get('name') !== '') {
$filters['name'] = $request->get('name');
}
$page = (int)($request->get('page', 1));
$pageSize = (int)($request->get('page_size', 20));
$service = $this->getService();
$result = $service->getTaskList($filters, $page, $pageSize);
return ApiResponseHelper::success($result, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务详情
*
* GET /api/data-collection-tasks/{task_id}
*/
public function detail(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)$#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$task = $service->getTask($taskId);
if ($task === null) {
return ApiResponseHelper::error('任务不存在', 404);
}
return ApiResponseHelper::success($task, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务进度
*
* GET /api/data-collection-tasks/{task_id}/progress
*/
public function progress(Request $request): Response
{
try {
// 从请求路径中解析 task_id
$path = $request->path();
if (preg_match('#/api/data-collection-tasks/([^/]+)/progress#', $path, $matches)) {
$taskId = $matches[1];
} else {
$taskId = $request->get('task_id');
if (!$taskId) {
throw new \InvalidArgumentException('缺少 task_id 参数');
}
}
$service = $this->getService();
$task = $service->getTask($taskId);
if ($task === null) {
return ApiResponseHelper::error('任务不存在', 404);
}
$progress = $task['progress'] ?? [];
return ApiResponseHelper::success($progress, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取数据源列表
*
* GET /api/data-collection-tasks/data-sources
*/
public function getDataSources(Request $request): Response
{
try {
// 优先使用数据库中的数据源,如果没有则使用配置文件
$service = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$result = $service->getDataSourceList(['status' => 1]);
if (!empty($result['list'])) {
// 使用数据库中的数据源
$list = array_map(function ($ds) {
return [
'id' => $ds['data_source_id'],
'name' => $ds['name'] ?? $ds['data_source_id'], // 添加名称字段
'type' => $ds['type'] ?? 'unknown',
'host' => $ds['host'] ?? '',
'port' => $ds['port'] ?? 0,
'database' => $ds['database'] ?? '',
];
}, $result['list']);
}
// 注意现在数据源配置统一从数据库读取不再使用config('data_sources')
// 如果数据库中没有数据源,返回空列表
if (!isset($list)) {
$list = [];
}
return ApiResponseHelper::success($list, '查询成功');
} catch (\MongoDB\Driver\Exception\Exception $e) {
// MongoDB 连接错误,返回友好提示
$errorMessage = '无法连接到 MongoDB 数据库,请检查数据库服务是否正常运行。错误详情:' . $e->getMessage();
return ApiResponseHelper::error($errorMessage, 500);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取数据源的数据库列表
*
* GET /api/data-collection-tasks/data-sources/{data_source_id}/databases
*/
public function getDatabases(Request $request, string $data_source_id): Response
{
try {
// 从数据库获取数据源配置
$service = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$dataSourceConfig = $service->getDataSourceConfig($data_source_id);
if (!$dataSourceConfig) {
return ApiResponseHelper::error('数据源不存在', 404);
}
$dataSource = $dataSourceConfig;
// 如果是MongoDB连接并获取数据库列表
if ($dataSource['type'] === 'mongodb') {
$client = $this->getMongoClient($dataSource);
$databases = $client->listDatabases();
$list = [];
foreach ($databases as $database) {
$dbName = $database->getName();
// 同时返回原始名称和base64编码的IDURL友好
$list[] = [
'name' => $dbName,
'id' => base64_encode($dbName), // URL友好的标识符
];
}
return ApiResponseHelper::success($list, '查询成功');
}
return ApiResponseHelper::error('不支持的数据源类型', 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取数据库的集合列表
*
* GET /api/data-collection-tasks/data-sources/{data_source_id}/databases/{database}/collections
*/
public function getCollections(Request $request, string $data_source_id, string $database): Response
{
try {
// 解码数据库名称支持base64编码和URL编码
$database = $this->decodeName($database);
// 从数据库获取数据源配置
$service = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$dataSourceConfig = $service->getDataSourceConfig($data_source_id);
if (!$dataSourceConfig) {
return ApiResponseHelper::error('数据源不存在', 404);
}
$dataSource = $dataSourceConfig;
// 如果是MongoDB连接并获取集合列表
if ($dataSource['type'] === 'mongodb') {
$client = $this->getMongoClient($dataSource);
$db = $client->selectDatabase($database);
$collections = $db->listCollections();
$list = [];
foreach ($collections as $collection) {
$collName = $collection->getName();
// 同时返回原始名称和base64编码的IDURL友好
$list[] = [
'name' => $collName,
'id' => base64_encode($collName), // URL友好的标识符
];
}
return ApiResponseHelper::success($list, '查询成功');
}
return ApiResponseHelper::error('不支持的数据源类型', 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取Handler的目标字段列表
*
* GET /api/data-collection-tasks/handlers/{handler_type}/target-fields
*/
public function getHandlerTargetFields(Request $request, string $handler_type): Response
{
try {
$fields = [];
switch ($handler_type) {
case 'consumption_record':
// 消费记录Handler的目标字段列表
// 包含原始输入字段(推荐)和转换后字段(可选)
// Handler会自动进行转换phone_number/id_card -> user_id, store_name -> store_id
$fields = [
// 用户标识字段(原始输入,推荐使用)
['name' => 'phone_number', 'label' => '手机号', 'type' => 'string', 'required' => false, 'description' => '手机号Handler会自动解析为user_id', 'is_original' => true],
['name' => 'id_card', 'label' => '身份证', 'type' => 'string', 'required' => false, 'description' => '身份证号Handler会自动解析为user_id', 'is_original' => true],
// 用户ID转换后字段由Handler自动生成不需要映射
['name' => 'user_id', 'label' => '用户ID', 'type' => 'string', 'required' => false, 'description' => '用户ID由Handler通过phone_number/id_card自动解析生成无需映射', 'is_original' => false, 'no_mapping' => true],
// 门店标识字段(原始输入,推荐使用)
['name' => 'store_name', 'label' => '门店名称', 'type' => 'string', 'required' => false, 'description' => '门店名称Handler会自动转换为store_id', 'is_original' => true],
// 门店ID转换后字段由Handler自动生成不需要映射
['name' => 'store_id', 'label' => '门店ID', 'type' => 'string', 'required' => false, 'description' => '门店ID由Handler通过store_name自动转换生成无需映射', 'is_original' => false, 'no_mapping' => true],
// 订单标识字段(用于去重)
['name' => 'source_order_id', 'label' => '原始订单ID', 'type' => 'string', 'required' => false, 'description' => '原始订单ID配合店铺名称做去重唯一标识建议配置', 'is_original' => true],
// 注意order_no 由系统自动生成(自动递增),不需要映射
// 金额和时间字段(直接字段)
['name' => 'amount', 'label' => '消费金额', 'type' => 'float', 'required' => true, 'description' => '消费金额(必填)', 'is_original' => true],
['name' => 'actual_amount', 'label' => '实际金额', 'type' => 'float', 'required' => true, 'description' => '实际支付金额(必填)', 'is_original' => true],
['name' => 'consume_time', 'label' => '消费时间', 'type' => 'datetime', 'required' => true, 'description' => '消费时间,用于时间分片存储(必填)', 'is_original' => true],
// 其他可选字段
['name' => 'currency', 'label' => '币种', 'type' => 'string', 'required' => false, 'description' => '币种默认CNY人民币', 'is_original' => true, 'fixed_options' => true, 'options' => [['value' => 'CNY', 'label' => '人民币(CNY)'], ['value' => 'USD', 'label' => '美元(USD)']], 'default_value' => 'CNY'],
['name' => 'status', 'label' => '记录状态', 'type' => 'int', 'required' => false, 'description' => '记录状态0-正常1-异常2-已删除。默认0。需要配置源状态值到标准状态值的映射', 'is_original' => true, 'value_mapping' => true, 'target_values' => [['value' => 0, 'label' => '正常(0)'], ['value' => 1, 'label' => '异常(1)'], ['value' => 2, 'label' => '已删除(2)']], 'default_value' => 0],
];
break;
case 'generic':
// 通用Handler - 没有固定的字段列表,由用户自定义
$fields = [];
break;
default:
return ApiResponseHelper::error("未知的Handler类型: {$handler_type}", 400);
}
return ApiResponseHelper::success($fields, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取集合的字段列表(采样)
*
* GET /api/data-collection-tasks/data-sources/{data_source_id}/databases/{database}/collections/{collection}/fields
*/
public function getFields(Request $request, string $data_source_id, string $database, string $collection): Response
{
try {
// 解码数据库名称和集合名称支持base64编码和URL编码
$database = $this->decodeName($database);
$collection = $this->decodeName($collection);
// 从数据库获取数据源配置
$service = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$dataSourceConfig = $service->getDataSourceConfig($data_source_id);
if (!$dataSourceConfig) {
return ApiResponseHelper::error('数据源不存在', 404);
}
$dataSource = $dataSourceConfig;
// 如果是MongoDB采样获取字段
if ($dataSource['type'] === 'mongodb') {
$client = $this->getMongoClient($dataSource);
$db = $client->selectDatabase($database);
$coll = $db->selectCollection($collection);
// 采样一条数据
$sample = $coll->findOne([]);
if ($sample) {
$fields = [];
$this->extractFields($sample, '', $fields);
return ApiResponseHelper::success($fields, '查询成功');
} else {
return ApiResponseHelper::success([], '集合为空,无法获取字段');
}
}
return ApiResponseHelper::error('不支持的数据源类型', 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 递归提取字段
*/
private function extractFields($data, string $prefix, array &$fields): void
{
if (is_array($data) || is_object($data)) {
foreach ($data as $key => $value) {
$fieldName = $prefix ? "{$prefix}.{$key}" : $key;
if (is_array($value) || is_object($value)) {
if (empty($value)) {
$fields[] = [
'name' => $fieldName,
'type' => 'array',
];
} else {
$this->extractFields($value, $fieldName, $fields);
}
} else {
$fields[] = [
'name' => $fieldName,
'type' => gettype($value),
];
}
}
}
}
/**
* 预览查询结果包含lookup
*
* POST /api/data-collection-tasks/preview-query
*/
public function previewQuery(Request $request): Response
{
try {
$data = $request->post();
$dataSourceId = $data['data_source_id'] ?? '';
$database = $data['database'] ?? '';
$collection = $data['collection'] ?? '';
$lookups = $data['lookups'] ?? [];
$filterConditions = $data['filter_conditions'] ?? [];
$limit = (int)($data['limit'] ?? 5); // 默认预览5条
if (empty($dataSourceId) || empty($database) || empty($collection)) {
return ApiResponseHelper::error('缺少必要参数data_source_id, database, collection', 400);
}
// 获取数据源配置
$service = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$dataSourceConfig = $service->getDataSourceConfig($dataSourceId);
if (!$dataSourceConfig) {
return ApiResponseHelper::error('数据源不存在', 404);
}
if ($dataSourceConfig['type'] !== 'mongodb') {
return ApiResponseHelper::error('目前只支持MongoDB数据源预览', 400);
}
// 连接MongoDB
$client = $this->getMongoClient($dataSourceConfig);
$db = $client->selectDatabase($database);
$coll = $db->selectCollection($collection);
// 构建聚合管道
$pipeline = [];
// 1. 添加过滤条件($match- 必须在最前面
$filter = $this->buildFilterForPreview($filterConditions);
if (!empty($filter)) {
$pipeline[] = ['$match' => $filter];
}
// 2. 添加lookup查询
foreach ($lookups as $lookup) {
if (empty($lookup['from']) || empty($lookup['local_field']) || empty($lookup['foreign_field'])) {
continue;
}
$lookupStage = [
'$lookup' => [
'from' => $lookup['from'],
'localField' => $lookup['local_field'],
'foreignField' => $lookup['foreign_field'],
'as' => $lookup['as'] ?? 'joined'
]
];
$pipeline[] = $lookupStage;
// 如果配置了解构
if (!empty($lookup['unwrap'])) {
$pipeline[] = [
'$unwind' => [
'path' => '$' . ($lookup['as'] ?? 'joined'),
'preserveNullAndEmptyArrays' => !empty($lookup['preserve_null'])
]
];
}
}
// 3. 限制返回数量
$pipeline[] = ['$limit' => $limit];
// 执行聚合查询
$cursor = $coll->aggregate($pipeline);
$results = [];
$fields = [];
foreach ($cursor as $doc) {
$docArray = $this->convertMongoDocumentToArray($doc);
$results[] = $docArray;
// 提取字段
$this->extractFields($docArray, '', $fields);
}
// 去重字段
$uniqueFields = [];
$fieldMap = [];
foreach ($fields as $field) {
if (!isset($fieldMap[$field['name']])) {
$fieldMap[$field['name']] = true;
$uniqueFields[] = $field;
}
}
return ApiResponseHelper::success([
'fields' => $uniqueFields,
'data' => $results,
'count' => count($results)
], '预览成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 将MongoDB文档转换为数组
*/
private function convertMongoDocumentToArray($document): array
{
if (is_array($document)) {
return $document;
}
if (is_object($document)) {
$array = [];
foreach ($document as $key => $value) {
if ($value instanceof \MongoDB\BSON\UTCDateTime) {
$array[$key] = $value->toDateTime()->format('Y-m-d H:i:s');
} elseif (is_object($value) && method_exists($value, '__toString')) {
$array[$key] = (string)$value;
} elseif (is_array($value) || is_object($value)) {
$array[$key] = $this->convertMongoDocumentToArray($value);
} else {
$array[$key] = $value;
}
}
return $array;
}
return [];
}
/**
* 构建过滤条件(用于预览查询)
*
* @param array $filterConditions 过滤条件列表
* @return array MongoDB查询过滤器
*/
private function buildFilterForPreview(array $filterConditions): array
{
$filter = [];
foreach ($filterConditions as $condition) {
$field = $condition['field'] ?? '';
$operator = $condition['operator'] ?? 'eq';
$value = $condition['value'] ?? null;
if (empty($field)) {
continue;
}
// 处理值的类型转换
if ($value !== null && $value !== '') {
// 尝试转换为数字(如果是数字字符串)
if (is_numeric($value)) {
// 判断是整数还是浮点数
if (strpos($value, '.') !== false) {
$value = (float)$value;
} else {
$value = (int)$value;
}
}
}
switch ($operator) {
case 'eq':
$filter[$field] = $value;
break;
case 'ne':
$filter[$field] = ['$ne' => $value];
break;
case 'gt':
$filter[$field] = ['$gt' => $value];
break;
case 'gte':
$filter[$field] = ['$gte' => $value];
break;
case 'lt':
$filter[$field] = ['$lt' => $value];
break;
case 'lte':
$filter[$field] = ['$lte' => $value];
break;
case 'in':
// in操作符的值应该是数组
$valueArray = is_array($value) ? $value : explode(',', (string)$value);
$filter[$field] = ['$in' => $valueArray];
break;
case 'nin':
// nin操作符的值应该是数组
$valueArray = is_array($value) ? $value : explode(',', (string)$value);
$filter[$field] = ['$nin' => $valueArray];
break;
}
}
return $filter;
}
/**
* 解码数据库或集合名称支持base64编码和URL编码
*
* @param string $name 编码后的名称
* @return string 解码后的名称
*/
private function decodeName(string $name): string
{
// 尝试base64解码如果前端使用的是编码后的ID
// 检查是否可能是base64编码只包含base64字符且长度合理
if (preg_match('/^[A-Za-z0-9+\/]*={0,2}$/', $name) && strlen($name) > 0) {
$decoded = @base64_decode($name, true);
if ($decoded !== false && $decoded !== '') {
// 解码成功,使用解码后的值
return $decoded;
}
}
// 不是base64格式或解码失败使用URL解码处理中文等特殊字符
return rawurldecode($name);
}
/**
* 获取MongoDB客户端
*/
private function getMongoClient(array $config): \MongoDB\Client
{
$host = $config['host'] ?? '';
$port = (int)($config['port'] ?? 27017);
$username = $config['username'] ?? '';
$password = $config['password'] ?? '';
$authSource = $config['auth_source'] ?? 'admin';
if (!empty($username) && !empty($password)) {
$dsn = "mongodb://{$username}:{$password}@{$host}:{$port}/{$authSource}";
} else {
$dsn = "mongodb://{$host}:{$port}";
}
return new \MongoDB\Client($dsn, $config['options'] ?? []);
}
}

View File

@@ -0,0 +1,173 @@
<?php
namespace app\controller;
use app\repository\DataSourceRepository;
use app\service\DataSourceService;
use app\utils\ApiResponseHelper;
use support\Request;
use support\Response;
/**
* 数据源管理控制器
*/
class DataSourceController
{
/**
* 获取数据源服务实例
*/
private function getService(): DataSourceService
{
return new DataSourceService(new DataSourceRepository());
}
/**
* 获取数据源列表
*
* GET /api/data-sources
*/
public function list(Request $request): Response
{
try {
$service = $this->getService();
$filters = [
'type' => $request->get('type'),
'status' => $request->get('status'),
'name' => $request->get('name'),
'page' => $request->get('page', 1),
'page_size' => $request->get('page_size', 20),
];
$result = $service->getDataSourceList($filters);
return ApiResponseHelper::success([
'data_sources' => $result['list'],
'total' => $result['total'],
'page' => (int)$filters['page'],
'page_size' => (int)$filters['page_size'],
], '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取数据源详情
*
* GET /api/data-sources/{data_source_id}
*/
public function detail(Request $request, string $data_source_id): Response
{
try {
$service = $this->getService();
$dataSource = $service->getDataSourceDetail($data_source_id);
if (!$dataSource) {
return ApiResponseHelper::error('数据源不存在', 404);
}
return ApiResponseHelper::success($dataSource, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 创建数据源
*
* POST /api/data-sources
*/
public function create(Request $request): Response
{
try {
$data = $request->post();
$service = $this->getService();
$dataSource = $service->createDataSource($data);
// 不返回密码
$result = $dataSource->toArray();
unset($result['password']);
return ApiResponseHelper::success($result, '创建成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 更新数据源
*
* PUT /api/data-sources/{data_source_id}
*/
public function update(Request $request, string $data_source_id): Response
{
try {
$data = $request->post();
$service = $this->getService();
$result = $service->updateDataSource($data_source_id, $data);
if ($result) {
return ApiResponseHelper::success(null, '更新成功');
} else {
return ApiResponseHelper::error('更新失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除数据源
*
* DELETE /api/data-sources/{data_source_id}
*/
public function delete(Request $request, string $data_source_id): Response
{
try {
$service = $this->getService();
$result = $service->deleteDataSource($data_source_id);
if ($result) {
return ApiResponseHelper::success(null, '删除成功');
} else {
return ApiResponseHelper::error('删除失败', 500);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 测试数据源连接
*
* POST /api/data-sources/test-connection
*/
public function testConnection(Request $request): Response
{
try {
$data = $request->post();
// 验证必填字段
$requiredFields = ['type', 'host', 'port', 'database'];
foreach ($requiredFields as $field) {
if (empty($data[$field])) {
return ApiResponseHelper::error("缺少必填字段: {$field}", 400);
}
}
$service = $this->getService();
$connected = $service->testConnection($data);
if ($connected) {
return ApiResponseHelper::success(['connected' => true], '连接成功');
} else {
return ApiResponseHelper::error('连接失败,请检查配置', 400);
}
} catch (\Throwable $e) {
return ApiResponseHelper::error('连接测试失败: ' . $e->getMessage(), 400);
}
}
}

View File

@@ -0,0 +1,455 @@
<?php
namespace app\controller;
use app\service\DatabaseSyncService;
use app\utils\ApiResponseHelper;
use support\Request;
use support\Response;
/**
* 数据库同步控制器
*
* 提供同步进度查询接口
*/
class DatabaseSyncController
{
/**
* 获取同步进度看板页面
*
* GET /database-sync/dashboard
*/
public function dashboard(Request $request): Response
{
$htmlPath = __DIR__ . '/../../public/database-sync-dashboard.html';
if (!file_exists($htmlPath)) {
return response('<h1>404 Not Found</h1><p>看板页面不存在</p>', 404)
->withHeader('Content-Type', 'text/html; charset=utf-8');
}
$html = file_get_contents($htmlPath);
return response($html)->withHeader('Content-Type', 'text/html; charset=utf-8');
}
/**
* 获取同步进度
*
* GET /api/database-sync/progress
*/
public function progress(Request $request): Response
{
try {
// 创建 DatabaseSyncService 实例(传递最小配置,仅用于读取进度)
// 注意:数据库同步功能已迁移到 data_collection_tasks.php这里仅用于查询进度
$minimalConfig = [
'source' => ['host' => '', 'port' => 27017], // 占位符,不会实际连接
'target' => ['host' => '', 'port' => 27017], // 占位符,不会实际连接
'sync' => [],
'monitoring' => [],
];
$syncService = new DatabaseSyncService($minimalConfig);
// 加载最新进度
$syncService->loadProgress();
$progress = $syncService->getProgress();
// 获取多进程状态信息
$workerStatus = $this->getWorkerStatus();
$progress['worker_status'] = $workerStatus;
// 获取数据库连接状态
$connectionStatus = $this->getConnectionStatus();
$progress['connection_status'] = $connectionStatus;
// 获取数据库列表信息(已完成和待同步)
$databaseList = $this->getDatabaseList($syncService);
$progress['database_list'] = $databaseList;
// 检查进度文件最后修改时间
$runtimePath = function_exists('runtime_path') ? runtime_path() : (config('app.runtime_path', base_path() . DIRECTORY_SEPARATOR . 'runtime'));
$progressFile = $runtimePath . DIRECTORY_SEPARATOR . 'database_sync_progress.json';
if (file_exists($progressFile)) {
$fileTime = filemtime($progressFile);
$progress['progress_file_last_modified'] = date('Y-m-d H:i:s', $fileTime);
$progress['progress_file_age_seconds'] = time() - $fileTime;
} else {
$progress['progress_file_last_modified'] = null;
$progress['progress_file_age_seconds'] = null;
}
// 如果状态是idle且没有开始时间尝试检查是否真的在运行
if ($progress['status'] === 'idle' && $progress['time']['start_time'] === null) {
if (!file_exists($progressFile)) {
$progress['hint'] = '请执行: php start.php status 查看 data_sync_scheduler 进程是否运行(数据库同步任务由 data_sync_scheduler 管理)';
}
}
return ApiResponseHelper::success($progress, '同步进度查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取 Worker 进程状态信息
*
* @return array<string, mixed> Worker 状态信息
*/
private function getWorkerStatus(): array
{
$status = [
'total_workers' => 0,
'active_workers' => 0,
'workers' => [],
];
try {
// 从配置中获取 Worker 数量
$processConfig = config('process.data_sync_scheduler', []);
$totalWorkers = (int)($processConfig['count'] ?? 10);
$status['total_workers'] = $totalWorkers;
// 检查进度文件中的 checkpoints推断每个 Worker 处理的数据库
$runtimePath = function_exists('runtime_path') ? runtime_path() : (config('app.runtime_path', base_path() . DIRECTORY_SEPARATOR . 'runtime'));
$progressFile = $runtimePath . DIRECTORY_SEPARATOR . 'database_sync_progress.json';
if (file_exists($progressFile)) {
// 使用文件锁读取,避免并发问题
$fp = fopen($progressFile, 'r');
if ($fp && flock($fp, LOCK_SH)) {
try {
$content = stream_get_contents($fp);
$progressData = json_decode($content, true);
if ($progressData && isset($progressData['checkpoints'])) {
$checkpoints = $progressData['checkpoints'];
$databases = array_keys($checkpoints);
// 根据数据库分配推断每个 Worker 的状态(使用取模分配)
foreach ($databases as $index => $database) {
$workerId = $index % $totalWorkers;
if (!isset($status['workers'][$workerId])) {
$status['workers'][$workerId] = [
'worker_id' => $workerId,
'databases' => [],
'collections' => 0,
'documents_processed' => 0,
'status' => 'active',
];
}
$status['workers'][$workerId]['databases'][] = $database;
// 统计该 Worker 处理的集合和文档数
if (isset($checkpoints[$database]) && is_array($checkpoints[$database])) {
foreach ($checkpoints[$database] as $collection => $checkpoint) {
$status['workers'][$workerId]['collections']++;
if (isset($checkpoint['processed'])) {
$status['workers'][$workerId]['documents_processed'] += (int)$checkpoint['processed'];
}
}
}
}
$status['active_workers'] = count($status['workers']);
// 将 workers 数组转换为索引数组(便于前端遍历)
$status['workers'] = array_values($status['workers']);
}
} finally {
flock($fp, LOCK_UN);
fclose($fp);
}
} else {
if ($fp) {
fclose($fp);
}
}
}
// 如果没有活动的 Worker但配置了 Worker 数量,显示所有 Worker等待状态
if ($status['active_workers'] === 0 && $status['total_workers'] > 0) {
// 创建所有 Worker 的占位信息
for ($i = 0; $i < $totalWorkers; $i++) {
$status['workers'][] = [
'worker_id' => $i,
'databases' => [],
'collections' => 0,
'documents_processed' => 0,
'status' => 'waiting', // 等待状态
];
}
$status['message'] = '所有 Worker 处于等待状态,同步尚未开始或进度文件为空';
} elseif ($status['total_workers'] === 0) {
$status['message'] = '未配置 Worker 数量,请检查 config/process.php';
}
} catch (\Throwable $e) {
$status['error'] = '获取 Worker 状态失败: ' . $e->getMessage();
}
return $status;
}
/**
* 获取同步统计信息
*
* GET /api/database-sync/stats
*/
public function stats(Request $request): Response
{
try {
// 创建 DatabaseSyncService 实例(传递最小配置,仅用于读取统计)
$minimalConfig = [
'source' => ['host' => '', 'port' => 27017],
'target' => ['host' => '', 'port' => 27017],
];
$syncService = new DatabaseSyncService($minimalConfig);
$stats = $syncService->getStats();
return ApiResponseHelper::success($stats, '统计信息查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 重置同步进度
*
* POST /api/database-sync/reset
*/
public function reset(Request $request): Response
{
try {
// 创建 DatabaseSyncService 实例(传递最小配置,仅用于重置进度)
$minimalConfig = [
'source' => ['host' => '', 'port' => 27017],
'target' => ['host' => '', 'port' => 27017],
];
$syncService = new DatabaseSyncService($minimalConfig);
$syncService->resetProgress();
return ApiResponseHelper::success(null, '同步进度已重置');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 跳过错误数据库,继续同步
*
* POST /api/database-sync/skip-error
*/
public function skipError(Request $request): Response
{
try {
// 创建 DatabaseSyncService 实例(传递最小配置,仅用于跳过错误)
$minimalConfig = [
'source' => ['host' => '', 'port' => 27017],
'target' => ['host' => '', 'port' => 27017],
];
$syncService = new DatabaseSyncService($minimalConfig);
$syncService->loadProgress();
$skipped = $syncService->skipErrorDatabase();
if ($skipped) {
return ApiResponseHelper::success(null, '已跳过错误数据库,将继续同步下一个数据库');
} else {
return ApiResponseHelper::error('当前没有错误数据库需要跳过', 400);
}
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取数据库连接状态
*
* @return array<string, mixed> 连接状态信息
*/
private function getConnectionStatus(): array
{
$status = [
'source' => [
'connected' => false,
'host' => '',
'port' => 0,
'error' => null,
],
'target' => [
'connected' => false,
'host' => '',
'port' => 0,
'error' => null,
],
];
try {
// 从数据库获取数据源配置
$dataSourceService = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
// 检查源数据库连接
$sourceDataSourceId = 'kr_mongodb'; // 默认源数据库ID可以从任务配置中获取
$sourceConfig = $dataSourceService->getDataSourceConfigById($sourceDataSourceId);
if ($sourceConfig) {
$status['source']['host'] = $sourceConfig['host'] ?? '';
$status['source']['port'] = (int)($sourceConfig['port'] ?? 27017);
// 尝试连接源数据库
try {
$sourceDsn = $this->buildDsn($sourceConfig);
$sourceClient = new \MongoDB\Client($sourceDsn, $sourceConfig['options'] ?? []);
// 执行一个简单的命令来测试连接
$sourceClient->selectDatabase('admin')->command(['ping' => 1]);
$status['source']['connected'] = true;
} catch (\Throwable $e) {
$status['source']['connected'] = false;
$status['source']['error'] = $e->getMessage();
}
} else {
$status['source']['error'] = '源数据库配置不存在';
}
// 检查目标数据库连接
$targetDataSourceId = 'sync_mongodb'; // 默认目标数据库ID可以从任务配置中获取
$targetConfig = $dataSourceService->getDataSourceConfigById($targetDataSourceId);
if ($targetConfig) {
$status['target']['host'] = $targetConfig['host'] ?? '';
$status['target']['port'] = (int)($targetConfig['port'] ?? 27017);
// 尝试连接目标数据库
try {
$targetDsn = $this->buildDsn($targetConfig);
$targetClient = new \MongoDB\Client($targetDsn, $targetConfig['options'] ?? []);
// 执行一个简单的命令来测试连接
$targetClient->selectDatabase('admin')->command(['ping' => 1]);
$status['target']['connected'] = true;
} catch (\Throwable $e) {
$status['target']['connected'] = false;
$status['target']['error'] = $e->getMessage();
}
} else {
$status['target']['error'] = '目标数据库配置不存在';
}
} catch (\Throwable $e) {
$status['error'] = '检查连接状态失败: ' . $e->getMessage();
}
return $status;
}
/**
* 构建 MongoDB DSN
*
* @param array<string, mixed> $config 数据库配置
* @return string DSN 字符串
*/
private function buildDsn(array $config): string
{
$host = $config['host'] ?? '';
$port = (int)($config['port'] ?? 27017);
$dsn = 'mongodb://';
if (!empty($config['username']) && !empty($config['password'])) {
$dsn .= urlencode($config['username']) . ':' . urlencode($config['password']) . '@';
}
$dsn .= $host . ':' . $port;
if (!empty($config['auth_source'])) {
$dsn .= '/?authSource=' . urlencode($config['auth_source']);
}
return $dsn;
}
/**
* 获取数据库列表信息(已完成和待同步)
*
* @param DatabaseSyncService $syncService 同步服务实例
* @return array<string, mixed> 数据库列表信息
*/
private function getDatabaseList(DatabaseSyncService $syncService): array
{
$list = [
'completed' => [],
'pending' => [],
'processing' => [],
];
try {
$runtimePath = function_exists('runtime_path') ? runtime_path() : (config('app.runtime_path', base_path() . DIRECTORY_SEPARATOR . 'runtime'));
$progressFile = $runtimePath . DIRECTORY_SEPARATOR . 'database_sync_progress.json';
if (file_exists($progressFile)) {
$fp = fopen($progressFile, 'r');
if ($fp && flock($fp, LOCK_SH)) {
try {
$content = stream_get_contents($fp);
$progressData = json_decode($content, true);
if ($progressData) {
$checkpoints = $progressData['checkpoints'] ?? [];
$collectionsSnapshot = $progressData['collections_snapshot'] ?? [];
$currentDatabase = $progressData['current_database'] ?? null;
// 获取所有数据库名称
$allDatabases = array_keys($collectionsSnapshot);
foreach ($allDatabases as $database) {
$dbCheckpoints = $checkpoints[$database] ?? [];
// 检查该数据库的所有集合是否都已完成
$collections = $collectionsSnapshot[$database] ?? [];
$allCompleted = true;
$hasData = false;
foreach ($collections as $collection) {
$checkpoint = $dbCheckpoints[$collection] ?? null;
if ($checkpoint) {
$hasData = true;
if (!($checkpoint['completed'] ?? false)) {
$allCompleted = false;
break;
}
} else {
$allCompleted = false;
}
}
if ($database === $currentDatabase) {
$list['processing'][] = [
'name' => $database,
'collections' => count($collections),
'collections_completed' => count(array_filter($dbCheckpoints, fn($cp) => $cp['completed'] ?? false)),
];
} elseif ($allCompleted && $hasData) {
$list['completed'][] = [
'name' => $database,
'collections' => count($collections),
];
} else {
$list['pending'][] = [
'name' => $database,
'collections' => count($collections),
];
}
}
}
} finally {
flock($fp, LOCK_UN);
fclose($fp);
}
} else {
if ($fp) {
fclose($fp);
}
}
}
} catch (\Throwable $e) {
// 忽略错误,返回空列表
}
return $list;
}
}

View File

@@ -0,0 +1,169 @@
<?php
namespace app\controller;
use app\service\PersonMergeService;
use app\service\IdentifierService;
use app\repository\UserProfileRepository;
use app\repository\UserTagRepository;
use app\repository\UserPhoneRelationRepository;
use app\service\UserPhoneService;
use app\service\TagService;
use app\repository\TagDefinitionRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\ApiResponseHelper;
use app\utils\LoggerHelper;
use support\Request;
use support\Response;
/**
* 身份合并控制器
*
* 提供身份合并相关接口实现场景4手机号发现身份证后合并
*/
class PersonMergeController
{
/**
* 合并手机号到身份证场景4的实现
*
* 如果某个手机号发现了对应的身份证号,查询该身份下是否有标签,
* 如果有就会将对应的这个身份证号的所有标签重新计算同步。
*
* POST /api/person-merge/phone-to-id-card
*/
public function mergePhoneToIdCard(Request $request): Response
{
try {
LoggerHelper::logRequest('POST', '/api/person-merge/phone-to-id-card');
$rawBody = $request->rawBody();
if (empty($rawBody)) {
return ApiResponseHelper::error('请求体为空,请确保 Content-Type 为 application/json 并发送有效的 JSON 数据', 400);
}
$body = json_decode($rawBody, true);
if (json_last_error() !== JSON_ERROR_NONE) {
return ApiResponseHelper::error('JSON 格式错误: ' . json_last_error_msg(), 400);
}
// 验证必填字段
if (empty($body['phone_number'])) {
throw new \InvalidArgumentException('缺少必填字段phone_number');
}
if (empty($body['id_card'])) {
throw new \InvalidArgumentException('缺少必填字段id_card');
}
$phoneNumber = (string)$body['phone_number'];
$idCard = (string)$body['id_card'];
// 创建服务实例
$mergeService = new PersonMergeService(
new UserProfileRepository(),
new UserTagRepository(),
new UserPhoneService(
new UserPhoneRelationRepository()
),
new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
)
);
// 执行合并
$formalUserId = $mergeService->mergePhoneToIdCard($phoneNumber, $idCard);
LoggerHelper::logBusiness('person_merge_phone_to_id_card', [
'phone_number' => $phoneNumber,
'id_card_provided' => true,
'formal_user_id' => $formalUserId,
]);
return ApiResponseHelper::success([
'phone_number' => $phoneNumber,
'formal_user_id' => $formalUserId,
'message' => '身份合并成功,标签已重新计算',
]);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 合并临时人到正式人
*
* POST /api/person-merge/temporary-to-formal
*/
public function mergeTemporaryToFormal(Request $request): Response
{
try {
LoggerHelper::logRequest('POST', '/api/person-merge/temporary-to-formal');
$rawBody = $request->rawBody();
if (empty($rawBody)) {
return ApiResponseHelper::error('请求体为空,请确保 Content-Type 为 application/json 并发送有效的 JSON 数据', 400);
}
$body = json_decode($rawBody, true);
if (json_last_error() !== JSON_ERROR_NONE) {
return ApiResponseHelper::error('JSON 格式错误: ' . json_last_error_msg(), 400);
}
// 验证必填字段
if (empty($body['user_id'])) {
throw new \InvalidArgumentException('缺少必填字段user_id');
}
if (empty($body['id_card'])) {
throw new \InvalidArgumentException('缺少必填字段id_card');
}
$tempUserId = (string)$body['user_id'];
$idCard = (string)$body['id_card'];
// 创建服务实例
$mergeService = new PersonMergeService(
new UserProfileRepository(),
new UserTagRepository(),
new UserPhoneService(
new UserPhoneRelationRepository()
),
new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
)
);
// 执行合并
$formalUserId = $mergeService->mergeTemporaryToFormal($tempUserId, $idCard);
LoggerHelper::logBusiness('person_merge_temporary_to_formal', [
'temp_user_id' => $tempUserId,
'formal_user_id' => $formalUserId,
]);
return ApiResponseHelper::success([
'temp_user_id' => $tempUserId,
'formal_user_id' => $formalUserId,
'message' => '临时人已转为正式人,标签已重新计算',
]);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
}

View File

@@ -0,0 +1,308 @@
<?php
namespace app\controller;
use app\repository\TagCohortRepository;
use app\repository\UserTagRepository;
use app\service\TagService;
use app\repository\TagDefinitionRepository;
use app\repository\UserProfileRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\ApiResponseHelper;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid;
use support\Request;
use support\Response;
class TagCohortController
{
/**
* 获取人群快照列表
*
* GET /api/tag-cohorts
*/
public function list(Request $request): Response
{
try {
LoggerHelper::logRequest('GET', '/api/tag-cohorts');
$page = (int)($request->get('page') ?? 1);
$pageSize = (int)($request->get('page_size') ?? 20);
if ($page < 1) {
$page = 1;
}
if ($pageSize < 1 || $pageSize > 100) {
$pageSize = 20;
}
$cohortRepo = new TagCohortRepository();
$total = $cohortRepo->newQuery()->count();
$cohorts = $cohortRepo->newQuery()
->orderBy('created_at', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get();
$result = [];
foreach ($cohorts as $cohort) {
$result[] = [
'cohort_id' => $cohort->cohort_id,
'name' => $cohort->name,
'user_count' => $cohort->user_count ?? 0,
'created_at' => $cohort->created_at ? $cohort->created_at->format('Y-m-d H:i:s') : null,
];
}
LoggerHelper::logBusiness('get_tag_cohort_list', [
'total' => $total,
'page' => $page,
]);
return ApiResponseHelper::success([
'cohorts' => $result,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
]);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取人群快照详情
*
* GET /api/tag-cohorts/{cohort_id}
*/
public function detail(Request $request, string $cohortId): Response
{
try {
LoggerHelper::logRequest('GET', "/api/tag-cohorts/{$cohortId}");
$cohortRepo = new TagCohortRepository();
$cohort = $cohortRepo->newQuery()->where('cohort_id', $cohortId)->first();
if (!$cohort) {
return ApiResponseHelper::error('人群快照不存在', 404, 404);
}
$result = [
'cohort_id' => $cohort->cohort_id,
'name' => $cohort->name,
'description' => $cohort->description ?? '',
'conditions' => $cohort->conditions ?? [],
'logic' => $cohort->logic ?? 'AND',
'user_ids' => $cohort->user_ids ?? [],
'user_count' => $cohort->user_count ?? 0,
'created_at' => $cohort->created_at ? $cohort->created_at->format('Y-m-d H:i:s') : null,
];
LoggerHelper::logBusiness('get_tag_cohort_detail', [
'cohort_id' => $cohortId,
]);
return ApiResponseHelper::success($result);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 创建人群快照
*
* POST /api/tag-cohorts
*/
public function create(Request $request): Response
{
try {
LoggerHelper::logRequest('POST', '/api/tag-cohorts');
$rawBody = $request->rawBody();
if (empty($rawBody)) {
return ApiResponseHelper::error('请求体为空,请确保 Content-Type 为 application/json 并发送有效的 JSON 数据', 400);
}
$body = json_decode($rawBody, true);
if (json_last_error() !== JSON_ERROR_NONE) {
return ApiResponseHelper::error('JSON 格式错误: ' . json_last_error_msg(), 400);
}
// 验证必填字段
if (empty($body['name'])) {
throw new \InvalidArgumentException('缺少必填字段name');
}
if (empty($body['conditions']) || !is_array($body['conditions'])) {
throw new \InvalidArgumentException('缺少必填字段conditions必须为数组');
}
if (empty($body['user_ids']) || !is_array($body['user_ids'])) {
throw new \InvalidArgumentException('缺少必填字段user_ids必须为数组');
}
// 使用标签筛选服务获取用户列表
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
$conditions = $body['conditions'];
$logic = $body['logic'] ?? 'AND';
// 筛选用户(获取所有用户,不包含用户信息)
$filterResult = $tagService->filterUsersByTags(
$conditions,
$logic,
1,
10000, // 最多获取10000个用户
false
);
// 从返回结果中提取用户ID
$userIds = [];
if (isset($filterResult['users']) && is_array($filterResult['users'])) {
foreach ($filterResult['users'] as $user) {
if (isset($user['user_id'])) {
$userIds[] = $user['user_id'];
} elseif (is_string($user)) {
// 如果直接返回的是用户ID字符串
$userIds[] = $user;
}
}
}
// 如果用户提供了 user_ids使用提供的列表优先级更高
if (!empty($body['user_ids']) && is_array($body['user_ids'])) {
$userIds = $body['user_ids'];
}
// 创建人群快照
$cohortRepo = new TagCohortRepository();
$cohort = new TagCohortRepository();
$cohort->cohort_id = Uuid::uuid4()->toString();
$cohort->name = $body['name'];
$cohort->description = $body['description'] ?? '';
$cohort->conditions = $conditions;
$cohort->logic = $logic;
$cohort->user_ids = $userIds;
$cohort->user_count = count($userIds);
$cohort->created_by = $body['created_by'] ?? 'system';
$cohort->created_at = new \DateTime();
$cohort->updated_at = new \DateTime();
$cohort->save();
LoggerHelper::logBusiness('create_tag_cohort', [
'cohort_id' => $cohort->cohort_id,
'name' => $cohort->name,
'user_count' => $cohort->user_count,
]);
return ApiResponseHelper::success([
'cohort_id' => $cohort->cohort_id,
'name' => $cohort->name,
'user_count' => $cohort->user_count,
], '人群快照创建成功');
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除人群快照
*
* DELETE /api/tag-cohorts/{cohort_id}
*/
public function delete(Request $request, string $cohortId): Response
{
try {
LoggerHelper::logRequest('DELETE', "/api/tag-cohorts/{$cohortId}");
$cohortRepo = new TagCohortRepository();
$cohort = $cohortRepo->newQuery()->where('cohort_id', $cohortId)->first();
if (!$cohort) {
return ApiResponseHelper::error('人群快照不存在', 404, 404);
}
$cohort->delete();
LoggerHelper::logBusiness('delete_tag_cohort', [
'cohort_id' => $cohortId,
]);
return ApiResponseHelper::success(null, '人群快照删除成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 导出人群快照
*
* POST /api/tag-cohorts/{cohort_id}/export
*/
public function export(Request $request, string $cohortId): Response
{
try {
LoggerHelper::logRequest('POST', "/api/tag-cohorts/{$cohortId}/export");
$cohortRepo = new TagCohortRepository();
$cohort = $cohortRepo->newQuery()->where('cohort_id', $cohortId)->first();
if (!$cohort) {
return ApiResponseHelper::error('人群快照不存在', 404, 404);
}
$userIds = $cohort->user_ids ?? [];
$userProfileRepo = new UserProfileRepository();
// 获取用户信息
$users = [];
foreach ($userIds as $userId) {
$user = $userProfileRepo->findByUserId($userId);
if ($user) {
$users[] = [
'user_id' => $user->user_id,
'phone' => $user->phone ?? '',
'name' => $user->name ?? '',
];
}
}
// 生成 CSV 内容
$csvContent = "用户ID,手机号,姓名\n";
foreach ($users as $user) {
$csvContent .= sprintf(
"%s,%s,%s\n",
$user['user_id'],
$user['phone'],
$user['name']
);
}
LoggerHelper::logBusiness('export_tag_cohort', [
'cohort_id' => $cohortId,
'user_count' => count($users),
]);
// 返回 CSV 文件
return response($csvContent)
->header('Content-Type', 'text/csv; charset=utf-8')
->header('Content-Disposition', "attachment; filename=\"cohort_{$cohortId}.csv\"");
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
}

View File

@@ -0,0 +1,521 @@
<?php
namespace app\controller;
use app\repository\TagDefinitionRepository;
use app\repository\UserProfileRepository;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagService;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\ApiResponseHelper;
use app\utils\DataMaskingHelper;
use app\utils\LoggerHelper;
use support\Request;
use support\Response;
class TagController
{
/**
* 查询用户的标签列表
*
* GET /api/users/{user_id}/tags
*/
public function listByUser(Request $request): Response
{
try {
// 从请求路径中解析 user_id
$path = $request->path();
if (preg_match('#/api/users/([^/]+)/tags#', $path, $matches)) {
$userId = $matches[1];
} else {
// 如果路径解析失败,尝试从查询参数获取
$userId = $request->get('user_id');
if (!$userId) {
throw new \InvalidArgumentException('缺少 user_id 参数');
}
}
LoggerHelper::logRequest('GET', $path, ['user_id' => $userId]);
$userTagRepo = new UserTagRepository();
$tagDefRepo = new TagDefinitionRepository();
// 查询用户的所有标签
$userTags = $userTagRepo->newQuery()
->where('user_id', $userId)
->get();
// 关联标签定义信息
$result = [];
foreach ($userTags as $userTag) {
$tagDef = $tagDefRepo->newQuery()
->where('tag_id', $userTag->tag_id)
->first();
$result[] = [
'tag_id' => $userTag->tag_id,
'tag_code' => $tagDef ? $tagDef->tag_code : null,
'tag_name' => $tagDef ? $tagDef->tag_name : null,
'category' => $tagDef ? $tagDef->category : null,
'tag_value' => $userTag->tag_value,
'tag_value_type' => $userTag->tag_value_type,
'confidence' => $userTag->confidence,
'effective_time' => $userTag->effective_time,
'expire_time' => $userTag->expire_time,
'update_time' => $userTag->update_time,
];
}
LoggerHelper::logBusiness('get_user_tags', [
'user_id' => $userId,
'tag_count' => count($result),
]);
return ApiResponseHelper::success([
'user_id' => $userId,
'tags' => $result,
'count' => count($result),
]);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 更新/计算用户标签
*
* PUT /api/users/{user_id}/tags
*/
public function calculate(Request $request): Response
{
$startTime = microtime(true);
try {
// 从请求路径中解析 user_id
$path = $request->path();
if (preg_match('#/api/users/([^/]+)/tags#', $path, $matches)) {
$userId = $matches[1];
} else {
// 如果路径解析失败,尝试从查询参数获取
$userId = $request->get('user_id');
if (!$userId) {
throw new \InvalidArgumentException('缺少 user_id 参数');
}
}
LoggerHelper::logRequest('PUT', $path, ['user_id' => $userId]);
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
$tags = $tagService->calculateTags($userId);
$duration = microtime(true) - $startTime;
LoggerHelper::logBusiness('calculate_tags', [
'user_id' => $userId,
'updated_count' => count($tags),
], 'info');
LoggerHelper::logPerformance('tag_calculation', $duration, [
'user_id' => $userId,
'tag_count' => count($tags),
]);
return ApiResponseHelper::success([
'user_id' => $userId,
'updated_tags' => $tags,
'count' => count($tags),
]);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除用户的指定标签
*
* DELETE /api/users/{user_id}/tags/{tag_id}
*/
public function destroy(Request $request): Response
{
try {
// 从请求路径中解析 user_id 和 tag_id
$path = $request->path();
if (preg_match('#/api/users/([^/]+)/tags/([^/]+)#', $path, $matches)) {
$userId = $matches[1];
$tagId = $matches[2];
} else {
throw new \InvalidArgumentException('缺少 user_id 或 tag_id 参数');
}
LoggerHelper::logRequest('DELETE', $path, ['user_id' => $userId, 'tag_id' => $tagId]);
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
$deleted = $tagService->deleteUserTag($userId, $tagId);
if (!$deleted) {
return ApiResponseHelper::error('标签不存在', 404, 404);
}
LoggerHelper::logBusiness('tag_deleted', [
'user_id' => $userId,
'tag_id' => $tagId,
]);
return ApiResponseHelper::success(null, '标签删除成功');
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 根据标签筛选用户
*
* POST /api/tags/filter
*/
public function filter(Request $request): Response
{
try {
LoggerHelper::logRequest('POST', '/api/tags/filter');
$rawBody = $request->rawBody();
if (empty($rawBody)) {
return ApiResponseHelper::error('请求体为空,请确保 Content-Type 为 application/json 并发送有效的 JSON 数据', 400);
}
$body = json_decode($rawBody, true);
if (json_last_error() !== JSON_ERROR_NONE) {
return ApiResponseHelper::error('JSON 格式错误: ' . json_last_error_msg(), 400);
}
// 验证必填字段
if (empty($body['tag_conditions']) || !is_array($body['tag_conditions'])) {
throw new \InvalidArgumentException('缺少必填字段tag_conditions必须为数组');
}
// 验证条件格式
foreach ($body['tag_conditions'] as $condition) {
if (!isset($condition['tag_code']) || !isset($condition['operator']) || !isset($condition['value'])) {
throw new \InvalidArgumentException('每个条件必须包含 tag_code、operator 和 value 字段');
}
}
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
$conditions = $body['tag_conditions'];
$logic = $body['logic'] ?? 'AND';
$page = (int)($body['page'] ?? 1);
$pageSize = (int)($body['page_size'] ?? 20);
$includeUserInfo = (bool)($body['include_user_info'] ?? false);
if ($page < 1) {
$page = 1;
}
if ($pageSize < 1 || $pageSize > 100) {
$pageSize = 20;
}
$result = $tagService->filterUsersByTags(
$conditions,
$logic,
$page,
$pageSize,
$includeUserInfo
);
// 对返回的用户信息进行脱敏处理
if ($includeUserInfo && isset($result['users']) && is_array($result['users'])) {
foreach ($result['users'] as &$user) {
$user = DataMaskingHelper::maskArray($user, ['phone', 'email']);
}
unset($user);
}
LoggerHelper::logBusiness('filter_users_by_tags', [
'conditions_count' => count($conditions),
'logic' => $logic,
'result_count' => $result['total'] ?? 0,
]);
return ApiResponseHelper::success($result);
} catch (\InvalidArgumentException $e) {
return ApiResponseHelper::error($e->getMessage(), 400);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 批量初始化标签定义
*
* POST /api/tag-definitions/batch
*/
public function init(Request $request): Response
{
try {
LoggerHelper::logRequest('POST', '/api/tag-definitions/batch');
$initService = new \app\service\TagInitService(
new TagDefinitionRepository()
);
$initService->initBasicTags();
LoggerHelper::logBusiness('init_tags', []);
return ApiResponseHelper::success([
'message' => '标签初始化完成',
]);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取标签统计信息
*
* GET /api/tags/statistics
*/
public function statistics(Request $request): Response
{
try {
LoggerHelper::logRequest('GET', '/api/tags/statistics');
$tagId = $request->get('tag_id');
$startDate = $request->get('start_date');
$endDate = $request->get('end_date');
$userTagRepo = new UserTagRepository();
$tagDefRepo = new TagDefinitionRepository();
$result = [
'value_distribution' => [],
'trend_data' => [],
'coverage_stats' => [],
];
// 如果指定了 tag_id统计该标签的值分布
if ($tagId) {
$tagDef = $tagDefRepo->newQuery()->where('tag_id', $tagId)->first();
if ($tagDef) {
// 统计标签值分布
$userTags = $userTagRepo->newQuery()
->where('tag_id', $tagId)
->get(['tag_value']);
$valueCounts = [];
foreach ($userTags as $userTag) {
$value = (string)$userTag->tag_value;
if (!isset($valueCounts[$value])) {
$valueCounts[$value] = 0;
}
$valueCounts[$value]++;
}
// 按数量排序
arsort($valueCounts);
$valueCounts = array_slice($valueCounts, 0, 20, true);
foreach ($valueCounts as $value => $count) {
$result['value_distribution'][] = [
'value' => $value,
'count' => $count
];
}
// 统计标签覆盖度
$totalUsers = $userTagRepo->newQuery()
->distinct('user_id')
->count();
$taggedUsers = $userTagRepo->newQuery()
->where('tag_id', $tagId)
->distinct('user_id')
->count();
$result['coverage_stats'][] = [
'tag_id' => $tagId,
'tag_name' => $tagDef->tag_name ?? '',
'total_users' => $totalUsers,
'tagged_users' => $taggedUsers,
'coverage_rate' => $totalUsers > 0 ? round($taggedUsers / $totalUsers * 100, 2) : 0
];
}
} else {
// 统计所有标签的覆盖度
$tagDefs = $tagDefRepo->newQuery()->where('status', 1)->get();
$totalUsers = $userTagRepo->newQuery()->distinct('user_id')->count();
foreach ($tagDefs as $tagDef) {
$taggedUsers = $userTagRepo->newQuery()
->where('tag_id', $tagDef->tag_id)
->distinct('user_id')
->count();
$result['coverage_stats'][] = [
'tag_id' => $tagDef->tag_id,
'tag_name' => $tagDef->tag_name ?? '',
'total_users' => $totalUsers,
'tagged_users' => $taggedUsers,
'coverage_rate' => $totalUsers > 0 ? round($taggedUsers / $totalUsers * 100, 2) : 0
];
}
}
// 趋势数据(如果有时间范围)
if ($startDate && $endDate) {
$historyRepo = new TagHistoryRepository();
$start = new \DateTime($startDate);
$end = new \DateTime($endDate);
// 按日期统计标签变更次数
$trendData = [];
$current = clone $start;
while ($current <= $end) {
$dateStr = $current->format('Y-m-d');
$nextDay = clone $current;
$nextDay->modify('+1 day');
$count = $historyRepo->newQuery()
->where('change_time', '>=', $current)
->where('change_time', '<', $nextDay)
->count();
$trendData[] = [
'date' => $dateStr,
'count' => $count
];
$current->modify('+1 day');
}
$result['trend_data'] = $trendData;
}
LoggerHelper::logBusiness('get_tag_statistics', [
'tag_id' => $tagId,
'start_date' => $startDate,
'end_date' => $endDate,
]);
return ApiResponseHelper::success($result);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取标签历史记录
*
* GET /api/tags/history
*/
public function history(Request $request): Response
{
try {
LoggerHelper::logRequest('GET', '/api/tags/history');
$userId = $request->get('user_id');
$tagId = $request->get('tag_id');
$startDate = $request->get('start_date');
$endDate = $request->get('end_date');
$page = (int)($request->get('page') ?? 1);
$pageSize = (int)($request->get('page_size') ?? 20);
if ($page < 1) {
$page = 1;
}
if ($pageSize < 1 || $pageSize > 100) {
$pageSize = 20;
}
$historyRepo = new TagHistoryRepository();
$tagDefRepo = new TagDefinitionRepository();
$query = $historyRepo->newQuery();
if ($userId) {
$query->where('user_id', $userId);
}
if ($tagId) {
$query->where('tag_id', $tagId);
}
if ($startDate) {
$query->where('change_time', '>=', new \DateTime($startDate));
}
if ($endDate) {
$endDateTime = new \DateTime($endDate);
$endDateTime->modify('+1 day');
$query->where('change_time', '<', $endDateTime);
}
$total = $query->count();
$histories = $query->orderBy('change_time', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get();
$items = [];
foreach ($histories as $history) {
$tagDef = $tagDefRepo->newQuery()->where('tag_id', $history->tag_id)->first();
$items[] = [
'user_id' => $history->user_id,
'tag_id' => $history->tag_id,
'tag_name' => $tagDef ? $tagDef->tag_name : null,
'old_value' => $history->old_value,
'new_value' => $history->new_value,
'change_reason' => $history->change_reason,
'change_time' => $history->change_time ? $history->change_time->format('Y-m-d H:i:s') : null,
'operator' => $history->operator,
];
}
LoggerHelper::logBusiness('get_tag_history', [
'user_id' => $userId,
'tag_id' => $tagId,
'total' => $total,
]);
return ApiResponseHelper::success([
'items' => $items,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
]);
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
}

View File

@@ -0,0 +1,155 @@
<?php
namespace app\controller;
use app\repository\TagDefinitionRepository;
use app\utils\ApiResponseHelper;
use support\Request;
use support\Response;
/**
* 标签定义管理控制器
*/
class TagDefinitionController
{
/**
* 获取标签定义列表
*
* GET /api/tag-definitions
*/
public function list(Request $request): Response
{
try {
$repo = new TagDefinitionRepository();
$query = $repo->query();
// 筛选条件
if ($request->get('category')) {
$query->where('category', $request->get('category'));
}
if ($request->get('status')) {
$query->where('status', $request->get('status'));
}
if ($request->get('name')) {
$query->where('tag_name', 'like', '%' . $request->get('name') . '%');
}
$page = (int)($request->get('page', 1));
$pageSize = (int)($request->get('page_size', 20));
$total = $query->count();
$definitions = $query->orderBy('created_at', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get()
->toArray();
return ApiResponseHelper::success([
'definitions' => $definitions,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => ceil($total / $pageSize),
], '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取标签定义详情
*
* GET /api/tag-definitions/{tag_id}
*/
public function detail(Request $request, string $tagId): Response
{
try {
$repo = new TagDefinitionRepository();
$definition = $repo->find($tagId);
if (!$definition) {
return ApiResponseHelper::error('标签定义不存在', 404);
}
return ApiResponseHelper::success($definition->toArray(), '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 创建标签定义
*
* POST /api/tag-definitions
*/
public function create(Request $request): Response
{
try {
$data = $request->post();
$requiredFields = ['tag_code', 'tag_name', 'category'];
foreach ($requiredFields as $field) {
if (empty($data[$field])) {
return ApiResponseHelper::error("缺少必填字段: {$field}", 400);
}
}
$repo = new TagDefinitionRepository();
$definition = $repo->create($data);
return ApiResponseHelper::success($definition->toArray(), '创建成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 更新标签定义
*
* PUT /api/tag-definitions/{tag_id}
*/
public function update(Request $request, string $tagId): Response
{
try {
$data = $request->post();
$repo = new TagDefinitionRepository();
$definition = $repo->find($tagId);
if (!$definition) {
return ApiResponseHelper::error('标签定义不存在', 404);
}
$definition->fill($data);
$definition->save();
return ApiResponseHelper::success($definition->toArray(), '更新成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除标签定义
*
* DELETE /api/tag-definitions/{tag_id}
*/
public function delete(Request $request, string $tagId): Response
{
try {
$repo = new TagDefinitionRepository();
$definition = $repo->find($tagId);
if (!$definition) {
return ApiResponseHelper::error('标签定义不存在', 404);
}
$definition->delete();
return ApiResponseHelper::success(null, '删除成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
}

View File

@@ -0,0 +1,227 @@
<?php
namespace app\controller;
use app\service\TagTaskService;
use app\repository\TagTaskRepository;
use app\repository\TagTaskExecutionRepository;
use app\repository\UserProfileRepository;
use app\service\TagService;
use app\repository\TagDefinitionRepository;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\ApiResponseHelper;
use support\Request;
use support\Response;
/**
* 标签任务管理控制器
*/
class TagTaskController
{
public function __construct()
{
// 初始化服务(使用依赖注入或直接创建)
}
/**
* 创建标签任务
*
* POST /api/tag-tasks
*/
public function create(Request $request): Response
{
try {
$data = $request->post();
$requiredFields = ['name', 'task_type'];
foreach ($requiredFields as $field) {
if (empty($data[$field])) {
return ApiResponseHelper::error("缺少必填字段: {$field}", 400);
}
}
$service = $this->getService();
$task = $service->createTask($data);
return ApiResponseHelper::success($task, '任务创建成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 更新任务
*
* PUT /api/tag-tasks/{task_id}
*/
public function update(Request $request, string $taskId): Response
{
try {
$data = $request->post();
$service = $this->getService();
$result = $service->updateTask($taskId, $data);
return ApiResponseHelper::success(null, '任务更新成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 删除任务
*
* DELETE /api/tag-tasks/{task_id}
*/
public function delete(Request $request, string $taskId): Response
{
try {
$service = $this->getService();
$result = $service->deleteTask($taskId);
return ApiResponseHelper::success(null, '任务删除成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 启动任务
*
* POST /api/tag-tasks/{task_id}/start
*/
public function start(Request $request, string $taskId): Response
{
try {
$service = $this->getService();
$result = $service->startTask($taskId);
return ApiResponseHelper::success(null, '任务启动成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 暂停任务
*
* POST /api/tag-tasks/{task_id}/pause
*/
public function pause(Request $request, string $taskId): Response
{
try {
$service = $this->getService();
$result = $service->pauseTask($taskId);
return ApiResponseHelper::success(null, '任务暂停成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 停止任务
*
* POST /api/tag-tasks/{task_id}/stop
*/
public function stop(Request $request, string $taskId): Response
{
try {
$service = $this->getService();
$result = $service->stopTask($taskId);
return ApiResponseHelper::success(null, '任务停止成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务列表
*
* GET /api/tag-tasks
*/
public function list(Request $request): Response
{
try {
$filters = [
'status' => $request->get('status'),
'task_type' => $request->get('task_type'),
'name' => $request->get('name'),
];
$page = (int)($request->get('page', 1));
$pageSize = (int)($request->get('page_size', 20));
$service = $this->getService();
$result = $service->getTaskList($filters, $page, $pageSize);
return ApiResponseHelper::success($result, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务详情
*
* GET /api/tag-tasks/{task_id}
*/
public function detail(Request $request, string $taskId): Response
{
try {
$service = $this->getService();
$task = $service->getTask($taskId);
if ($task === null) {
return ApiResponseHelper::error('任务不存在', 404);
}
return ApiResponseHelper::success($task, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取任务执行记录
*
* GET /api/tag-tasks/{task_id}/executions
*/
public function executions(Request $request, string $taskId): Response
{
try {
$page = (int)($request->get('page', 1));
$pageSize = (int)($request->get('page_size', 20));
$service = $this->getService();
$result = $service->getExecutions($taskId, $page, $pageSize);
return ApiResponseHelper::success($result, '查询成功');
} catch (\Throwable $e) {
return ApiResponseHelper::exception($e);
}
}
/**
* 获取服务实例
*/
private function getService(): TagTaskService
{
return new TagTaskService(
new TagTaskRepository(),
new TagTaskExecutionRepository(),
new UserProfileRepository(),
new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
)
);
}
}

View File

@@ -0,0 +1,695 @@
<?php
namespace app\process;
use Workerman\Worker;
use Workerman\Timer;
use app\service\DataSource\DataSourceAdapterFactory;
use app\service\DataCollectionTaskService;
use app\service\DataSourceService;
use app\repository\DataCollectionTaskRepository;
use app\repository\DataSourceRepository;
use app\utils\QueueService;
use app\utils\RedisHelper;
use app\utils\LoggerHelper;
use Cron\CronExpression;
/**
* 数据采集任务调度器
*
* 职责:
* - 定时执行数据采集任务
* - 多进程并行处理(每个 Worker 处理分配给它的任务)
* - 使用分布式锁防止重复执行
* - 调用业务处理类执行数据采集逻辑
* - 支持从配置文件和数据库读取任务
*/
class DataSyncScheduler
{
private array $tasks = [];
private array $globalConfig = [];
private array $dataSourcesConfig = []; // 缓存的数据源配置(从数据库读取)
private DataCollectionTaskService $taskService;
private DataSourceService $dataSourceService;
private ?\app\service\TagTaskService $tagTaskService = null; // 延迟初始化
private array $runningTasks = []; // 正在运行的任务(实时监听模式)
public function onWorkerStart(Worker $worker): void
{
// 初始化任务服务
$this->taskService = new DataCollectionTaskService(
new DataCollectionTaskRepository()
);
// 初始化数据源服务
$this->dataSourceService = new DataSourceService(
new DataSourceRepository()
);
// 初始化标签任务服务(避免在多个方法中重复实例化)
$this->tagTaskService = new \app\service\TagTaskService(
new \app\repository\TagTaskRepository(),
new \app\repository\TagTaskExecutionRepository(),
new \app\repository\UserProfileRepository(),
new \app\service\TagService(
new \app\repository\TagDefinitionRepository(),
new \app\repository\UserProfileRepository(),
new \app\repository\UserTagRepository(),
new \app\repository\TagHistoryRepository(),
new \app\service\TagRuleEngine\SimpleRuleEngine()
)
);
// 加载任务采集配置(配置文件中的任务)
$taskConfig = config('data_collection_tasks', []);
$this->tasks = $taskConfig['tasks'] ?? [];
$this->globalConfig = $taskConfig['global'] ?? [];
// 从数据库加载数据源配置替代config('data_sources')
$this->loadDataSourcesConfig();
LoggerHelper::logBusiness('data_collection_scheduler_started', [
'worker_id' => $worker->id,
'total_workers' => $worker->count,
'config_task_count' => count($this->tasks),
'data_source_count' => count($this->dataSourcesConfig),
]);
// 加载配置文件中的任务
$this->loadConfigTasks($worker);
// 加载数据库中的动态任务
$this->loadDatabaseTasks($worker);
// 每30秒刷新一次数据库任务列表检查新任务、状态变更等
Timer::add(30, function () use ($worker) {
$this->refreshDatabaseTasks($worker);
});
}
/**
* 从数据库加载数据源配置
*/
private function loadDataSourcesConfig(): void
{
try {
$this->dataSourcesConfig = $this->dataSourceService->getAllEnabledDataSources();
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'loadDataSourcesConfig',
]);
// 如果加载失败,使用空数组,避免后续错误
$this->dataSourcesConfig = [];
}
}
/**
* 加载配置文件中的任务
*/
private function loadConfigTasks(Worker $worker): void
{
// 为每个启用的任务设置定时任务
foreach ($this->tasks as $taskId => $taskConfig) {
if (!($taskConfig['enabled'] ?? true)) {
continue;
}
// 检查是否应该由当前 Worker 处理(分片分配)
if (!$this->shouldHandleTask($taskId, $taskConfig, $worker)) {
continue;
}
// 获取调度配置
$schedule = $taskConfig['schedule'] ?? [];
// 如果调度被禁用,直接启动任务(持续运行)
if (!($schedule['enabled'] ?? true)) {
// 使用 Timer 延迟启动,避免阻塞 Worker 启动
Timer::add(0, function () use ($taskId, $taskConfig, $worker) {
$this->executeTask($taskId, $taskConfig, $worker);
}, [], false);
continue;
}
$cronExpression = $schedule['cron'] ?? '*/5 * * * *'; // 默认每5分钟
// 创建定时任务
$this->scheduleTask($taskId, $taskConfig, $cronExpression, $worker);
}
}
/**
* 加载数据库中的动态任务
*/
private function loadDatabaseTasks(Worker $worker): void
{
try {
// 加载数据采集任务
$dataCollectionTaskService = new \app\service\DataCollectionTaskService(
new \app\repository\DataCollectionTaskRepository()
);
$runningCollectionTasks = $dataCollectionTaskService->getRunningTasks();
\Workerman\Worker::safeEcho("[DataSyncScheduler] 从数据库加载到 " . count($runningCollectionTasks) . " 个运行中的任务 (worker_id={$worker->id})\n");
foreach ($runningCollectionTasks as $task) {
$taskId = $task['task_id'];
$taskName = $task['name'] ?? $taskId;
if (!$this->shouldHandleDatabaseTask($taskId, $worker)) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 跳过任务 [{$taskName}],由其他 Worker 处理 (worker_id={$worker->id})\n");
continue;
}
\Workerman\Worker::safeEcho("[DataSyncScheduler] ✓ 准备启动任务: [{$taskName}] task_id={$taskId} (worker_id={$worker->id})\n");
$taskConfig = $this->convertDatabaseTaskToConfig($task);
// 使用统一的任务启动方法
$this->startTask($taskId, $taskConfig, $worker);
}
// 加载标签任务(使用已初始化的服务)
$runningTagTasks = $this->tagTaskService->getTaskList(['status' => 'running'], 1, 1000);
foreach ($runningTagTasks['tasks'] as $task) {
$taskId = $task['task_id'];
if (!$this->shouldHandleDatabaseTask($taskId, $worker)) {
continue;
}
\Workerman\Worker::safeEcho("[DataSyncScheduler] ✓ 准备启动标签任务: task_id={$taskId} (worker_id={$worker->id})\n");
$taskConfig = $this->convertDatabaseTaskToConfig($task);
// 标签任务通常是批量执行,根据调度配置执行
$schedule = $taskConfig['schedule'] ?? [];
if (!($schedule['enabled'] ?? true)) {
// 立即执行一次
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 立即执行标签任务\n");
Timer::add(0, function () use ($taskId, $taskConfig, $worker) {
$this->executeTask($taskId, $taskConfig, $worker);
}, [], false);
} else {
$cronExpression = $schedule['cron'] ?? '0 2 * * *'; // 默认每天凌晨2点
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 定时执行标签任务Cron: {$cronExpression}\n");
$this->scheduleTask($taskId, $taskConfig, $cronExpression, $worker);
}
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'loadDatabaseTasks',
]);
}
}
/**
* 刷新数据库任务列表
*/
private function refreshDatabaseTasks(Worker $worker): void
{
try {
// 刷新数据采集任务
$dataCollectionTaskService = new \app\service\DataCollectionTaskService(
new \app\repository\DataCollectionTaskRepository()
);
$runningCollectionTasks = $dataCollectionTaskService->getRunningTasks();
$collectionTaskIds = array_column($runningCollectionTasks, 'task_id');
foreach ($runningCollectionTasks as $task) {
$taskId = $task['task_id'];
if (isset($this->runningTasks[$taskId])) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 任务 {$taskId} 已在运行中,跳过\n");
continue;
}
\Workerman\Worker::safeEcho("[DataSyncScheduler] 检测到新的运行中任务: {$taskId}\n");
$taskConfig = $this->convertDatabaseTaskToConfig($task);
// 使用统一的任务启动方法
$this->startTask($taskId, $taskConfig, $worker);
}
// 刷新标签任务(使用已初始化的服务)
$runningTagTasks = $this->tagTaskService->getTaskList(['status' => 'running'], 1, 1000);
$tagTaskIds = array_column($runningTagTasks['tasks'], 'task_id');
foreach ($runningTagTasks['tasks'] as $task) {
$taskId = $task['task_id'];
if (isset($this->runningTasks[$taskId])) {
continue;
}
if (RedisHelper::exists("tag_task:{$taskId}:start")) {
RedisHelper::del("tag_task:{$taskId}:start");
$taskConfig = $this->convertDatabaseTaskToConfig($task);
Timer::add(0, function () use ($taskId, $taskConfig, $worker) {
$this->executeTask($taskId, $taskConfig, $worker);
}, [], false);
$this->runningTasks[$taskId] = true;
}
}
// 检查是否有任务需要停止或暂停
$allTaskIds = array_merge($collectionTaskIds, $tagTaskIds);
foreach (array_keys($this->runningTasks) as $taskId) {
if (!in_array($taskId, $allTaskIds)) {
// 任务不在运行列表中了,移除
unset($this->runningTasks[$taskId]);
} elseif (RedisHelper::exists("data_collection_task:{$taskId}:stop") ||
RedisHelper::exists("tag_task:{$taskId}:stop")) {
// 检查停止标志
RedisHelper::del("data_collection_task:{$taskId}:stop");
RedisHelper::del("tag_task:{$taskId}:stop");
unset($this->runningTasks[$taskId]);
\Workerman\Worker::safeEcho("[DataSyncScheduler] 检测到停止信号,任务 {$taskId} 已停止\n");
} elseif (RedisHelper::exists("data_collection_task:{$taskId}:pause")) {
// 检查暂停标志
RedisHelper::del("data_collection_task:{$taskId}:pause");
unset($this->runningTasks[$taskId]);
\Workerman\Worker::safeEcho("[DataSyncScheduler] 检测到暂停信号,任务 {$taskId} 已暂停(正在执行的任务会在下次检查时停止)\n");
}
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'refreshDatabaseTasks',
]);
}
}
/**
* 判断是否应该处理数据库任务
*/
private function shouldHandleDatabaseTask(string $taskId, Worker $worker): bool
{
// 简单策略:按 Worker ID 取模分配
// 可以根据需要调整分配策略
return ($worker->id % $worker->count) === (hexdec(substr($taskId, 0, 8)) % $worker->count);
}
/**
* 将数据库任务转换为配置格式
*/
private function convertDatabaseTaskToConfig(array $task): array
{
// 判断任务类型:数据采集任务还是标签任务
// 如果任务有 data_source_id说明是数据采集任务
// 如果任务有 target_tag_ids说明是标签任务
if (isset($task['data_source_id']) && !empty($task['data_source_id'])) {
// 数据采集任务
// 根据 target_type 选择不同的 Handler
$targetType = $task['target_type'] ?? 'generic';
$handlerClass = \app\service\DataCollection\Handler\GenericCollectionHandler::class;
if ($targetType === 'consumption_record') {
$handlerClass = \app\service\DataCollection\Handler\ConsumptionCollectionHandler::class;
}
$config = [
'task_id' => $task['task_id'],
'name' => $task['name'] ?? '',
'data_source_id' => $task['data_source_id'],
'data_source' => $task['data_source_id'],
'database' => $task['database'] ?? '',
'collection' => $task['collection'] ?? null,
'collections' => $task['collections'] ?? null,
'target_type' => $targetType,
'target_data_source_id' => $task['target_data_source_id'] ?? null,
'target_database' => $task['target_database'] ?? null,
'target_collection' => $task['target_collection'] ?? null,
'mode' => $task['mode'] ?? 'batch',
'field_mappings' => $task['field_mappings'] ?? [],
'collection_field_mappings' => $task['collection_field_mappings'] ?? [],
'lookups' => $task['lookups'] ?? [],
'collection_lookups' => $task['collection_lookups'] ?? [],
'filter_conditions' => $task['filter_conditions'] ?? [],
'schedule' => $task['schedule'] ?? [
'enabled' => false,
'cron' => null,
],
'handler_class' => $handlerClass,
'batch_size' => $task['batch_size'] ?? 1000,
];
// 对于consumption_record类型的任务添加source_type字段如果存在
if ($targetType === 'consumption_record' && isset($task['source_type'])) {
$config['source_type'] = $task['source_type'];
}
return $config;
} elseif (isset($task['target_tag_ids']) || isset($task['task_type'])) {
// 标签任务
return [
'task_id' => $task['task_id'],
'name' => $task['name'] ?? '',
'task_type' => $task['task_type'] ?? 'full',
'target_tag_ids' => $task['target_tag_ids'] ?? [],
'user_scope' => $task['user_scope'] ?? ['type' => 'all'],
'schedule' => $task['schedule'] ?? [
'enabled' => false,
'cron' => null,
],
'config' => $task['config'] ?? [
'concurrency' => 10,
'batch_size' => 100,
'error_handling' => 'skip',
],
'handler_class' => \app\service\DataCollection\Handler\TagTaskHandler::class,
];
} else {
throw new \InvalidArgumentException("无法识别任务类型: {$task['task_id']}");
}
}
/**
* 判断当前 Worker 是否应该处理该任务(分片分配)
*
* @param string $taskId 任务ID
* @param array<string, mixed> $taskConfig 任务配置
* @param Worker $worker Worker 实例
* @return bool 是否应该处理
*/
private function shouldHandleTask(string $taskId, array $taskConfig, Worker $worker): bool
{
$sharding = $taskConfig['sharding'] ?? [];
$strategy = $sharding['strategy'] ?? 'none';
// 如果不需要分片,所有 Worker 都处理(但通过分布式锁保证只有一个执行)
if ($strategy === 'none') {
return true;
}
// by_database 策略:所有 Worker 都处理,但每个 Worker 处理不同的数据库(在 Handler 中分配)
if ($strategy === 'by_database') {
return true; // 所有 Worker 都处理,数据库分配在 Handler 中进行
}
// 其他分片策略:按 Worker ID 取模分配
$shardCount = $sharding['shard_count'] ?? 1;
if ($shardCount <= 1) {
// 如果 shard_count <= 1所有 Worker 都处理
return true;
}
$assignedShardId = $worker->id % $shardCount;
// 对于分片策略,只处理分配给当前 Worker 的分片
return $assignedShardId === ($worker->id % $shardCount);
}
/**
* 为任务设置定时任务
*
* @param string $taskId 任务ID
* @param array<string, mixed> $taskConfig 任务配置
* @param string $cronExpression Cron 表达式
* @param Worker $worker Worker 实例
* @return void
*/
private function scheduleTask(string $taskId, array $taskConfig, string $cronExpression, Worker $worker): void
{
try {
$cron = CronExpression::factory($cronExpression);
// 计算下次执行时间
$nextRunTime = $cron->getNextRunDate()->getTimestamp();
$now = time();
$delay = max(0, $nextRunTime - $now);
LoggerHelper::logBusiness('data_collection_task_scheduled', [
'task_id' => $taskId,
'task_name' => $taskConfig['name'] ?? $taskId,
'cron' => $cronExpression,
'next_run_time' => date('Y-m-d H:i:s', $nextRunTime),
'delay' => $delay,
]);
// 设置定时器延迟执行第一次然后每60秒检查一次
Timer::add(60, function () use ($taskId, $taskConfig, $cron, $worker) {
$now = time();
$nextRunTime = $cron->getNextRunDate()->getTimestamp();
// 如果到了执行时间允许1秒误差
if ($nextRunTime <= $now + 1) {
$this->executeTask($taskId, $taskConfig, $worker);
}
}, [], false, $delay);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'scheduleTask',
'task_id' => $taskId,
'cron' => $cronExpression,
]);
}
}
/**
* 执行采集任务
*
* @param string $taskId 任务ID
* @param array<string, mixed> $taskConfig 任务配置
* @param Worker $worker Worker 实例
* @return void
*/
private function executeTask(string $taskId, array $taskConfig, Worker $worker): void
{
// 在执行前,检查任务状态,如果已经是 completed就不应该再执行
try {
$task = $this->taskService->getTask($taskId);
if ($task && isset($task['status'])) {
if ($task['status'] === 'completed') {
\Workerman\Worker::safeEcho("[DataSyncScheduler] ⚠️ 任务已完成,跳过执行: task_id={$taskId}\n");
LoggerHelper::logBusiness('data_collection_skipped_completed', [
'task_id' => $taskId,
'worker_id' => $worker->id,
]);
return;
}
if (in_array($task['status'], ['stopped', 'paused', 'error'])) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] ⚠️ 任务状态为 {$task['status']},跳过执行: task_id={$taskId}\n");
LoggerHelper::logBusiness('data_collection_skipped_status', [
'task_id' => $taskId,
'status' => $task['status'],
'worker_id' => $worker->id,
]);
return;
}
}
} catch (\Throwable $e) {
// 如果查询任务状态失败,记录日志但继续执行(避免因为查询失败而阻塞任务)
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'executeTask',
'task_id' => $taskId,
'message' => '检查任务状态失败,继续执行任务',
]);
}
$sharding = $taskConfig['sharding'] ?? [];
$strategy = $sharding['strategy'] ?? 'none';
// 对于 by_database 策略,不使用全局锁,让所有 Worker 并行执行
// 每个 Worker 会在 Handler 中分配不同的数据库
$useLock = ($strategy !== 'by_database');
if ($useLock) {
$lockKey = "data_collection:{$taskId}";
$lockConfig = $this->globalConfig['distributed_lock'] ?? [];
$ttl = $lockConfig['ttl'] ?? 300;
$retryTimes = $lockConfig['retry_times'] ?? 3;
$retryDelay = $lockConfig['retry_delay'] ?? 1000;
// 尝试获取分布式锁
if (!RedisHelper::acquireLock($lockKey, $ttl, $retryTimes, $retryDelay)) {
LoggerHelper::logBusiness('data_collection_skipped_locked', [
'task_id' => $taskId,
'worker_id' => $worker->id,
]);
return;
}
}
try {
LoggerHelper::logBusiness('data_collection_started', [
'task_id' => $taskId,
'task_name' => $taskConfig['name'] ?? $taskId,
'worker_id' => $worker->id,
]);
// 控制台提示,标记具体哪个采集/同步任务被当前 Worker 拉起,方便排查任务是否真正执行
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤1-任务启动】执行采集任务task_id={$taskId}, task_name=" . ($taskConfig['name'] ?? $taskId) . ", worker_id={$worker->id}\n");
// 判断任务类型:数据采集任务需要数据源适配器,标签任务不需要
$handlerClass = $taskConfig['handler_class'] ?? '';
$isTagTask = strpos($handlerClass, 'TagTaskHandler') !== false;
$adapter = null;
if (!$isTagTask) {
// 数据采集任务:需要数据源适配器
// 支持两种配置方式:
// 1. 单数据源data_source用于普通采集任务
// 2. 多数据源source_data_source 和 target_data_source用于数据库同步任务
// 3. 动态任务data_source_id从数据库读取的任务
$dataSourceId = $taskConfig['data_source'] ?? $taskConfig['data_source_id'] ?? '';
$sourceDataSourceId = $taskConfig['source_data_source'] ?? '';
// 如果配置了 source_data_source说明是多数据源任务如数据库同步
// 这种情况下,只需要创建源数据源适配器,目标数据源由 Handler 内部处理
if (!empty($sourceDataSourceId)) {
$dataSourceId = $sourceDataSourceId;
}
// 从缓存或数据库获取数据源配置
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】开始查询数据源配置: data_source_id={$dataSourceId}\n");
$dataSourceConfig = $this->dataSourcesConfig[$dataSourceId] ?? null;
// 如果缓存中没有,尝试从数据库加载
if (empty($dataSourceConfig)) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】缓存中没有,从数据库加载: data_source_id={$dataSourceId}\n");
$dataSourceConfig = $this->dataSourceService->getDataSourceConfigById($dataSourceId);
if ($dataSourceConfig) {
// 更新缓存(使用原始 dataSourceId 作为 key
$this->dataSourcesConfig[$dataSourceId] = $dataSourceConfig;
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】✓ 数据源配置加载成功: host={$dataSourceConfig['host']}, port={$dataSourceConfig['port']}, database={$dataSourceConfig['database']}\n");
} else {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】✗ 数据源配置不存在: data_source_id={$dataSourceId}\n");
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】提示:请检查 data_sources 表中是否存在该数据源,或检查 name 字段是否匹配\n");
}
} else {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤2-查询数据源配置】✓ 从缓存获取数据源配置: host={$dataSourceConfig['host']}, port={$dataSourceConfig['port']}\n");
}
if (empty($dataSourceConfig)) {
throw new \InvalidArgumentException("数据源配置不存在: {$dataSourceId}");
}
// 创建数据源适配器
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤3-创建数据源适配器】开始创建适配器: type={$dataSourceConfig['type']}\n");
$adapter = DataSourceAdapterFactory::create(
$dataSourceConfig['type'],
$dataSourceConfig
);
// 确保适配器已连接
if (!$adapter->isConnected()) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤3-创建数据源适配器】适配器未连接,尝试连接...\n");
if (!$adapter->connect($dataSourceConfig)) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤3-创建数据源适配器】✗ 连接失败: data_source_id={$dataSourceId}\n");
throw new \RuntimeException("无法连接到数据源: {$dataSourceId}");
}
}
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤3-创建数据源适配器】✓ 数据源适配器创建并连接成功\n");
}
// 创建业务处理类(处理逻辑在业务代码中)
$handlerClass = $taskConfig['handler_class'] ?? '';
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】开始创建Handler: handler_class={$handlerClass}\n");
if (empty($handlerClass) || !class_exists($handlerClass)) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】✗ Handler类不存在: {$handlerClass}\n");
throw new \InvalidArgumentException("处理类不存在或未配置: {$handlerClass}");
}
// 实例化处理类
$handler = new $handlerClass();
// 检查处理类是否实现了采集接口
if (!method_exists($handler, 'collect')) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】✗ Handler未实现collect方法: {$handlerClass}\n");
throw new \InvalidArgumentException("处理类必须实现 collect 方法: {$handlerClass}");
}
// 添加任务ID到配置中
$taskConfig['task_id'] = $taskId;
// 添加 Worker 信息到配置中(用于多进程数据库分配)
$taskConfig['worker_id'] = $worker->id;
$taskConfig['worker_count'] = $worker->count;
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】✓ Handler创建成功开始调用collect方法\n");
// 调用业务处理类的采集方法
try {
$handler->collect($adapter, $taskConfig);
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】✓ Handler.collect()方法执行完成\n");
} catch (\Throwable $collectException) {
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】✗ Handler.collect()方法执行失败: " . $collectException->getMessage() . "\n");
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【步骤4-创建Handler】异常堆栈: " . $collectException->getTraceAsString() . "\n");
throw $collectException; // 重新抛出异常让外层catch处理
}
LoggerHelper::logBusiness('data_collection_completed', [
'task_id' => $taskId,
'task_name' => $taskConfig['name'] ?? $taskId,
]);
} catch (\Throwable $e) {
// 在控制台输出异常信息,方便排查
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【异常捕获】任务执行失败: task_id={$taskId}, error=" . $e->getMessage() . "\n");
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【异常捕获】异常文件: " . $e->getFile() . ":" . $e->getLine() . "\n");
\Workerman\Worker::safeEcho("[DataSyncScheduler] 【异常捕获】异常堆栈: " . $e->getTraceAsString() . "\n");
LoggerHelper::logError($e, [
'component' => 'DataSyncScheduler',
'action' => 'executeTask',
'task_id' => $taskId,
'worker_id' => $worker->id,
]);
} finally {
// 释放锁(仅在使用锁的情况下)
if (isset($useLock) && $useLock && isset($lockKey)) {
RedisHelper::releaseLock($lockKey);
}
}
}
/**
* 统一的任务启动方法
*
* @param string $taskId 任务ID
* @param array<string, mixed> $taskConfig 任务配置
* @param Worker $worker Worker 实例
* @return void
*/
private function startTask(string $taskId, array $taskConfig, Worker $worker): void
{
$taskName = $taskConfig['name'] ?? $taskId;
if ($taskConfig['mode'] === 'realtime') {
// 实时模式:立即启动,持续运行
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 实时模式任务: [{$taskName}]\n");
Timer::add(0, function () use ($taskId, $taskConfig, $worker) {
$this->executeTask($taskId, $taskConfig, $worker);
}, [], false);
$this->runningTasks[$taskId] = true;
} else {
// 批量模式:根据调度配置执行
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 批量模式任务: [{$taskName}]\n");
$schedule = $taskConfig['schedule'] ?? [];
if (!($schedule['enabled'] ?? true)) {
// 调度被禁用,立即执行一次
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 立即执行(调度已禁用)\n");
Timer::add(0, function () use ($taskId, $taskConfig, $worker) {
$this->executeTask($taskId, $taskConfig, $worker);
}, [], false);
} else {
// 使用 Cron 表达式定时执行
$cronExpression = $schedule['cron'] ?? '*/5 * * * *';
\Workerman\Worker::safeEcho("[DataSyncScheduler] → 定时执行Cron: {$cronExpression}\n");
$this->scheduleTask($taskId, $taskConfig, $cronExpression, $worker);
}
}
}
public function onWorkerStop(Worker $worker): void
{
LoggerHelper::logBusiness('data_collection_scheduler_stopped', [
'worker_id' => $worker->id,
]);
}
}

View File

@@ -0,0 +1,212 @@
<?php
namespace app\process;
use Workerman\Worker;
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Channel\AMQPChannel;
use PhpAmqpLib\Message\AMQPMessage;
use app\service\DataSyncService;
use app\repository\ConsumptionRecordRepository;
use app\repository\UserProfileRepository;
use app\utils\LoggerHelper;
/**
* 数据同步 Worker
*
* 职责:
* - 消费 RabbitMQ 队列中的数据同步消息
* - 调用 DataSyncService 同步数据到 MongoDB
* - 处理同步失败和重试
*/
class DataSyncWorker
{
private ?AMQPStreamConnection $connection = null;
private ?AMQPChannel $channel = null;
private array $config = [];
private ?DataSyncService $dataSyncService = null;
public function onWorkerStart(Worker $worker): void
{
$this->config = config('queue.connections.rabbitmq', []);
// 初始化 DataSyncService
$this->dataSyncService = new DataSyncService(
new ConsumptionRecordRepository(),
new UserProfileRepository()
);
try {
// 建立 RabbitMQ 连接
$this->connection = new AMQPStreamConnection(
$this->config['host'],
$this->config['port'],
$this->config['user'],
$this->config['password'],
$this->config['vhost'],
false, // insist
'AMQPLAIN', // login_method
null, // login_response
'en_US', // locale
$this->config['timeout'] ?? 10.0, // connection_timeout
$this->config['timeout'] ?? 10.0, // read_write_timeout
null, // context
false, // keepalive
$this->config['heartbeat'] ?? 0 // heartbeat
);
$this->channel = $this->connection->channel();
// 声明队列(确保队列存在)
$queueConfig = $this->config['queues']['data_sync'];
$this->channel->queue_declare(
$queueConfig['name'],
false, // passive
$queueConfig['durable'],
false, // exclusive
$queueConfig['auto_delete'],
false, // nowait
$queueConfig['arguments'] ?? []
);
// 设置 QoS批量处理
$consumerConfig = config('queue.consumer.data_sync', []);
$this->channel->basic_qos(
null, // prefetch_size
$consumerConfig['prefetch_count'] ?? 10, // prefetch_count每次处理10条消息
false // global
);
// 注册消费者回调
$this->channel->basic_consume(
$queueConfig['name'],
'', // consumer_tag
false, // no_local
false, // no_ack - 设为 false手动确认
false, // exclusive
false, // nowait
[$this, 'processMessage']
);
LoggerHelper::logBusiness('data_sync_worker_started', [
'worker_id' => $worker->id,
]);
// 循环消费消息
while ($this->channel->is_consuming()) {
$this->channel->wait();
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncWorker',
'action' => 'onWorkerStart',
'worker_id' => $worker->id,
]);
$this->closeConnection();
}
}
/**
* 处理 RabbitMQ 消息
*
* @param AMQPMessage $msg 消息对象
* @return void
*/
public function processMessage(AMQPMessage $msg): void
{
$payload = json_decode($msg->getBody(), true);
$sourceId = $payload['source_id'] ?? 'unknown';
$dataCount = count($payload['data'] ?? []);
if (empty($payload['data'])) {
LoggerHelper::logBusiness('data_sync_message_empty', [
'source_id' => $sourceId,
]);
$msg->ack(); // 确认消息,不再重试
return;
}
LoggerHelper::logBusiness('processing_data_sync_message', [
'source_id' => $sourceId,
'data_count' => $dataCount,
'worker_id' => Worker::getCurrentWorker()->id,
]);
try {
// 调用数据同步服务
$result = $this->dataSyncService->syncData($payload);
if ($result['success']) {
$msg->ack(); // 成功处理,发送 ACK
LoggerHelper::logBusiness('data_sync_message_processed', [
'source_id' => $sourceId,
'synced_count' => $result['synced_count'],
'skipped_count' => $result['skipped_count'],
]);
} else {
// 处理失败,但不重试(避免重复数据)
$msg->ack();
LoggerHelper::logError(new \RuntimeException("数据同步失败"), [
'component' => 'DataSyncWorker',
'action' => 'processMessage',
'source_id' => $sourceId,
'result' => $result,
]);
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncWorker',
'action' => 'processMessage',
'source_id' => $sourceId,
'payload' => $payload,
'worker_id' => Worker::getCurrentWorker()->id,
]);
// 业务错误不重试,直接 ACK避免重复数据
// 系统错误(如数据库连接断开)可以考虑重试,这里简化处理为直接 ACK
$msg->ack();
// 如果需要重试,可以 basic_reject($msg->delivery_info['delivery_tag'], true);
// 或者推送到死信队列
}
}
/**
* Worker 停止时关闭连接
*
* @param Worker $worker Worker 实例
* @return void
*/
public function onWorkerStop(Worker $worker): void
{
LoggerHelper::logBusiness('data_sync_worker_stopping', [
'worker_id' => $worker->id,
]);
$this->closeConnection();
}
/**
* 关闭 RabbitMQ 连接和通道
*
* @return void
*/
private function closeConnection(): void
{
try {
if ($this->channel) {
$this->channel->close();
}
if ($this->connection && $this->connection->isConnected()) {
$this->connection->close();
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncWorker',
'action' => 'closeConnection',
]);
} finally {
$this->channel = null;
$this->connection = null;
}
}
}

View File

@@ -0,0 +1,294 @@
<?php
namespace app\process;
use Workerman\Worker;
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Channel\AMQPChannel;
use PhpAmqpLib\Message\AMQPMessage;
use app\utils\QueueService;
use app\service\TagService;
use app\repository\TagDefinitionRepository;
use app\repository\UserProfileRepository;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\LoggerHelper;
/**
* 标签计算 Worker
*
* 职责:
* - 消费 RabbitMQ 队列中的标签计算消息
* - 异步触发标签计算
* - 处理计算失败和重试
*/
class TagCalculationWorker
{
private ?AMQPStreamConnection $connection = null;
private ?AMQPChannel $channel = null;
private array $config = [];
public function __construct()
{
// 构造函数Workerman 会自动调用
}
public function onWorkerStart(Worker $worker): void
{
// 设置内存限制512MB避免内存溢出
ini_set('memory_limit', '512M');
$this->config = config('queue.connections.rabbitmq', []);
// 使用 Timer 延迟初始化,避免阻塞 Worker 启动
\Workerman\Timer::add(0, function () use ($worker) {
$this->initConnection($worker);
}, [], false);
}
/**
* 初始化 RabbitMQ 连接
*
* @param Worker $worker Worker 实例
* @return void
*/
private function initConnection(Worker $worker): void
{
static $retryCount = 0;
$maxRetries = 10;
$retryInterval = 5;
try {
// 清理之前的连接
$this->closeConnection();
// 建立 RabbitMQ 连接
$this->connection = new AMQPStreamConnection(
$this->config['host'],
$this->config['port'],
$this->config['user'],
$this->config['password'],
$this->config['vhost'],
false, // insist
'AMQPLAIN', // login_method
null, // login_response
'en_US', // locale
$this->config['timeout'] ?? 10.0, // connection_timeout
$this->config['timeout'] ?? 10.0, // read_write_timeout
null, // context
false, // keepalive
$this->config['heartbeat'] ?? 0 // heartbeat
);
$this->channel = $this->connection->channel();
// 声明队列(确保队列存在)
$queueConfig = $this->config['queues']['tag_calculation'];
$this->channel->queue_declare(
$queueConfig['name'],
false, // passive
$queueConfig['durable'],
false, // exclusive
$queueConfig['auto_delete'],
false, // nowait
$queueConfig['arguments'] ?? []
);
// 设置 QoS每次只处理一条消息
$consumerConfig = config('queue.consumer.tag_calculation', []);
$this->channel->basic_qos(
null, // prefetch_size
$consumerConfig['prefetch_count'] ?? 1, // prefetch_count
false // global
);
// 开始消费消息
$this->channel->basic_consume(
$queueConfig['name'],
'', // consumer_tag
false, // no_local
$consumerConfig['no_ack'] ?? false, // no_ack
false, // exclusive
false, // nowait
[$this, 'processMessage'] // callback
);
LoggerHelper::logBusiness('tag_calculation_worker_started', [
'queue' => $queueConfig['name'],
'worker_id' => $worker->id,
]);
// 重置重试计数
$retryCount = 0;
// 监听消息(使用 Timer 定期检查,避免阻塞)
\Workerman\Timer::add(0.1, function () use ($worker) {
if ($this->channel && $this->channel->is_consuming()) {
try {
// 非阻塞方式检查消息
$this->channel->wait(null, false, 0); // timeout 0非阻塞
} catch (\PhpAmqpLib\Exception\AMQPTimeoutException $e) {
// 超时是正常的,继续等待
return;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'TagCalculationWorker',
'action' => 'channel_wait',
]);
// 连接断开,重新初始化(使用当前 Worker
$currentWorker = \Workerman\Worker::getCurrentWorker();
if ($currentWorker) {
$this->initConnection($currentWorker);
}
}
}
}, [], true); // 持续执行
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'TagCalculationWorker',
'action' => 'initConnection',
'retry_count' => $retryCount,
]);
// 重试连接最多重试10次
if ($retryCount < $maxRetries) {
$retryCount++;
\Workerman\Timer::add($retryInterval, function () use ($worker) {
$this->initConnection($worker);
}, [], false);
} else {
LoggerHelper::logError(new \RuntimeException("RabbitMQ 连接失败,已达到最大重试次数"), [
'component' => 'TagCalculationWorker',
'action' => 'initConnection',
'max_retries' => $maxRetries,
]);
}
}
}
/**
* 关闭连接
*
* @return void
*/
private function closeConnection(): void
{
try {
if ($this->channel !== null) {
$this->channel->close();
$this->channel = null;
}
if ($this->connection !== null && $this->connection->isConnected()) {
$this->connection->close();
$this->connection = null;
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'TagCalculationWorker',
'action' => 'closeConnection',
]);
}
}
/**
* 处理消息
*
* @param AMQPMessage $message
*/
public function processMessage(AMQPMessage $message): void
{
$startTime = microtime(true);
$deliveryTag = $message->getDeliveryTag();
try {
// 解析消息
$body = $message->getBody();
$data = json_decode($body, true);
if (json_last_error() !== JSON_ERROR_NONE) {
throw new \InvalidArgumentException('消息格式错误: ' . json_last_error_msg());
}
// 验证必要字段
if (empty($data['user_id'])) {
throw new \InvalidArgumentException('消息缺少 user_id 字段');
}
$userId = (string)$data['user_id'];
$tagIds = $data['tag_ids'] ?? null;
$triggerType = $data['trigger_type'] ?? 'consumption_record';
$recordId = $data['record_id'] ?? null;
LoggerHelper::logBusiness('tag_calculation_message_received', [
'user_id' => $userId,
'tag_ids' => $tagIds,
'trigger_type' => $triggerType,
'record_id' => $recordId,
]);
// 创建 TagService 实例
$tagService = new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
);
// 执行标签计算
$tags = $tagService->calculateTags($userId, $tagIds);
$duration = microtime(true) - $startTime;
LoggerHelper::logBusiness('tag_calculation_completed', [
'user_id' => $userId,
'updated_count' => count($tags),
'duration' => $duration,
]);
LoggerHelper::logPerformance('tag_calculation_async', $duration, [
'user_id' => $userId,
'tag_count' => count($tags),
]);
// 确认消息(只有在 no_ack = false 时才需要)
$consumerConfig = config('queue.consumer.tag_calculation', []);
if (!($consumerConfig['no_ack'] ?? false)) {
$message->ack();
}
} catch (\Throwable $e) {
$duration = microtime(true) - $startTime;
LoggerHelper::logError($e, [
'component' => 'TagCalculationWorker',
'action' => 'processMessage',
'delivery_tag' => $deliveryTag,
'duration' => $duration,
]);
// 根据错误类型决定是否重试
// 如果是业务逻辑错误(如用户不存在),直接确认消息,不重试
// 如果是系统错误(如数据库连接失败),可以重试
if ($e instanceof \InvalidArgumentException) {
// 业务逻辑错误,确认消息,不重试
$consumerConfig = config('queue.consumer.tag_calculation', []);
if (!($consumerConfig['no_ack'] ?? false)) {
$message->ack();
}
} else {
// 系统错误,拒绝消息并重新入队(重试)
$message->nack(false, true); // requeue = true
}
}
}
public function onWorkerStop(Worker $worker): void
{
$this->closeConnection();
LoggerHelper::logBusiness('tag_calculation_worker_stopped', [
'worker_id' => $worker->id,
]);
}
}

View File

@@ -0,0 +1,90 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 消费记录仓储
*
* 对应集合consumption_records_YYYYMM
* 第一阶段实现中可先固定写入当前月份集合例如consumption_records_202512。
*/
class ConsumptionRecordRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 默认集合名MongoDB Laravel 4.8+ 使用 $table
*
* 注意:实际使用时建议根据消费时间动态切换集合名。
*
* @var string
*/
protected $table = 'consumption_records_202512';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'record_id';
/**
* 主键非自增
*
* @var bool
*/
public $incrementing = false;
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 允许批量赋值的字段(按数据库字段文档筛选与当前闭环相关的部分)
*
* @var array<int, string>
*/
protected $fillable = [
'record_id',
'user_id',
'consume_time',
'amount',
'actual_amount',
'currency',
'store_id',
'status',
'create_time',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'amount' => 'float',
'actual_amount'=> 'float',
'consume_time' => 'datetime',
'create_time' => 'datetime',
'status' => 'int',
];
/**
* 禁用 Laravel 默认时间戳
*
* @var bool
*/
public $timestamps = false;
}

View File

@@ -0,0 +1,112 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 数据采集任务仓储
*
* 对应集合data_collection_tasks
* 存储所有通过前端配置创建的采集任务
*/
class DataCollectionTaskRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 集合名
*
* @var string
*/
protected $table = 'data_collection_tasks';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'task_id';
/**
* 主键非自增
*
* @var bool
*/
public $incrementing = false;
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 允许批量赋值的字段
*
* @var array<int, string>
*/
protected $fillable = [
'task_id',
'name',
'description',
'data_source_id', // 源数据源ID
'database', // 源数据库
'collection', // 源集合(单集合模式)
'collections', // 源集合列表(多集合模式)
'target_data_source_id', // 目标数据源ID
'target_database', // 目标数据库
'target_collection', // 目标集合
'target_type', // 目标类型consumption_record消费记录、generic通用集合
'mode', // batch: 批量采集, realtime: 实时监听
'field_mappings', // 字段映射配置(单集合模式)
'collection_field_mappings', // 字段映射配置(多集合模式)
'lookups', // 连表查询配置(单集合模式)
'collection_lookups', // 连表查询配置(多集合模式)
'filter_conditions', // 过滤条件
'schedule', // 调度配置
'status', // pending: 待启动, running: 运行中, paused: 已暂停, stopped: 已停止, error: 错误
'progress', // 进度信息
'statistics', // 统计信息
'created_by',
'created_at',
'updated_at',
];
/**
* 字段类型转换
*
* @var array<string, string>
*
* 注意MongoDB 原生支持数组类型,不需要对数组字段进行 cast
* 如果进行 cast当数据已经是数组时Laravel 可能会尝试使用 Json cast 导致错误
*/
protected $casts = [
// 数组字段不进行 cast让 MongoDB 直接处理
// 'field_mappings' => 'array',
// 'collection_field_mappings' => 'array',
// 'lookups' => 'array',
// 'collection_lookups' => 'array',
// 'filter_conditions' => 'array',
// 'schedule' => 'array',
// 'progress' => 'array',
// 'statistics' => 'array',
// 'collections' => 'array',
'created_at' => 'datetime',
'updated_at' => 'datetime',
];
/**
* 启用 Laravel 默认时间戳
*
* @var bool
*/
public $timestamps = true;
}

View File

@@ -0,0 +1,99 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 数据源仓储
*
* 对应集合data_sources
*/
class DataSourceRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名
*
* @var string
*/
protected $table = 'data_sources';
protected $primaryKey = 'data_source_id';
public $incrementing = false;
protected $keyType = 'string';
protected $fillable = [
'data_source_id',
'name',
'type',
'host',
'port',
'database',
'username',
'password',
'auth_source',
'options',
'description',
'status',
'is_tag_engine', // 是否为标签引擎数据库ckb数据库
'created_at',
'updated_at',
];
protected $casts = [
'port' => 'int',
'options' => 'array',
'status' => 'int',
'is_tag_engine' => 'bool',
'created_at' => 'datetime',
'updated_at' => 'datetime',
];
public $timestamps = true;
const CREATED_AT = 'created_at';
const UPDATED_AT = 'updated_at';
/**
* 转换为配置格式兼容原有的config格式
*
* @return array<string, mixed>
*/
public function toConfigArray(): array
{
$config = [
'type' => $this->type,
'host' => $this->host,
'port' => $this->port,
'database' => $this->database,
];
if ($this->username) {
$config['username'] = $this->username;
}
if ($this->password) {
$config['password'] = $this->password;
}
if ($this->auth_source) {
$config['auth_source'] = $this->auth_source;
}
if ($this->options) {
$config['options'] = $this->options;
}
return $config;
}
}

View File

@@ -0,0 +1,123 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 门店信息仓储
*
* 对应集合stores
* 字段定义参考:`提示词/202511/数据库字段.md` 中 stores 段落。
*/
class StoreRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名
*
* @var string
*/
protected $table = 'stores';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'store_id';
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 是否自增主键
*
* @var bool
*/
public $incrementing = false;
/**
* 允许批量赋值的字段
*
* @var array<int, string>
*/
protected $fillable = [
'store_id',
'store_code',
'store_name',
'store_type',
'store_level',
'industry_id',
'industry_detail_id',
'store_address',
'store_province',
'store_city',
'store_district',
'store_business_area',
'store_longitude',
'store_latitude',
'store_phone',
'status',
'create_time',
'update_time',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'store_longitude' => 'float',
'store_latitude' => 'float',
'status' => 'int',
'create_time' => 'datetime',
'update_time' => 'datetime',
];
/**
* 禁用 Laravel 默认时间戳
*
* @var bool
*/
public $timestamps = false;
/**
* 根据门店名称查找门店
*
* @param string $storeName 门店名称
* @return StoreRepository|null
*/
public function findByStoreName(string $storeName): ?StoreRepository
{
return $this->newQuery()
->where('store_name', $storeName)
->where('status', 0) // 只查询正常状态的门店
->first();
}
/**
* 根据门店编码查找门店
*
* @param string $storeCode 门店编码
* @return StoreRepository|null
*/
public function findByStoreCode(string $storeCode): ?StoreRepository
{
return $this->newQuery()
->where('store_code', $storeCode)
->first();
}
}

View File

@@ -0,0 +1,57 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 人群快照仓储
*
* 对应集合tag_cohorts
*/
class TagCohortRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名MongoDB Laravel 4.8+ 使用 $table
*
* @var string
*/
protected $table = 'tag_cohorts';
protected $primaryKey = 'cohort_id';
public $incrementing = false;
protected $keyType = 'string';
protected $fillable = [
'cohort_id',
'name',
'description',
'conditions',
'logic',
'user_ids',
'user_count',
'created_by',
'created_at',
'updated_at',
];
protected $casts = [
'conditions' => 'array',
'user_ids' => 'array',
'user_count' => 'int',
'created_at' => 'datetime',
'updated_at' => 'datetime',
];
public $timestamps = false;
}

View File

@@ -0,0 +1,64 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 标签定义仓储
*
* 对应集合tag_definitions
*/
class TagDefinitionRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名MongoDB Laravel 4.8+ 使用 $table
*
* @var string
*/
protected $table = 'tag_definitions';
protected $primaryKey = 'tag_id';
public $incrementing = false;
protected $keyType = 'string';
protected $fillable = [
'tag_id',
'tag_code',
'tag_name',
'category',
'rule_type',
'rule_config',
'update_frequency',
'priority',
'dependencies',
'description',
'status',
'version',
'create_time',
'update_time',
];
protected $casts = [
'rule_config' => 'array',
'dependencies' => 'array',
'priority' => 'int',
'status' => 'int',
'version' => 'int',
'create_time' => 'datetime',
'update_time' => 'datetime',
];
public $timestamps = false;
}

View File

@@ -0,0 +1,52 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 标签历史仓储
*
* 对应集合tag_history
*/
class TagHistoryRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名MongoDB Laravel 4.8+ 使用 $table
*
* @var string
*/
protected $table = 'tag_history';
protected $primaryKey = 'history_id';
public $incrementing = false;
protected $keyType = 'string';
protected $fillable = [
'history_id',
'user_id',
'tag_id',
'old_value',
'new_value',
'change_reason',
'change_time',
'operator',
];
protected $casts = [
'change_time' => 'datetime',
];
public $timestamps = false;
}

View File

@@ -0,0 +1,86 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 标签任务执行记录仓储
*
* 对应集合tag_task_executions
* 存储标签任务每次执行的记录
*/
class TagTaskExecutionRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 集合名
*
* @var string
*/
protected $table = 'tag_task_executions';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'execution_id';
/**
* 主键非自增
*
* @var bool
*/
public $incrementing = false;
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 允许批量赋值的字段
*
* @var array<int, string>
*/
protected $fillable = [
'execution_id',
'task_id',
'started_at',
'finished_at',
'status', // running, completed, failed, cancelled
'processed_users',
'success_count',
'error_count',
'error_message',
'created_at',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'started_at' => 'datetime',
'finished_at' => 'datetime',
'created_at' => 'datetime',
];
/**
* 启用 Laravel 默认时间戳
*
* @var bool
*/
public $timestamps = true;
}

View File

@@ -0,0 +1,95 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 标签任务仓储
*
* 对应集合tag_tasks
* 存储标签计算任务配置
*/
class TagTaskRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 集合名
*
* @var string
*/
protected $table = 'tag_tasks';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'task_id';
/**
* 主键非自增
*
* @var bool
*/
public $incrementing = false;
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 允许批量赋值的字段
*
* @var array<int, string>
*/
protected $fillable = [
'task_id',
'name',
'description',
'task_type', // full: 全量计算, incremental: 增量计算, specific: 指定用户
'target_tag_ids', // 目标标签ID列表
'user_scope', // 用户范围配置
'schedule', // 调度配置
'config', // 高级配置
'status', // pending, running, paused, stopped, completed, error
'progress', // 进度信息
'statistics', // 统计信息
'created_by',
'created_at',
'updated_at',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'target_tag_ids' => 'array',
'user_scope' => 'array',
'schedule' => 'array',
'config' => 'array',
'progress' => 'array',
'statistics' => 'array',
'created_at' => 'datetime',
'updated_at' => 'datetime',
];
/**
* 启用 Laravel 默认时间戳
*
* @var bool
*/
public $timestamps = true;
}

View File

@@ -0,0 +1,168 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
use MongoDB\Laravel\Eloquent\Builder;
/**
* 用户手机号关联仓储
*
* 对应集合user_phone_relations
* 字段定义参考:`提示词/数据库字段.md` 中 user_phone_relations 段落。
*/
class UserPhoneRelationRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名
*
* @var string
*/
protected $table = 'user_phone_relations';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'relation_id';
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 是否自增主键
*
* @var bool
*/
public $incrementing = false;
/**
* 允许批量赋值的字段
*
* @var array<int, string>
*/
protected $fillable = [
'relation_id',
'phone_number',
'phone_hash',
'user_id',
'effective_time',
'expire_time',
'is_active',
'type',
'is_verified',
'source',
'create_time',
'update_time',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'effective_time' => 'datetime',
'expire_time' => 'datetime',
'is_active' => 'boolean',
'is_verified' => 'boolean',
'create_time' => 'datetime',
'update_time' => 'datetime',
];
/**
* 禁用 Laravel 默认的 created_at/updated_at
*
* 我们使用 create_time / update_time 字段。
*
* @var bool
*/
public $timestamps = false;
/**
* 根据 relation_id 获取关联记录
*
* @param string $relationId
* @return self|null
*/
public function findByRelationId(string $relationId): ?self
{
/** @var Builder $query */
$query = static::query();
return $query->where('relation_id', $relationId)->first();
}
/**
* 根据手机号哈希查找当前有效的关联
*
* @param string $phoneHash
* @param \DateTimeInterface|null $atTime 查询时间点(默认为当前时间)
* @return self|null
*/
public function findActiveByPhoneHash(string $phoneHash, ?\DateTimeInterface $atTime = null): ?self
{
$queryTime = $atTime ?? new \DateTimeImmutable('now');
/** @var Builder $query */
$query = static::query();
return $query->where('phone_hash', $phoneHash)
->where('effective_time', '<=', $queryTime)
->where(function($q) use ($queryTime) {
$q->whereNull('expire_time')
->orWhere('expire_time', '>=', $queryTime);
})
->where('is_active', true)
->orderBy('effective_time', 'desc')
->first();
}
/**
* 根据用户ID查找所有手机号关联当前有效
*
* @param string $userId
* @param bool $includeHistory 是否包含历史记录
* @return array<self>
*/
public function findByUserId(string $userId, bool $includeHistory = false): array
{
/** @var Builder $query */
$query = static::query();
$query->where('user_id', $userId);
if (!$includeHistory) {
$query->where('is_active', true)
->whereNull('expire_time');
}
return $query->orderBy('effective_time', 'desc')->get()->all();
}
/**
* 根据手机号哈希查找所有历史关联记录
*
* @param string $phoneHash
* @return array<self>
*/
public function findHistoryByPhoneHash(string $phoneHash): array
{
/** @var Builder $query */
$query = static::query();
return $query->where('phone_hash', $phoneHash)
->orderBy('effective_time', 'desc')
->get()
->all();
}
}

View File

@@ -0,0 +1,227 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
use MongoDB\Laravel\Eloquent\Builder;
use MongoDB\Client;
/**
* 用户主信息仓储
*
* 对应集合user_profile
* 字段定义参考:`提示词/数据库字段.md` 中 user_profile 段落。
*/
class UserProfileRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 对应的 MongoDB 集合名MongoDB Laravel 4.8+ 使用 $table
*
* @var string
*/
protected $table = 'user_profile';
/**
* 主键字段
*
* @var string
*/
protected $primaryKey = 'user_id';
/**
* 主键类型
*
* @var string
*/
protected $keyType = 'string';
/**
* 是否自增主键
*
* @var bool
*/
public $incrementing = false;
/**
* 允许批量赋值的字段
*
* 仅保留与标签系统直接相关的字段,其他字段按需补充。
*
* @var array<int, string>
*/
protected $fillable = [
'user_id',
'id_card_hash',
'id_card_encrypted',
'id_card_type',
'name',
'phone',
'address',
'email',
'gender',
'birthday',
'total_amount',
'total_count',
'last_consume_time',
'tags_update_time',
'is_temporary', // 是否为临时人true=临时人false=正式人)
'merged_from_user_id', // 如果是从临时人合并而来记录原user_id
'status',
'create_time',
'update_time',
];
/**
* 字段类型转换
*
* @var array<string, string>
*/
protected $casts = [
'total_amount' => 'float',
'total_count' => 'int',
'last_consume_time' => 'datetime',
'tags_update_time' => 'datetime',
'birthday' => 'datetime',
'is_temporary' => 'bool',
'status' => 'int',
'create_time' => 'datetime',
'update_time' => 'datetime',
];
/**
* 禁用 Laravel 默认的 created_at/updated_at
*
* 我们使用 create_time / update_time 字段。
*
* @var bool
*/
public $timestamps = false;
/**
* 根据 user_id 获取用户记录(不存在时返回 null
*/
public function findByUserId(string $userId): ?self
{
/** @var Builder $query */
$query = static::query();
return $query->where('user_id', $userId)->first();
}
/**
* 创建或更新用户的基础统计信息
*
* 仅用于标签系统第一阶段:更新总金额、总次数、最后消费时间。
*
* @param string $userId
* @param float $amount 本次消费金额
* @param \DateTimeInterface $consumeTime 消费时间
*/
public function increaseStats(string $userId, float $amount, \DateTimeInterface $consumeTime): self
{
$now = new \DateTimeImmutable('now');
/** @var self|null $user */
$user = $this->findByUserId($userId);
if (!$user) {
$user = new self([
'user_id' => $userId,
'total_amount' => $amount,
'total_count' => 1,
'last_consume_time'=> $consumeTime,
'is_temporary' => true, // 默认创建为临时人
'status' => 0,
'create_time' => $now,
'update_time' => $now,
]);
} else {
$user->total_amount = (float)$user->total_amount + $amount;
$user->total_count = (int)$user->total_count + 1;
// 只在消费时间更晚时更新
if (!$user->last_consume_time || $consumeTime > $user->last_consume_time) {
$user->last_consume_time = $consumeTime;
}
$user->update_time = $now;
}
$user->save();
return $user;
}
/**
* 根据身份证哈希查找用户
*
* @param string $idCardHash 身份证哈希值
* @return self|null
*/
public function findByIdCardHash(string $idCardHash): ?self
{
return $this->newQuery()
->where('id_card_hash', $idCardHash)
->where('status', 0)
->first();
}
/**
* 查找所有临时人
*
* @return \Illuminate\Database\Eloquent\Collection
*/
public function findTemporaryUsers()
{
return $this->newQuery()
->where('is_temporary', true)
->where('status', 0)
->get();
}
/**
* 标记用户为正式人
*
* @param string $userId
* @param string|null $idCardHash 身份证哈希
* @param string|null $idCardEncrypted 加密的身份证
* @param string|null $idCard 原始身份证号(用于提取基础信息,可选)
* @return bool
*/
public function markAsFormal(string $userId, ?string $idCardHash = null, ?string $idCardEncrypted = null, ?string $idCard = null): bool
{
$user = $this->findByUserId($userId);
if (!$user) {
return false;
}
$user->is_temporary = false;
if ($idCardHash !== null) {
$user->id_card_hash = $idCardHash;
}
if ($idCardEncrypted !== null) {
$user->id_card_encrypted = $idCardEncrypted;
}
// 如果有原始身份证号,自动提取基础信息(如果字段为空才更新)
if ($idCard !== null && !empty($idCard)) {
$idCardInfo = \app\utils\IdCardHelper::extractInfo($idCard);
if ($idCardInfo['birthday'] !== null && $user->birthday === null) {
$user->birthday = $idCardInfo['birthday'];
}
// 只有当性别解析成功且当前值为 null 时才更新0 也被认为是未设置)
if ($idCardInfo['gender'] > 0 && ($user->gender === null || $user->gender === 0)) {
$user->gender = $idCardInfo['gender'];
}
}
$user->update_time = new \DateTimeImmutable('now');
return $user->save();
}
}

View File

@@ -0,0 +1,62 @@
<?php
namespace app\repository;
use MongoDB\Laravel\Eloquent\Model;
/**
* 用户标签仓储
*
* 对应分片集合user_tags_shard_{0-15}
* 第一阶段实现中先固定一个集合,后续可根据 user_id 哈希路由到不同分片。
*/
class UserTagRepository extends Model
{
/**
* 指定使用的数据库连接
*
* @var string
*/
protected $connection = 'mongodb';
/**
* 默认集合名MongoDB Laravel 4.8+ 使用 $table
*
* @var string
*/
protected $table = 'user_tags_shard_0';
/**
* 复合主键在 Mongo Eloquent 中由业务自行控制
*
* 这里仍然使用默认 _id 作为物理主键user_id + tag_id 通过唯一索引约束(由运维侧负责)。
*/
protected $primaryKey = '_id';
protected $keyType = 'string';
protected $fillable = [
'user_id',
'tag_id',
'tag_value',
'tag_value_type',
'confidence',
'effective_time',
'expire_time',
'create_time',
'update_time',
];
protected $casts = [
'tag_value' => 'string',
'confidence' => 'float',
'effective_time' => 'datetime',
'expire_time' => 'datetime',
'create_time' => 'datetime',
'update_time' => 'datetime',
];
public $timestamps = false;
}

View File

@@ -0,0 +1,282 @@
<?php
namespace app\service;
use app\repository\ConsumptionRecordRepository;
use app\repository\UserProfileRepository;
use app\service\TagService;
use app\service\IdentifierService;
use app\utils\QueueService;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 消费记录服务
*
* 职责:
* - 校验基础入参
* - 根据手机号/身份证解析用户IDperson_id
* - 写入消费记录集合
* - 更新用户在 user_profile 中的基础统计信息
*/
class ConsumptionService
{
public function __construct(
protected ConsumptionRecordRepository $consumptionRecordRepository,
protected UserProfileRepository $userProfileRepository,
protected IdentifierService $identifierService,
protected ?TagService $tagService = null
) {
}
/**
* 创建一条消费记录并更新用户统计信息
*
* 支持两种方式指定用户:
* 1. 直接提供 user_id
* 2. 提供 phone_number 或 id_card或两者系统自动解析用户ID
*
* @param array<string, mixed> $payload
* @return array<string, mixed>|null 如果手机号和身份证号都为空返回null跳过该记录
*/
public function createRecord(array $payload): ?array
{
// 基础必填字段校验
foreach (['amount', 'actual_amount', 'store_id', 'consume_time'] as $field) {
if (!isset($payload[$field]) || $payload[$field] === '') {
throw new \InvalidArgumentException("缺少必填字段:{$field}");
}
}
$amount = (float)$payload['amount'];
$actual = (float)$payload['actual_amount'];
$storeId = (string)$payload['store_id'];
$consumeTime = new \DateTimeImmutable((string)$payload['consume_time']);
// 解析用户ID优先使用user_id如果没有则通过手机号/身份证解析
$userId = null;
if (!empty($payload['user_id'])) {
$userId = (string)$payload['user_id'];
} else {
// 通过手机号或身份证解析用户ID
$phoneNumber = trim($payload['phone_number'] ?? '');
$idCard = trim($payload['id_card'] ?? '');
// 如果手机号和身份证号都为空,直接跳过该记录
if (empty($phoneNumber) && empty($idCard)) {
LoggerHelper::logBusiness('consumption_record_skipped_no_identifier', [
'reason' => 'phone_number and id_card are both empty',
'consume_time' => $consumeTime->format('Y-m-d H:i:s'),
]);
return null;
}
// 传入 consume_time 作为查询时间点
$userId = $this->identifierService->resolvePersonId($phoneNumber, $idCard, $consumeTime);
// 如果同时提供了手机号和身份证,检查是否需要合并
if (!empty($phoneNumber) && !empty($idCard)) {
$userId = $this->handleMergeIfNeeded($phoneNumber, $idCard, $userId, $consumeTime);
}
}
$now = new \DateTimeImmutable('now');
$recordId = UuidGenerator::uuid4()->toString();
// 写入消费记录
$record = new ConsumptionRecordRepository();
$record->record_id = $recordId;
$record->user_id = $userId;
$record->consume_time = $consumeTime;
$record->amount = $amount;
$record->actual_amount = $actual;
$record->currency = $payload['currency'] ?? 'CNY';
$record->store_id = $storeId;
$record->status = 0;
$record->create_time = $now;
$record->save();
// 更新用户统计信息
$user = $this->userProfileRepository->increaseStats($userId, $actual, $consumeTime);
// 触发标签计算(异步方式)
$tags = [];
$useAsync = getenv('TAG_CALCULATION_ASYNC') !== 'false'; // 默认使用异步,可通过环境变量关闭
if ($useAsync) {
// 异步方式:推送到消息队列
try {
$success = QueueService::pushTagCalculation([
'user_id' => $userId,
'tag_ids' => null, // null 表示计算所有 real_time 标签
'trigger_type' => 'consumption_record',
'record_id' => $recordId,
'timestamp' => time(),
]);
if ($success) {
LoggerHelper::logBusiness('tag_calculation_queued', [
'user_id' => $userId,
'record_id' => $recordId,
]);
} else {
// 如果推送失败,降级到同步调用
LoggerHelper::logBusiness('tag_calculation_queue_failed_fallback', [
'user_id' => $userId,
'record_id' => $recordId,
]);
$useAsync = false;
}
} catch (\Throwable $e) {
// 如果队列服务异常,降级到同步调用
LoggerHelper::logError($e, [
'component' => 'ConsumptionService',
'action' => 'pushTagCalculation',
'user_id' => $userId,
]);
$useAsync = false;
}
}
// 同步方式(降级方案或配置关闭异步时使用)
if (!$useAsync && $this->tagService) {
try {
$tags = $this->tagService->calculateTags($userId);
} catch (\Throwable $e) {
// 标签计算失败不影响消费记录写入,只记录错误
LoggerHelper::logError($e, [
'component' => 'ConsumptionService',
'action' => 'calculateTags',
'user_id' => $userId,
]);
}
}
return [
'record_id' => $recordId,
'user_id' => $userId,
'user' => [
'total_amount' => $user->total_amount ?? 0,
'total_count' => $user->total_count ?? 0,
'last_consume_time' => $user->last_consume_time,
],
'tags' => $tags, // 异步模式下为空数组,同步模式下包含标签信息
'tag_calculation_mode' => $useAsync ? 'async' : 'sync',
];
}
/**
* 当手机号和身份证号同时出现时,检查是否需要合并用户
*
* @param string $phoneNumber 手机号
* @param string $idCard 身份证号
* @param string $currentUserId 当前解析出的用户ID
* @param \DateTimeInterface $consumeTime 消费时间
* @return string 最终使用的用户ID
*/
private function handleMergeIfNeeded(
string $phoneNumber,
string $idCard,
string $currentUserId,
\DateTimeInterface $consumeTime
): string {
// 通过身份证查找用户
$userIdByIdCard = $this->identifierService->resolvePersonIdByIdCard($idCard);
// 在消费时间点查询手机号关联(使用反射或公共方法)
$userPhoneService = new \app\service\UserPhoneService(
new \app\repository\UserPhoneRelationRepository()
);
$userIdByPhone = $userPhoneService->findUserByPhone($phoneNumber, $consumeTime);
// 如果身份证找到用户A手机号关联到用户B且A≠B
if ($userIdByIdCard && $userIdByPhone && $userIdByIdCard !== $userIdByPhone) {
// 检查用户B是否为临时用户
$userB = $this->userProfileRepository->findByUserId($userIdByPhone);
$userA = $this->userProfileRepository->findByUserId($userIdByIdCard);
if ($userB && $userB->is_temporary) {
// 情况1用户B是临时用户 → 合并到正式用户A
// 需要合并服务,动态创建
$tagService = $this->tagService ?? new TagService(
new \app\repository\TagDefinitionRepository(),
$this->userProfileRepository,
new \app\repository\UserTagRepository(),
new \app\repository\TagHistoryRepository(),
new \app\service\TagRuleEngine\SimpleRuleEngine()
);
$mergeService = new PersonMergeService(
$this->userProfileRepository,
new \app\repository\UserTagRepository(),
$userPhoneService,
$tagService
);
// 合并临时用户B到正式用户A
$mergeService->mergeUsers($userIdByPhone, $userIdByIdCard);
// 将旧的手机关联标记为过期(使用消费时间作为过期时间)
$userPhoneService->removePhoneFromUser($userIdByPhone, $phoneNumber, $consumeTime);
// 建立新的手机关联到用户A使用消费时间作为生效时间
$userPhoneService->addPhoneToUser($userIdByIdCard, $phoneNumber, [
'source' => 'merge_after_id_card_binding',
'effective_time' => $consumeTime,
'type' => 'personal',
]);
LoggerHelper::logBusiness('auto_merge_triggered', [
'phone_number' => $phoneNumber,
'source_user_id' => $userIdByPhone,
'target_user_id' => $userIdByIdCard,
'consume_time' => $consumeTime->format('Y-m-d H:i:s'),
'reason' => 'temporary_user_merge',
]);
return $userIdByIdCard;
} elseif ($userA && !$userA->is_temporary && $userB && !$userB->is_temporary) {
// 情况2两者都是正式用户如酒店预订代订场景
// 策略:以身份证为准,消费记录归属到身份证用户,但手机号关联保持不变
// 原因:手机号和身份证同时出现时,身份证更可信;但手机号可能是代订,不应自动转移
// 检查手机号在消费时间点是否已经关联到用户A可能之前已经转移过
$phoneRelationAtTime = $userPhoneService->findUserByPhone($phoneNumber, $consumeTime);
if ($phoneRelationAtTime !== $userIdByIdCard) {
// 手机号在消费时间点还未关联到身份证用户
// 记录异常情况,但不强制转移手机号(可能是代订场景)
LoggerHelper::logBusiness('phone_id_card_mismatch_formal_users', [
'phone_number' => $phoneNumber,
'phone_user_id' => $userIdByPhone,
'id_card_user_id' => $userIdByIdCard,
'consume_time' => $consumeTime->format('Y-m-d H:i:s'),
'decision' => 'use_id_card_user',
'note' => '正式用户冲突,以身份证为准(可能是代订场景)',
]);
}
// 以身份证用户为准返回身份证用户ID
return $userIdByIdCard;
} else {
// 其他情况(理论上不应该发生),记录日志并返回身份证用户
LoggerHelper::logBusiness('phone_id_card_mismatch_unknown', [
'phone_number' => $phoneNumber,
'phone_user_id' => $userIdByPhone,
'id_card_user_id' => $userIdByIdCard,
'phone_user_is_temporary' => $userB ? $userB->is_temporary : 'unknown',
'id_card_user_is_temporary' => $userA ? $userA->is_temporary : 'unknown',
'consume_time' => $consumeTime->format('Y-m-d H:i:s'),
]);
// 默认返回身份证用户(更可信)
return $userIdByIdCard;
}
}
return $currentUserId;
}
}

View File

@@ -0,0 +1,115 @@
<?php
namespace app\service\DataCollection\Handler;
use app\repository\DataSourceRepository;
use app\service\DataSourceService;
use app\utils\LoggerHelper;
use app\utils\MongoDBHelper;
use MongoDB\Client;
/**
* 数据采集 Handler 基类
*
* 提供通用的数据采集功能:
* - MongoDB 客户端创建
* - 数据源配置获取
* - 目标数据源连接
* - 公共服务实例IdentifierService、ConsumptionService、StoreService
*/
abstract class BaseCollectionHandler
{
use Trait\DataCollectionHelperTrait;
protected DataSourceService $dataSourceService;
protected \app\service\IdentifierService $identifierService;
protected \app\service\ConsumptionService $consumptionService;
protected \app\service\StoreService $storeService;
public function __construct()
{
$this->dataSourceService = new DataSourceService(
new DataSourceRepository()
);
// 初始化公共服务(避免在子类中重复实例化)
$this->identifierService = new \app\service\IdentifierService(
new \app\repository\UserProfileRepository(),
new \app\service\UserPhoneService(
new \app\repository\UserPhoneRelationRepository()
)
);
$this->consumptionService = new \app\service\ConsumptionService(
new \app\repository\ConsumptionRecordRepository(),
new \app\repository\UserProfileRepository(),
$this->identifierService
);
$this->storeService = new \app\service\StoreService(
new \app\repository\StoreRepository()
);
}
/**
* 获取 MongoDB 客户端
*
* @param array<string, mixed> $taskConfig 任务配置
* @return Client MongoDB 客户端实例
* @throws \InvalidArgumentException 如果数据源配置不存在
*/
protected function getMongoClient(array $taskConfig): Client
{
$dataSourceId = $taskConfig['data_source_id']
?? $taskConfig['data_source']
?? 'sync_mongodb';
$dataSourceConfig = $this->dataSourceService->getDataSourceConfigById($dataSourceId);
if (empty($dataSourceConfig)) {
throw new \InvalidArgumentException("数据源配置不存在: {$dataSourceId}");
}
return MongoDBHelper::createClient($dataSourceConfig);
}
/**
* 连接到目标数据源
*
* @param string $targetDataSourceId 目标数据源ID
* @param string|null $targetDatabase 目标数据库名(可选,默认使用数据源配置中的数据库)
* @return array{client: Client, database: \MongoDB\Database, dbName: string, config: array} 连接信息
* @throws \InvalidArgumentException 如果目标数据源配置不存在
*/
protected function connectToTargetDataSource(
string $targetDataSourceId,
?string $targetDatabase = null
): array {
$targetDataSourceConfig = $this->dataSourceService->getDataSourceConfigById($targetDataSourceId);
if (empty($targetDataSourceConfig)) {
throw new \InvalidArgumentException("目标数据源配置不存在: {$targetDataSourceId}");
}
$client = MongoDBHelper::createClient($targetDataSourceConfig);
$dbName = $targetDatabase ?? $targetDataSourceConfig['database'] ?? 'ckb';
$database = $client->selectDatabase($dbName);
return [
'client' => $client,
'database' => $database,
'dbName' => $dbName,
'config' => $targetDataSourceConfig,
];
}
/**
* 采集数据(抽象方法,由子类实现)
*
* @param mixed $adapter 数据源适配器
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
abstract public function collect($adapter, array $taskConfig): void;
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,427 @@
<?php
namespace app\service\DataCollection\Handler;
use app\service\DataSource\DataSourceAdapterInterface;
use app\service\DatabaseSyncService;
use app\service\DataSourceService;
use app\repository\DataSourceRepository;
use app\utils\LoggerHelper;
use MongoDB\Client;
/**
* 数据库同步采集处理类
*
* 职责:
* - 从源数据库同步数据到目标数据库
* - 支持全量同步和增量同步Change Streams
* - 处理同步进度和错误恢复
*/
class DatabaseSyncHandler
{
private DatabaseSyncService $syncService;
private array $taskConfig;
private int $progressTimerId = 0; // 进度日志定时器ID
/**
* 采集/同步数据库
*
* @param DataSourceAdapterInterface $adapter 数据源适配器(源数据库)
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
public function collect(DataSourceAdapterInterface $adapter, array $taskConfig): void
{
$this->taskConfig = $taskConfig;
$taskId = $taskConfig['task_id'] ?? '';
$taskName = $taskConfig['name'] ?? '数据库同步';
LoggerHelper::logBusiness('database_sync_collection_started', [
'task_id' => $taskId,
'task_name' => $taskName,
]);
// 控制台直接输出一条提示,方便在启动时观察数据库同步任务是否真正开始执行
error_log("[DatabaseSyncHandler] 数据库同步任务已启动task_id={$taskId}, task_name={$taskName}");
try {
// 创建 DatabaseSyncService使用任务配置中的源和目标数据源
$this->syncService = $this->createSyncService($taskConfig);
// 获取要同步的数据库列表
$databases = $this->getDatabasesToSync($taskConfig);
if (empty($databases)) {
LoggerHelper::logBusiness('database_sync_no_databases', [
'task_id' => $taskId,
'message' => '没有找到要同步的数据库',
]);
return;
}
// 启动进度日志定时器(定期输出同步进度)
$this->startProgressTimer($taskConfig);
// 是否执行全量同步(从业务配置中获取)
$businessConfig = $this->getBusinessConfig();
$fullSyncEnabled = $businessConfig['change_stream']['full_sync_on_start'] ?? false;
if ($fullSyncEnabled) {
// 执行全量同步
$this->performFullSync($databases, $taskConfig);
}
// 启动增量同步监听Change Streams
$this->startIncrementalSync($databases, $taskConfig);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DatabaseSyncHandler',
'action' => 'collect',
'task_id' => $taskId,
]);
throw $e;
}
}
/**
* 获取业务配置(从独立配置文件或使用默认值)
*
* @return array<string, mixed> 业务配置
*/
private function getBusinessConfig(): array
{
// 可以从独立配置文件读取,或使用默认值
// 这里使用默认值,业务逻辑统一在代码中管理
return [
// 数据库同步配置
'databases' => [], // 空数组表示同步所有数据库
'exclude_databases' => ['admin', 'local', 'config'], // 排除的系统数据库
'exclude_collections' => ['system.profile', 'system.js'], // 排除的系统集合
// Change Streams 配置
'change_stream' => [
'batch_size' => 100,
'max_await_time_ms' => 1000,
'full_sync_on_start' => true, // 首次启动时是否执行全量同步
'full_sync_batch_size' => 1000,
],
// 重试配置
'retry' => [
'max_connect_retries' => 10,
'retry_interval' => 5,
'max_sync_retries' => 3,
'sync_retry_interval' => 2,
],
// 性能配置
'performance' => [
'concurrent_databases' => 5,
'concurrent_collections' => 10,
'batch_write_size' => 5000,
// 为了让断点续传逻辑简单可靠,这里关闭集合级并行同步
// 后续如果需要再做更复杂的分片断点策略,可以重新打开
'enable_parallel_sync' => false,
'max_parallel_tasks_per_collection' => 4,
'documents_per_task' => 100000,
],
// 监控配置
'monitoring' => [
'log_sync' => true,
'log_detail' => false,
'stats_interval' => 10, // 每10秒输出一次进度日志
],
];
}
/**
* 创建 DatabaseSyncService 实例
*
* @param array<string, mixed> $taskConfig 任务配置
* @return DatabaseSyncService
*/
private function createSyncService(array $taskConfig): DatabaseSyncService
{
// 从数据库获取源和目标数据源配置
$dataSourceService = new DataSourceService(new DataSourceRepository());
$sourceDataSourceId = $taskConfig['source_data_source'] ?? 'kr_mongodb';
$targetDataSourceId = $taskConfig['target_data_source'] ?? 'sync_mongodb';
$sourceConfig = $dataSourceService->getDataSourceConfigById($sourceDataSourceId);
$targetConfig = $dataSourceService->getDataSourceConfigById($targetDataSourceId);
if (empty($sourceConfig) || empty($targetConfig)) {
throw new \InvalidArgumentException("数据源配置不存在: source={$sourceDataSourceId}, target={$targetDataSourceId}");
}
// 获取业务配置(统一在代码中管理)
$businessConfig = $this->getBusinessConfig();
// 构建同步配置
$syncConfig = [
'enabled' => true,
'source' => [
'host' => $sourceConfig['host'],
'port' => $sourceConfig['port'],
'username' => $sourceConfig['username'] ?? '',
'password' => $sourceConfig['password'] ?? '',
'auth_source' => $sourceConfig['auth_source'] ?? 'admin',
'options' => array_merge([
'connectTimeoutMS' => 10000,
'socketTimeoutMS' => 30000,
'serverSelectionTimeoutMS' => 10000,
'heartbeatFrequencyMS' => 10000,
], $sourceConfig['options'] ?? []),
],
'target' => [
'host' => $targetConfig['host'],
'port' => $targetConfig['port'],
'username' => $targetConfig['username'] ?? '',
'password' => $targetConfig['password'] ?? '',
'auth_source' => $targetConfig['auth_source'] ?? 'admin',
'options' => array_merge([
'connectTimeoutMS' => 10000,
'socketTimeoutMS' => 30000,
'serverSelectionTimeoutMS' => 10000,
], $targetConfig['options'] ?? []),
],
'sync' => [
'databases' => $businessConfig['databases'],
'exclude_databases' => $businessConfig['exclude_databases'],
'exclude_collections' => $businessConfig['exclude_collections'],
'change_stream' => $businessConfig['change_stream'],
'retry' => $businessConfig['retry'],
'performance' => $businessConfig['performance'],
],
'monitoring' => $businessConfig['monitoring'],
];
// 直接传递配置给 DatabaseSyncService 构造函数
return new DatabaseSyncService($syncConfig);
}
/**
* 获取要同步的数据库列表
*
* @param array<string, mixed> $taskConfig 任务配置
* @return array<string> 数据库名称列表
*/
private function getDatabasesToSync(array $taskConfig): array
{
return $this->syncService->getDatabasesToSync();
}
/**
* 执行全量同步(支持多进程数据库级并行)
*
* @param array<string> $databases 数据库列表
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
private function performFullSync(array $databases, array $taskConfig): void
{
// 获取 Worker 信息(用于多进程分配)
$workerId = $taskConfig['worker_id'] ?? 0;
$workerCount = $taskConfig['worker_count'] ?? 1;
// 分配数据库给当前 Worker负载均衡算法
$assignedDatabases = $this->assignDatabasesToWorker($databases, $workerId, $workerCount);
LoggerHelper::logBusiness('database_sync_full_sync_start', [
'worker_id' => $workerId,
'worker_count' => $workerCount,
'total_databases' => count($databases),
'assigned_databases' => $assignedDatabases,
'assigned_count' => count($assignedDatabases),
]);
foreach ($assignedDatabases as $databaseName) {
try {
$this->syncService->fullSyncDatabase($databaseName);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DatabaseSyncHandler',
'action' => 'performFullSync',
'database' => $databaseName,
'worker_id' => $workerId,
]);
// 继续同步其他数据库
}
}
LoggerHelper::logBusiness('database_sync_full_sync_completed', [
'worker_id' => $workerId,
'databases' => $assignedDatabases,
]);
}
/**
* 分配数据库给当前 Worker负载均衡算法
*
* 策略:
* 1. 按数据库大小排序(小库优先,提升完成感)
* 2. 使用贪心算法:每次分配给当前负载最小的 Worker
* 3. 考虑 Worker 当前处理的数据库数量
*
* @param array<string> $databases 数据库列表(已按大小排序)
* @param int $workerId 当前 Worker ID
* @param int $workerCount Worker 总数
* @return array<string> 分配给当前 Worker 的数据库列表
*/
private function assignDatabasesToWorker(array $databases, int $workerId, int $workerCount): array
{
// 如果只有一个 Worker返回所有数据库
if ($workerCount <= 1) {
return $databases;
}
// 方案A简单取模分配快速实现
// 适用于数据库数量较多且大小相近的场景
$assignedDatabases = [];
foreach ($databases as $index => $databaseName) {
if ($index % $workerCount === $workerId) {
$assignedDatabases[] = $databaseName;
}
}
// 方案B负载均衡分配推荐但需要数据库大小信息
// 由于 getDatabasesToSync 已经按大小排序,简单取模即可实现较好的负载均衡
// 如果后续需要更精确的负载均衡,可以从 DatabaseSyncService 获取数据库大小信息
return $assignedDatabases;
}
/**
* 启动增量同步监听(支持多进程数据库级并行)
*
* @param array<string> $databases 数据库列表
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
private function startIncrementalSync(array $databases, array $taskConfig): void
{
// 获取 Worker 信息(用于多进程分配)
$workerId = $taskConfig['worker_id'] ?? 0;
$workerCount = $taskConfig['worker_count'] ?? 1;
// 分配数据库给当前 Worker与全量同步使用相同的分配策略
$assignedDatabases = $this->assignDatabasesToWorker($databases, $workerId, $workerCount);
LoggerHelper::logBusiness('database_sync_incremental_sync_start', [
'worker_id' => $workerId,
'worker_count' => $workerCount,
'total_databases' => count($databases),
'assigned_databases' => $assignedDatabases,
'assigned_count' => count($assignedDatabases),
]);
// 为分配给当前 Worker 的数据库启动监听(在后台进程中)
foreach ($assignedDatabases as $databaseName) {
// 使用 Timer 在后台启动监听,避免阻塞
\Workerman\Timer::add(0, function () use ($databaseName) {
try {
$this->syncService->watchDatabase($databaseName);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DatabaseSyncHandler',
'action' => 'startIncrementalSync',
'database' => $databaseName,
]);
// 重试逻辑(从业务配置中获取)
$businessConfig = $this->getBusinessConfig();
$retryConfig = $businessConfig['retry'] ?? [];
$maxRetries = $retryConfig['max_connect_retries'] ?? 10;
$retryInterval = $retryConfig['retry_interval'] ?? 5;
static $retryCount = [];
if (!isset($retryCount[$databaseName])) {
$retryCount[$databaseName] = 0;
}
if ($retryCount[$databaseName] < $maxRetries) {
$retryCount[$databaseName]++;
\Workerman\Timer::add($retryInterval, function () use ($databaseName) {
$this->startIncrementalSync([$databaseName], $this->taskConfig);
}, [], false);
}
}
}, [], false);
}
}
/**
* 启动进度日志定时器
*
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
private function startProgressTimer(array $taskConfig): void
{
$businessConfig = $this->getBusinessConfig();
$statsInterval = $businessConfig['monitoring']['stats_interval'] ?? 10; // 默认10秒输出一次进度
// 使用 Workerman Timer 定期输出进度
$this->progressTimerId = \Workerman\Timer::add($statsInterval, function () use ($taskConfig) {
try {
// 重新加载最新进度(从文件读取)
$this->syncService->loadProgress();
$progress = $this->syncService->getProgress();
$stats = $this->syncService->getStats();
// 输出格式化的进度信息
$progressInfo = [
'task_id' => $taskConfig['task_id'] ?? '',
'task_name' => $taskConfig['name'] ?? '数据库同步',
'status' => $progress['status'],
'progress_percent' => $progress['progress_percent'] . '%',
'current_database' => $progress['current_database'] ?? '无',
'current_collection' => $progress['current_collection'] ?? '无',
'databases' => "{$progress['databases']['completed']}/{$progress['databases']['total']}",
'collections' => "{$progress['collections']['completed']}/{$progress['collections']['total']}",
'documents' => "{$progress['documents']['processed']}/{$progress['documents']['total']}",
'documents_inserted' => $stats['documents_inserted'],
'documents_updated' => $stats['documents_updated'],
'documents_deleted' => $stats['documents_deleted'],
'errors' => $stats['errors'],
'elapsed_time' => round($progress['time']['elapsed_seconds'], 2) . 's',
'estimated_remaining' => $progress['time']['estimated_remaining_seconds']
? round($progress['time']['estimated_remaining_seconds'], 2) . 's'
: '计算中...',
];
// 输出到日志
LoggerHelper::logBusiness('database_sync_progress_report', $progressInfo);
// 如果状态是错误,输出错误信息
if ($progress['status'] === 'error' && isset($progress['last_error'])) {
LoggerHelper::logBusiness('database_sync_error_info', [
'error_message' => $progress['last_error']['message'] ?? '未知错误',
'error_database' => $progress['error_database'] ?? '未知',
'error_collection' => $progress['last_error']['collection'] ?? '未知',
]);
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DatabaseSyncHandler',
'action' => 'startProgressTimer',
]);
}
});
}
/**
* 停止进度日志定时器
*
* @return void
*/
public function stopProgressTimer(): void
{
if ($this->progressTimerId > 0) {
\Workerman\Timer::del($this->progressTimerId);
$this->progressTimerId = 0;
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,74 @@
<?php
namespace app\service\DataCollection\Handler;
use app\service\TagTaskService;
use app\repository\TagTaskRepository;
use app\repository\TagTaskExecutionRepository;
use app\repository\UserProfileRepository;
use app\service\TagService;
use app\repository\TagDefinitionRepository;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\LoggerHelper;
/**
* 标签任务处理类
*
* 职责:
* - 执行标签计算任务
* - 批量遍历用户数据打标签
*/
class TagTaskHandler
{
/**
* 执行标签任务
*
* @param mixed $adapter 数据源适配器(标签任务不需要)
* @param array<string, mixed> $taskConfig 任务配置
* @return void
*/
public function collect($adapter, array $taskConfig): void
{
$taskId = $taskConfig['task_id'] ?? '';
$taskName = $taskConfig['name'] ?? '标签任务';
LoggerHelper::logBusiness('tag_task_handler_started', [
'task_id' => $taskId,
'task_name' => $taskName,
]);
try {
// 创建TagTaskService实例
$tagTaskService = new TagTaskService(
new TagTaskRepository(),
new TagTaskExecutionRepository(),
new UserProfileRepository(),
new TagService(
new TagDefinitionRepository(),
new UserProfileRepository(),
new UserTagRepository(),
new TagHistoryRepository(),
new SimpleRuleEngine()
)
);
// 执行任务
$tagTaskService->executeTask($taskId);
LoggerHelper::logBusiness('tag_task_handler_completed', [
'task_id' => $taskId,
'task_name' => $taskName,
]);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'TagTaskHandler',
'action' => 'collect',
'task_id' => $taskId,
]);
throw $e;
}
}
}

View File

@@ -0,0 +1,172 @@
<?php
namespace app\service\DataCollection\Handler\Trait;
use MongoDB\BSON\UTCDateTime;
/**
* 数据采集辅助方法 Trait
*
* 提供通用的工具方法给各个 Handler 使用
*/
trait DataCollectionHelperTrait
{
/**
* 将 MongoDB 文档转换为数组
*
* @param mixed $document MongoDB 文档对象或数组
* @return array<string, mixed> 数组格式的数据
*/
protected function convertMongoDocumentToArray($document): array
{
if (is_array($document)) {
return $document;
}
if (is_object($document) && method_exists($document, 'toArray')) {
return $document->toArray();
}
return json_decode(json_encode($document), true) ?? [];
}
/**
* 解析日期时间字符串
*
* @param mixed $dateTimeStr 日期时间字符串或对象
* @return \DateTimeImmutable|null 解析后的日期时间对象
*/
protected function parseDateTime($dateTimeStr): ?\DateTimeImmutable
{
if (empty($dateTimeStr)) {
return null;
}
// 如果是 MongoDB 的 UTCDateTime 对象
if ($dateTimeStr instanceof UTCDateTime) {
return \DateTimeImmutable::createFromMutable($dateTimeStr->toDateTime());
}
// 如果是 DateTime 对象
if ($dateTimeStr instanceof \DateTime || $dateTimeStr instanceof \DateTimeImmutable) {
if ($dateTimeStr instanceof \DateTime) {
return \DateTimeImmutable::createFromMutable($dateTimeStr);
}
return $dateTimeStr;
}
// 尝试解析字符串
try {
return new \DateTimeImmutable((string)$dateTimeStr);
} catch (\Exception $e) {
\app\utils\LoggerHelper::logBusiness('datetime_parse_failed', [
'input' => $dateTimeStr,
'error' => $e->getMessage(),
]);
return null;
}
}
/**
* 解析金额
*
* @param mixed $amount 金额字符串或数字
* @return float 解析后的金额
*/
protected function parseAmount($amount): float
{
if (is_numeric($amount)) {
return (float)$amount;
}
if (is_string($amount)) {
// 移除所有非数字字符(除了小数点)
$cleaned = preg_replace('/[^\d.]/', '', $amount);
return (float)$cleaned;
}
return 0.0;
}
/**
* 过滤手机号中的非数字字符
*
* @param string $phoneNumber 原始手机号
* @return string 过滤后的手机号(只包含数字)
*/
protected function filterPhoneNumber(string $phoneNumber): string
{
// 移除所有非数字字符
return preg_replace('/\D/', '', $phoneNumber);
}
/**
* 验证手机号格式
*
* @param string $phone 手机号(已经过滤过非数字字符)
* @return bool 是否有效11位数字1开头
*/
protected function isValidPhone(string $phone): bool
{
// 如果为空,直接返回 false
if (empty($phone)) {
return false;
}
// 中国大陆手机号11位数字以1开头
return preg_match('/^1[3-9]\d{9}$/', $phone) === 1;
}
/**
* 根据消费时间生成月份集合名
*
* @param string $baseCollectionName 基础集合名
* @param mixed $dateTimeStr 日期时间字符串或对象
* @return string 带月份后缀的集合名consumption_records_202512
*/
protected function getMonthlyCollectionName(string $baseCollectionName, $dateTimeStr = null): string
{
$consumeTime = $this->parseDateTime($dateTimeStr);
if ($consumeTime === null) {
$consumeTime = new \DateTimeImmutable();
}
$monthSuffix = $consumeTime->format('Ym');
return "{$baseCollectionName}_{$monthSuffix}";
}
/**
* 转换为 MongoDB UTCDateTime
*
* @param mixed $dateTimeStr 日期时间字符串或对象
* @return UTCDateTime|null MongoDB UTCDateTime 对象
*/
protected function convertToUTCDateTime($dateTimeStr): ?UTCDateTime
{
if (empty($dateTimeStr)) {
return null;
}
// 如果已经是 UTCDateTime直接返回
if ($dateTimeStr instanceof UTCDateTime) {
return $dateTimeStr;
}
// 如果是 DateTime 对象
if ($dateTimeStr instanceof \DateTime || $dateTimeStr instanceof \DateTimeImmutable) {
return new UTCDateTime($dateTimeStr->getTimestamp() * 1000);
}
// 尝试解析字符串
try {
$dateTime = new \DateTimeImmutable((string)$dateTimeStr);
return new UTCDateTime($dateTime->getTimestamp() * 1000);
} catch (\Exception $e) {
\app\utils\LoggerHelper::logBusiness('convert_to_utcdatetime_failed', [
'input' => $dateTimeStr,
'error' => $e->getMessage(),
]);
return null;
}
}
}

View File

@@ -0,0 +1,660 @@
<?php
namespace app\service;
use app\repository\DataCollectionTaskRepository;
use app\utils\LoggerHelper;
use app\utils\RedisHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 数据采集任务管理服务
*
* 职责:
* - 创建、更新、删除采集任务
* - 管理任务状态(启动、暂停、停止)
* - 追踪任务进度和统计信息
*/
class DataCollectionTaskService
{
public function __construct(
protected DataCollectionTaskRepository $taskRepository
) {
}
/**
* 创建采集任务
*
* @param array<string, mixed> $taskData 任务数据
* @return array<string, mixed> 创建的任务信息
*/
public function createTask(array $taskData): array
{
// 生成任务ID
$taskId = UuidGenerator::uuid4()->toString();
// 根据Handler类型自动处理目标数据源配置
$targetType = $taskData['target_type'] ?? '';
$targetDataSourceId = $taskData['target_data_source_id'] ?? '';
$targetDatabase = $taskData['target_database'] ?? '';
$targetCollection = $taskData['target_collection'] ?? '';
if ($targetType === 'consumption_record') {
// 消费记录Handler自动使用标签数据库配置
$dataSourceService = new \app\service\DataSourceService(new \app\repository\DataSourceRepository());
$dataSources = $dataSourceService->getDataSourceList(['status' => 1]);
// 查找标签数据库数据源通过名称或ID匹配
$tagDataSource = null;
foreach ($dataSources['list'] ?? [] as $ds) {
$dsName = strtolower($ds['name'] ?? '');
$dsId = strtolower($ds['data_source_id'] ?? '');
if ($dsId === 'tag_mongodb' ||
$dsName === 'tag_mongodb' ||
stripos($dsName, '标签') !== false ||
stripos($dsName, 'tag') !== false) {
$tagDataSource = $ds;
break;
}
}
if ($tagDataSource) {
$targetDataSourceId = $tagDataSource['data_source_id'];
$targetDatabase = $tagDataSource['database'] ?? 'ckb';
$targetCollection = 'consumption_records'; // 消费记录Handler会自动按时间分表
} else {
// 如果找不到,使用默认值
$targetDataSourceId = 'tag_mongodb'; // 尝试使用配置key作为ID
$targetDatabase = 'ckb';
$targetCollection = 'consumption_records';
}
} elseif ($targetType === 'generic') {
// 通用Handler验证用户是否提供了配置
if (empty($targetDataSourceId) || empty($targetDatabase) || empty($targetCollection)) {
throw new \InvalidArgumentException('通用Handler必须配置目标数据源、目标数据库和目标集合');
}
}
// 构建任务文档
$task = [
'task_id' => $taskId,
'name' => $taskData['name'] ?? '未命名任务',
'description' => $taskData['description'] ?? '',
'data_source_id' => $taskData['data_source_id'] ?? '',
'database' => $taskData['database'] ?? '',
'collection' => $taskData['collection'] ?? null,
'collections' => $taskData['collections'] ?? null,
'target_type' => $targetType,
'target_data_source_id' => $targetDataSourceId,
'target_database' => $targetDatabase,
'target_collection' => $targetCollection,
'mode' => $taskData['mode'] ?? 'batch', // batch: 批量采集, realtime: 实时监听
'field_mappings' => $this->cleanFieldMappings($taskData['field_mappings'] ?? []),
'collection_field_mappings' => $taskData['collection_field_mappings'] ?? [],
'lookups' => $taskData['lookups'] ?? [],
'collection_lookups' => $taskData['collection_lookups'] ?? [],
'filter_conditions' => $taskData['filter_conditions'] ?? [],
'schedule' => $taskData['schedule'] ?? [
'enabled' => false,
'cron' => null,
],
'status' => 'pending', // pending: 待启动, running: 运行中, paused: 已暂停, stopped: 已停止, error: 错误
'progress' => [
'status' => 'idle', // idle, running, paused, completed, error
'processed_count' => 0,
'success_count' => 0,
'error_count' => 0,
'total_count' => 0,
'percentage' => 0,
'start_time' => null,
'end_time' => null,
'last_sync_time' => null,
],
'statistics' => [
'total_processed' => 0,
'total_success' => 0,
'total_error' => 0,
'last_run_time' => null,
],
'created_by' => $taskData['created_by'] ?? 'system',
];
// 保存到数据库使用原生MongoDB客户端明确指定集合名
// 注意MongoDB Laravel的Model在数据中包含collection字段时可能会误用该字段作为集合名
// 因此使用原生客户端明确指定集合名为data_collection_tasks
$dbConfig = config('database.connections.mongodb');
// 使用 MongoDBHelper 创建客户端统一DSN构建逻辑
$client = \app\utils\MongoDBHelper::createClient([
'host' => parse_url($dbConfig['dsn'], PHP_URL_HOST) ?? '192.168.1.106',
'port' => parse_url($dbConfig['dsn'], PHP_URL_PORT) ?? 27017,
'username' => $dbConfig['username'] ?? '',
'password' => $dbConfig['password'] ?? '',
'auth_source' => $dbConfig['options']['authSource'] ?? 'admin',
], array_filter($dbConfig['options'] ?? [], function ($value) {
return $value !== '' && $value !== null;
}));
$database = $client->selectDatabase($dbConfig['database']);
$collection = $database->selectCollection('data_collection_tasks');
// 添加时间戳
$task['created_at'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
$task['updated_at'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
// 插入文档
$result = $collection->insertOne($task);
// 验证插入成功
if ($result->getInsertedCount() !== 1) {
throw new \RuntimeException("任务创建失败:未能插入到数据库");
}
// 如果任务状态是 running立即设置 Redis 启动标志,让调度器启动采集进程
if ($task['status'] === 'running') {
try {
\app\utils\RedisHelper::set("data_collection_task:{$taskId}:start", '1', 3600); // 1小时过期
LoggerHelper::logBusiness('data_collection_task_start_flag_set', [
'task_id' => $taskId,
'task_name' => $task['name'],
]);
} catch (\Throwable $e) {
// Redis 设置失败不影响任务创建,只记录日志
LoggerHelper::logError($e, [
'component' => 'DataCollectionTaskService',
'action' => 'createTask',
'task_id' => $taskId,
'message' => '设置启动标志失败',
]);
}
}
LoggerHelper::logBusiness('data_collection_task_created', [
'task_id' => $taskId,
'task_name' => $task['name'],
]);
return $task;
}
/**
* 清理字段映射数据,移除无效的映射项
*
* @param array $fieldMappings 原始字段映射数组
* @return array 清理后的字段映射数组
*/
private function cleanFieldMappings(array $fieldMappings): array
{
$cleaned = [];
foreach ($fieldMappings as $mapping) {
// 如果缺少target_field跳过该项
if (empty($mapping['target_field'])) {
continue;
}
// 清理状态值映射中的源状态值(移除多余的引号)
if (isset($mapping['value_mapping']) && is_array($mapping['value_mapping'])) {
foreach ($mapping['value_mapping'] as &$vm) {
if (isset($vm['source_value'])) {
// 移除字符串两端的单引号或双引号
$vm['source_value'] = trim($vm['source_value'], "'\"");
}
}
unset($vm); // 解除引用
}
$cleaned[] = $mapping;
}
return $cleaned;
}
/**
* 更新任务
*
* @param string $taskId 任务ID
* @param array<string, mixed> $taskData 任务数据
* @return bool 是否更新成功
*/
public function updateTask(string $taskId, array $taskData): bool
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
// 如果任务正在运行,完全禁止编辑(与前端逻辑保持一致)
if ($task->status === 'running') {
throw new \RuntimeException("运行中的任务不允许编辑,请先停止任务: {$taskId}");
}
// timestamps会自动处理updated_at
$result = $this->taskRepository->where('task_id', $taskId)->update($taskData);
LoggerHelper::logBusiness('data_collection_task_updated', [
'task_id' => $taskId,
'updated_fields' => array_keys($taskData),
]);
return $result > 0;
}
/**
* 删除任务
*
* 如果任务正在运行或已暂停,会先停止任务再删除
*
* @param string $taskId 任务ID
* @return bool 是否删除成功
*/
public function deleteTask(string $taskId): bool
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
// 如果任务正在运行或已暂停,先停止
if (in_array($task->status, ['running', 'paused'])) {
$this->stopTask($taskId);
}
$result = $this->taskRepository->where('task_id', $taskId)->delete();
LoggerHelper::logBusiness('data_collection_task_deleted', [
'task_id' => $taskId,
'previous_status' => $task->status,
]);
return $result > 0;
}
/**
* 启动任务
*
* 允许从以下状态启动:
* - pending (待启动) -> running
* - paused (已暂停) -> running (恢复)
* - stopped (已停止) -> running (重新启动)
* - completed (已完成) -> running (重新启动)
* - error (错误) -> running (重新启动)
*
* @param string $taskId 任务ID
* @return bool 是否启动成功
*/
public function startTask(string $taskId): bool
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
// 只允许从特定状态启动
$allowedStatuses = ['pending', 'paused', 'stopped', 'completed', 'error'];
if (!in_array($task->status, $allowedStatuses)) {
if ($task->status === 'running') {
throw new \RuntimeException("任务已在运行中: {$taskId}");
}
throw new \RuntimeException("任务当前状态不允许启动: {$taskId} (当前状态: {$task->status})");
}
// 如果是从 paused, stopped, completed, error 状态启动(重新启动),需要重置进度
$progress = $task->progress ?? [];
if (in_array($task->status, ['paused', 'stopped', 'completed', 'error'])) {
// 重新启动时重置进度保留总数为0表示重新开始
$progress = [
'status' => 'running',
'processed_count' => 0,
'success_count' => 0,
'error_count' => 0,
'total_count' => 0, // 总数量会在采集开始时设置
'percentage' => 0,
'start_time' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
'end_time' => null,
'last_sync_time' => null,
];
} else {
// 从 pending 状态启动,初始化进度
$progress['status'] = 'running';
$progress['start_time'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
}
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'running',
'progress' => $progress,
]);
// 清除之前的暂停和停止标志(如果存在)
RedisHelper::del("data_collection_task:{$taskId}:pause");
RedisHelper::del("data_collection_task:{$taskId}:stop");
// 设置Redis标志通知调度器启动任务
RedisHelper::set("data_collection_task:{$taskId}:start", '1', 3600);
LoggerHelper::logBusiness('data_collection_task_started', [
'task_id' => $taskId,
'previous_status' => $task->status,
]);
return true;
}
/**
* 暂停任务
*
* @param string $taskId 任务ID
* @return bool 是否暂停成功
*/
public function pauseTask(string $taskId): bool
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
if ($task->status !== 'running') {
throw new \RuntimeException("任务未在运行中: {$taskId}");
}
// 更新任务状态
// 注意需要使用完整的数组来更新嵌套字段timestamps会自动处理updated_at
$progress = $task->progress ?? [];
$progress['status'] = 'paused';
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'paused',
'progress' => $progress,
]);
// 设置Redis标志通知调度器暂停任务
RedisHelper::set("data_collection_task:{$taskId}:pause", '1', 3600);
LoggerHelper::logBusiness('data_collection_task_paused', [
'task_id' => $taskId,
]);
return true;
}
/**
* 停止任务
*
* 只允许从以下状态停止:
* - running (运行中) -> stopped
* - paused (已暂停) -> stopped
*
* @param string $taskId 任务ID
* @return bool 是否停止成功
*/
public function stopTask(string $taskId): bool
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
// 只允许从 running 或 paused 状态停止
if (!in_array($task->status, ['running', 'paused'])) {
throw new \RuntimeException("任务当前状态不允许停止: {$taskId} (当前状态: {$task->status})");
}
// 停止任务时,保持当前进度,不重置(只更新状态)
$currentProgress = $task->progress ?? [];
$progress = [
'status' => 'idle', // idle, running, paused, completed, error
'processed_count' => $currentProgress['processed_count'] ?? 0,
'success_count' => $currentProgress['success_count'] ?? 0,
'error_count' => $currentProgress['error_count'] ?? 0,
'total_count' => $currentProgress['total_count'] ?? 0,
'percentage' => $currentProgress['percentage'] ?? 0, // 保持当前进度百分比
'start_time' => $currentProgress['start_time'] ?? null,
'end_time' => new \MongoDB\BSON\UTCDateTime(time() * 1000), // 记录停止时间
'last_sync_time' => $currentProgress['last_sync_time'] ?? null,
];
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'stopped',
'progress' => $progress,
]);
// 设置Redis标志通知调度器停止任务
RedisHelper::set("data_collection_task:{$taskId}:stop", '1', 3600);
// 如果任务之前是 paused也需要清除暂停标志
if ($task->status === 'paused') {
RedisHelper::del("data_collection_task:{$taskId}:pause");
}
LoggerHelper::logBusiness('data_collection_task_stopped', [
'task_id' => $taskId,
'previous_status' => $task->status,
'progress_reset' => true,
]);
return true;
}
/**
* 获取任务列表
*
* @param array<string, mixed> $filters 过滤条件
* @param int $page 页码
* @param int $pageSize 每页数量
* @return array<string, mixed> 任务列表
*/
public function getTaskList(array $filters = [], int $page = 1, int $pageSize = 20): array
{
$query = $this->taskRepository->query();
// 应用过滤条件(只处理非空值,如果筛选条件为空则返回所有任务)
if (!empty($filters['status']) && $filters['status'] !== '') {
$query->where('status', $filters['status']);
}
if (!empty($filters['data_source_id']) && $filters['data_source_id'] !== '') {
$query->where('data_source_id', $filters['data_source_id']);
}
if (!empty($filters['name']) && $filters['name'] !== '') {
// MongoDB 使用正则表达式进行模糊查询
$namePattern = preg_quote($filters['name'], '/');
$query->where('name', 'regex', "/{$namePattern}/i");
}
// 分页
$total = $query->count();
$taskModels = $query->orderBy('created_at', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get();
// 手动转换为数组,避免 cast 机制对数组字段的错误处理
$tasks = [];
foreach ($taskModels as $model) {
$task = $model->getAttributes();
// 使用统一的日期字段处理方法
$task = $this->normalizeDateFields($task);
$tasks[] = $task;
}
return [
'tasks' => $tasks,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => ceil($total / $pageSize),
];
}
/**
* 获取任务详情
*
* @param string $taskId 任务ID
* @return array<string, mixed>|null 任务详情
*/
public function getTask(string $taskId): ?array
{
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if (!$task) {
return null;
}
// 手动转换为数组,避免 cast 机制对数组字段的错误处理
$taskArray = $task->getAttributes();
// 使用统一的日期字段处理方法
$taskArray = $this->normalizeDateFields($taskArray);
return $taskArray;
}
/**
* 更新任务进度
*
* @param string $taskId 任务ID
* @param array<string, mixed> $progress 进度信息
* @return bool 是否更新成功
*/
public function updateProgress(string $taskId, array $progress): bool
{
$updateData = [
'progress' => $progress,
];
// 如果进度包含统计信息,也更新统计
// 注意这里的统计应该是累加的但进度字段processed_count等应该直接设置
if (isset($progress['success_count']) || isset($progress['error_count'])) {
// 使用where查询因为主键是task_id而不是_id
$task = $this->taskRepository->where('task_id', $taskId)->first();
if ($task) {
$statistics = $task->statistics ?? [];
// 统计信息使用增量更新(累加本次运行的数据)
// 但这里需要判断是增量还是绝对值,如果是绝对值则应该直接设置
// 由于进度更新传入的是绝对值,所以这里应该直接使用最新值而不是累加
if (isset($progress['processed_count'])) {
$statistics['total_processed'] = $progress['processed_count'];
}
if (isset($progress['success_count'])) {
$statistics['total_success'] = $progress['success_count'];
}
if (isset($progress['error_count'])) {
$statistics['total_error'] = $progress['error_count'];
}
$statistics['last_run_time'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
$updateData['statistics'] = $statistics;
}
}
// 使用 where()->update() 更新文档
// 注意MongoDB Laravel Eloquent 的 update() 返回匹配的文档数量通常是1或0
$result = $this->taskRepository->where('task_id', $taskId)->update($updateData);
// 添加日志以便调试
if ($result === false || $result === 0) {
\Workerman\Worker::safeEcho("[DataCollectionTaskService] ⚠️ 更新进度失败: task_id={$taskId}, result={$result}\n");
} else {
\Workerman\Worker::safeEcho("[DataCollectionTaskService] ✅ 更新进度成功: task_id={$taskId}, 匹配文档数={$result}\n");
}
return $result > 0;
}
/**
* 统一处理日期字段,转换为 ISO 8601 字符串格式
*
* @param array<string, mixed> $task 任务数据
* @return array<string, mixed> 处理后的任务数据
*/
private function normalizeDateFields(array $task): array
{
foreach (['created_at', 'updated_at'] as $dateField) {
if (isset($task[$dateField])) {
if ($task[$dateField] instanceof \MongoDB\BSON\UTCDateTime) {
$task[$dateField] = $task[$dateField]->toDateTime()->format('Y-m-d\TH:i:s.000\Z');
} elseif ($task[$dateField] instanceof \DateTime || $task[$dateField] instanceof \DateTimeInterface) {
$task[$dateField] = $task[$dateField]->format('Y-m-d\TH:i:s.000\Z');
} elseif (is_array($task[$dateField]) && isset($task[$dateField]['$date'])) {
// 处理 JSON 编码后的日期格式
$dateValue = $task[$dateField]['$date'];
if (is_numeric($dateValue)) {
// 如果是数字,假设是毫秒时间戳
$timestamp = $dateValue / 1000;
$task[$dateField] = date('Y-m-d\TH:i:s.000\Z', (int)$timestamp);
} elseif (is_array($dateValue) && isset($dateValue['$numberLong'])) {
// MongoDB 扩展 JSON 格式:{"$date": {"$numberLong": "1640000000000"}}
$timestamp = intval($dateValue['$numberLong']) / 1000;
$task[$dateField] = date('Y-m-d\TH:i:s.000\Z', (int)$timestamp);
} else {
// 其他格式,尝试解析或保持原样
$task[$dateField] = is_string($dateValue) ? $dateValue : json_encode($dateValue);
}
}
}
}
return $task;
}
/**
* 获取所有运行中的任务
*
* @return array<int, array<string, mixed>> 运行中的任务列表
*/
public function getRunningTasks(): array
{
// 使用原生 MongoDB 查询,避免 Model 的 cast 机制导致数组字段被错误处理
$dbConfig = config('database.connections.mongodb');
// 使用 MongoDBHelper 创建客户端统一DSN构建逻辑
$client = \app\utils\MongoDBHelper::createClient([
'host' => parse_url($dbConfig['dsn'], PHP_URL_HOST) ?? '192.168.1.106',
'port' => parse_url($dbConfig['dsn'], PHP_URL_PORT) ?? 27017,
'username' => $dbConfig['username'] ?? '',
'password' => $dbConfig['password'] ?? '',
'auth_source' => $dbConfig['options']['authSource'] ?? 'admin',
], array_filter($dbConfig['options'] ?? [], function ($value) {
return $value !== '' && $value !== null;
}));
$database = $client->selectDatabase($dbConfig['database']);
$collection = $database->selectCollection('data_collection_tasks');
// 查询所有运行中的任务
$cursor = $collection->find(['status' => 'running']);
$tasks = [];
foreach ($cursor as $document) {
// MongoDB BSONDocument 需要转换为数组
if ($document instanceof \MongoDB\Model\BSONDocument) {
$task = json_decode(json_encode($document), true);
} elseif (is_array($document)) {
$task = $document;
} else {
// 其他类型,尝试转换为数组
$task = (array)$document;
}
// 处理 MongoDB 的 _id 字段
if (isset($task['_id'])) {
if (is_object($task['_id'])) {
$task['_id'] = (string)$task['_id'];
}
}
// 使用统一的日期字段处理方法
$task = $this->normalizeDateFields($task);
$tasks[] = $task;
}
return $tasks;
}
}

View File

@@ -0,0 +1,309 @@
<?php
namespace app\service\DataSource\Adapter;
use app\service\DataSource\DataSourceAdapterInterface;
use app\utils\LoggerHelper;
use MongoDB\Client;
use MongoDB\Driver\Exception\Exception as MongoDBException;
/**
* MongoDB 数据源适配器
*
* 职责:
* - 封装 MongoDB 数据库连接和查询操作
* - 实现 DataSourceAdapterInterface 接口
*/
class MongoDBAdapter implements DataSourceAdapterInterface
{
private ?Client $client = null;
private ?\MongoDB\Database $database = null;
private string $type = 'mongodb';
private string $databaseName = '';
/**
* 建立数据库连接
*
* @param array<string, mixed> $config 数据源配置
* @return bool 是否连接成功
*/
public function connect(array $config): bool
{
try {
$host = $config['host'] ?? '127.0.0.1';
$port = (int)($config['port'] ?? 27017);
$this->databaseName = $config['database'] ?? '';
$username = $config['username'] ?? '';
$password = $config['password'] ?? '';
$authSource = $config['auth_source'] ?? $this->databaseName;
// 构建 DSN
$dsn = "mongodb://";
if (!empty($username) && !empty($password)) {
$dsn .= urlencode($username) . ':' . urlencode($password) . '@';
}
$dsn .= "{$host}:{$port}";
if (!empty($this->databaseName)) {
$dsn .= "/{$this->databaseName}";
}
if (!empty($authSource)) {
$dsn .= "?authSource=" . urlencode($authSource);
}
// MongoDB 连接选项
$options = [];
if (isset($config['options'])) {
$options = array_filter($config['options'], function ($value) {
return $value !== '' && $value !== null;
});
}
// 设置超时选项
if (!isset($options['connectTimeoutMS'])) {
$options['connectTimeoutMS'] = ($config['timeout'] ?? 10) * 1000;
}
if (!isset($options['socketTimeoutMS'])) {
$options['socketTimeoutMS'] = ($config['timeout'] ?? 10) * 1000;
}
$this->client = new Client($dsn, $options);
// 选择数据库
if (!empty($this->databaseName)) {
$this->database = $this->client->selectDatabase($this->databaseName);
}
// 测试连接
$this->client->getManager()->selectServer();
LoggerHelper::logBusiness('mongodb_adapter_connected', [
'host' => $host,
'port' => $port,
'database' => $this->databaseName,
]);
return true;
} catch (MongoDBException $e) {
LoggerHelper::logError($e, [
'component' => 'MongoDBAdapter',
'action' => 'connect',
'config' => array_merge($config, ['password' => '***']), // 隐藏密码
]);
return false;
}
}
/**
* 关闭数据库连接
*
* @return void
*/
public function disconnect(): void
{
if ($this->client !== null) {
$this->client = null;
$this->database = null;
LoggerHelper::logBusiness('mongodb_adapter_disconnected', []);
}
}
/**
* 测试连接是否有效
*
* @return bool 连接是否有效
*/
public function isConnected(): bool
{
if ($this->client === null) {
return false;
}
try {
// 执行 ping 命令测试连接
$adminDb = $this->client->selectDatabase('admin');
$adminDb->command(['ping' => 1]);
return true;
} catch (MongoDBException $e) {
LoggerHelper::logError($e, [
'component' => 'MongoDBAdapter',
'action' => 'isConnected',
]);
return false;
}
}
/**
* 执行查询(返回多条记录)
*
* 注意:对于 MongoDB$sql 参数表示集合名称,$params 是一个包含 'filter' 和 'options' 的数组
*
* @param string $sql 集合名称MongoDB 中相当于表名)
* @param array<string, mixed> $params 查询参数,格式:['filter' => [...], 'options' => [...]]
* @return array<array<string, mixed>> 查询结果数组
*/
public function query(string $sql, array $params = []): array
{
if ($this->database === null) {
throw new \RuntimeException('数据库连接未建立或未选择数据库');
}
try {
$collection = $sql; // $sql 参数在 MongoDB 中表示集合名
$filter = $params['filter'] ?? [];
$options = $params['options'] ?? [];
$cursor = $this->database->selectCollection($collection)->find($filter, $options);
$results = [];
foreach ($cursor as $document) {
$results[] = $this->convertMongoDocumentToArray($document);
}
LoggerHelper::logBusiness('mongodb_query_executed', [
'collection' => $collection,
'filter' => $filter,
'result_count' => count($results),
]);
return $results;
} catch (MongoDBException $e) {
LoggerHelper::logError($e, [
'component' => 'MongoDBAdapter',
'action' => 'query',
'collection' => $sql,
'params' => $params,
]);
throw $e;
}
}
/**
* 执行查询(返回单条记录)
*
* 注意:对于 MongoDB$sql 参数表示集合名称,$params 是一个包含 'filter' 和 'options' 的数组
*
* @param string $sql 集合名称
* @param array<string, mixed> $params 查询参数,格式:['filter' => [...], 'options' => [...]]
* @return array<string, mixed>|null 查询结果(单条记录)或 null
*/
public function queryOne(string $sql, array $params = []): ?array
{
if ($this->database === null) {
throw new \RuntimeException('数据库连接未建立或未选择数据库');
}
try {
$collection = $sql; // $sql 参数在 MongoDB 中表示集合名
$filter = $params['filter'] ?? [];
$options = $params['options'] ?? [];
$document = $this->database->selectCollection($collection)->findOne($filter, $options);
if ($document === null) {
return null;
}
LoggerHelper::logBusiness('mongodb_query_one_executed', [
'collection' => $collection,
'filter' => $filter,
'has_result' => true,
]);
return $this->convertMongoDocumentToArray($document);
} catch (MongoDBException $e) {
LoggerHelper::logError($e, [
'component' => 'MongoDBAdapter',
'action' => 'queryOne',
'collection' => $sql,
'params' => $params,
]);
throw $e;
}
}
/**
* 批量查询(分页查询,用于大数据量场景)
*
* 注意:对于 MongoDB$sql 参数表示集合名称,$params 是一个包含 'filter' 和 'options' 的数组
*
* @param string $sql 集合名称
* @param array<string, mixed> $params 查询参数,格式:['filter' => [...], 'options' => [...]]
* @param int $offset 偏移量
* @param int $limit 每页数量
* @return array<array<string, mixed>> 查询结果数组
*/
public function queryBatch(string $sql, array $params = [], int $offset = 0, int $limit = 1000): array
{
if ($this->database === null) {
throw new \RuntimeException('数据库连接未建立或未选择数据库');
}
try {
$collection = $sql; // $sql 参数在 MongoDB 中表示集合名
$filter = $params['filter'] ?? [];
$options = $params['options'] ?? [];
// 设置分页选项
$options['skip'] = $offset;
$options['limit'] = $limit;
$cursor = $this->database->selectCollection($collection)->find($filter, $options);
$results = [];
foreach ($cursor as $document) {
$results[] = $this->convertMongoDocumentToArray($document);
}
LoggerHelper::logBusiness('mongodb_query_batch_executed', [
'collection' => $collection,
'offset' => $offset,
'limit' => $limit,
'result_count' => count($results),
]);
return $results;
} catch (MongoDBException $e) {
LoggerHelper::logError($e, [
'component' => 'MongoDBAdapter',
'action' => 'queryBatch',
'collection' => $sql,
'params' => $params,
'offset' => $offset,
'limit' => $limit,
]);
throw $e;
}
}
/**
* 获取数据源类型
*
* @return string 数据源类型
*/
public function getType(): string
{
return $this->type;
}
/**
* 将 MongoDB 文档转换为数组
*
* @param mixed $document MongoDB 文档对象
* @return array<string, mixed> 数组格式的数据
*/
private function convertMongoDocumentToArray($document): array
{
if (is_array($document)) {
return $document;
}
// MongoDB\BSON\Document 或 MongoDB\Model\BSONDocument
if (method_exists($document, 'toArray')) {
return $document->toArray();
}
// 转换为数组
return json_decode(json_encode($document), true) ?? [];
}
}

View File

@@ -0,0 +1,234 @@
<?php
namespace app\service\DataSource\Adapter;
use app\service\DataSource\DataSourceAdapterInterface;
use app\utils\LoggerHelper;
use PDO;
use PDOException;
/**
* MySQL 数据源适配器
*
* 职责:
* - 封装 MySQL 数据库连接和查询操作
* - 实现 DataSourceAdapterInterface 接口
*/
class MySQLAdapter implements DataSourceAdapterInterface
{
private ?PDO $connection = null;
private string $type = 'mysql';
/**
* 建立数据库连接
*
* @param array<string, mixed> $config 数据源配置
* @return bool 是否连接成功
*/
public function connect(array $config): bool
{
try {
$host = $config['host'] ?? '127.0.0.1';
$port = $config['port'] ?? 3306;
$database = $config['database'] ?? '';
$username = $config['username'] ?? '';
$password = $config['password'] ?? '';
$charset = $config['charset'] ?? 'utf8mb4';
// 构建 DSN
$dsn = "mysql:host={$host};port={$port};dbname={$database};charset={$charset}";
// PDO 选项
$options = [
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
PDO::ATTR_EMULATE_PREPARES => false, // 禁用预处理语句模拟
PDO::ATTR_PERSISTENT => $config['persistent'] ?? false, // 是否持久连接
PDO::ATTR_TIMEOUT => $config['timeout'] ?? 10, // 连接超时
];
$this->connection = new PDO($dsn, $username, $password, $options);
LoggerHelper::logBusiness('mysql_adapter_connected', [
'host' => $host,
'port' => $port,
'database' => $database,
]);
return true;
} catch (PDOException $e) {
LoggerHelper::logError($e, [
'component' => 'MySQLAdapter',
'action' => 'connect',
'config' => array_merge($config, ['password' => '***']), // 隐藏密码
]);
return false;
}
}
/**
* 关闭数据库连接
*
* @return void
*/
public function disconnect(): void
{
if ($this->connection !== null) {
$this->connection = null;
LoggerHelper::logBusiness('mysql_adapter_disconnected', []);
}
}
/**
* 测试连接是否有效
*
* @return bool 连接是否有效
*/
public function isConnected(): bool
{
if ($this->connection === null) {
return false;
}
try {
// 执行简单查询测试连接
$this->connection->query('SELECT 1');
return true;
} catch (PDOException $e) {
LoggerHelper::logError($e, [
'component' => 'MySQLAdapter',
'action' => 'isConnected',
]);
return false;
}
}
/**
* 执行查询(返回多条记录)
*
* @param string $sql SQL 查询语句
* @param array<string, mixed> $params 查询参数(绑定参数)
* @return array<array<string, mixed>> 查询结果数组
*/
public function query(string $sql, array $params = []): array
{
if ($this->connection === null) {
throw new \RuntimeException('数据库连接未建立');
}
try {
$stmt = $this->connection->prepare($sql);
$stmt->execute($params);
$results = $stmt->fetchAll(PDO::FETCH_ASSOC);
LoggerHelper::logBusiness('mysql_query_executed', [
'sql' => $sql,
'params_count' => count($params),
'result_count' => count($results),
]);
return $results;
} catch (PDOException $e) {
LoggerHelper::logError($e, [
'component' => 'MySQLAdapter',
'action' => 'query',
'sql' => $sql,
'params' => $params,
]);
throw $e;
}
}
/**
* 执行查询(返回单条记录)
*
* @param string $sql SQL 查询语句
* @param array<string, mixed> $params 查询参数
* @return array<string, mixed>|null 查询结果(单条记录)或 null
*/
public function queryOne(string $sql, array $params = []): ?array
{
if ($this->connection === null) {
throw new \RuntimeException('数据库连接未建立');
}
try {
$stmt = $this->connection->prepare($sql);
$stmt->execute($params);
$result = $stmt->fetch(PDO::FETCH_ASSOC);
LoggerHelper::logBusiness('mysql_query_one_executed', [
'sql' => $sql,
'params_count' => count($params),
'has_result' => $result !== false,
]);
return $result !== false ? $result : null;
} catch (PDOException $e) {
LoggerHelper::logError($e, [
'component' => 'MySQLAdapter',
'action' => 'queryOne',
'sql' => $sql,
'params' => $params,
]);
throw $e;
}
}
/**
* 批量查询(分页查询,用于大数据量场景)
*
* @param string $sql SQL 查询语句(需要包含 LIMIT 和 OFFSET或由适配器自动添加
* @param array<string, mixed> $params 查询参数
* @param int $offset 偏移量
* @param int $limit 每页数量
* @return array<array<string, mixed>> 查询结果数组
*/
public function queryBatch(string $sql, array $params = [], int $offset = 0, int $limit = 1000): array
{
if ($this->connection === null) {
throw new \RuntimeException('数据库连接未建立');
}
try {
// 如果 SQL 中已包含 LIMIT则直接使用否则自动添加
if (stripos($sql, 'LIMIT') === false) {
$sql .= " LIMIT {$limit} OFFSET {$offset}";
}
$stmt = $this->connection->prepare($sql);
$stmt->execute($params);
$results = $stmt->fetchAll(PDO::FETCH_ASSOC);
LoggerHelper::logBusiness('mysql_query_batch_executed', [
'sql' => $sql,
'offset' => $offset,
'limit' => $limit,
'result_count' => count($results),
]);
return $results;
} catch (PDOException $e) {
LoggerHelper::logError($e, [
'component' => 'MySQLAdapter',
'action' => 'queryBatch',
'sql' => $sql,
'params' => $params,
'offset' => $offset,
'limit' => $limit,
]);
throw $e;
}
}
/**
* 获取数据源类型
*
* @return string 数据源类型
*/
public function getType(): string
{
return $this->type;
}
}

View File

@@ -0,0 +1,116 @@
<?php
namespace app\service\DataSource;
use app\service\DataSource\Adapter\MySQLAdapter;
use app\service\DataSource\Adapter\MongoDBAdapter;
use app\utils\LoggerHelper;
/**
* 数据源适配器工厂
*
* 职责:
* - 根据数据源类型创建对应的适配器实例
* - 管理适配器实例(单例模式,避免重复创建连接)
*/
class DataSourceAdapterFactory
{
/**
* 适配器实例缓存(单例模式)
*
* @var array<string, DataSourceAdapterInterface>
*/
private static array $instances = [];
/**
* 创建数据源适配器
*
* @param string $type 数据源类型mysql、postgresql、mongodb 等)
* @param array<string, mixed> $config 数据源配置
* @return DataSourceAdapterInterface 适配器实例
* @throws \InvalidArgumentException 不支持的数据源类型
*/
public static function create(string $type, array $config): DataSourceAdapterInterface
{
// 生成缓存键(基于类型和配置)
$cacheKey = self::generateCacheKey($type, $config);
// 如果已存在实例,直接返回
if (isset(self::$instances[$cacheKey])) {
$adapter = self::$instances[$cacheKey];
// 检查连接是否有效
if ($adapter->isConnected()) {
return $adapter;
}
// 连接已断开,重新创建
unset(self::$instances[$cacheKey]);
}
// 根据类型创建适配器
$adapter = match (strtolower($type)) {
'mysql' => new MySQLAdapter(),
'mongodb' => new MongoDBAdapter(),
// 'postgresql' => new PostgreSQLAdapter(),
default => throw new \InvalidArgumentException("不支持的数据源类型: {$type}"),
};
// 建立连接
if (!$adapter->connect($config)) {
throw new \RuntimeException("无法连接到数据源: {$type}");
}
// 缓存实例
self::$instances[$cacheKey] = $adapter;
LoggerHelper::logBusiness('data_source_adapter_created', [
'type' => $type,
'cache_key' => $cacheKey,
]);
return $adapter;
}
/**
* 生成缓存键
*
* @param string $type 数据源类型
* @param array<string, mixed> $config 数据源配置
* @return string 缓存键
*/
private static function generateCacheKey(string $type, array $config): string
{
// 基于类型、主机、端口、数据库名生成唯一键
$host = $config['host'] ?? 'unknown';
$port = $config['port'] ?? 'unknown';
$database = $config['database'] ?? 'unknown';
return md5("{$type}:{$host}:{$port}:{$database}");
}
/**
* 清除所有适配器实例(用于测试或重新连接)
*
* @return void
*/
public static function clearInstances(): void
{
foreach (self::$instances as $adapter) {
try {
$adapter->disconnect();
} catch (\Throwable $e) {
LoggerHelper::logError($e, ['component' => 'DataSourceAdapterFactory', 'action' => 'clearInstances']);
}
}
self::$instances = [];
}
/**
* 获取所有已创建的适配器实例
*
* @return array<string, DataSourceAdapterInterface>
*/
public static function getInstances(): array
{
return self::$instances;
}
}

View File

@@ -0,0 +1,73 @@
<?php
namespace app\service\DataSource;
/**
* 数据源适配器接口
*
* 职责:
* - 定义统一的数据源访问接口
* - 支持多种数据库类型MySQL、PostgreSQL、MongoDB 等)
* - 提供基础查询能力
*/
interface DataSourceAdapterInterface
{
/**
* 建立数据库连接
*
* @param array<string, mixed> $config 数据源配置
* @return bool 是否连接成功
*/
public function connect(array $config): bool;
/**
* 关闭数据库连接
*
* @return void
*/
public function disconnect(): void;
/**
* 测试连接是否有效
*
* @return bool 连接是否有效
*/
public function isConnected(): bool;
/**
* 执行查询(返回多条记录)
*
* @param string $sql SQL 查询语句(或 MongoDB 查询条件)
* @param array<string, mixed> $params 查询参数(绑定参数或 MongoDB 查询选项)
* @return array<array<string, mixed>> 查询结果数组
*/
public function query(string $sql, array $params = []): array;
/**
* 执行查询(返回单条记录)
*
* @param string $sql SQL 查询语句(或 MongoDB 查询条件)
* @param array<string, mixed> $params 查询参数
* @return array<string, mixed>|null 查询结果(单条记录)或 null
*/
public function queryOne(string $sql, array $params = []): ?array;
/**
* 批量查询(分页查询,用于大数据量场景)
*
* @param string $sql SQL 查询语句
* @param array<string, mixed> $params 查询参数
* @param int $offset 偏移量
* @param int $limit 每页数量
* @return array<array<string, mixed>> 查询结果数组
*/
public function queryBatch(string $sql, array $params = [], int $offset = 0, int $limit = 1000): array;
/**
* 获取数据源类型
*
* @return string 数据源类型mysql、postgresql、mongodb 等)
*/
public function getType(): string;
}

View File

@@ -0,0 +1,68 @@
<?php
namespace app\service\DataSource;
use app\service\DataSource\Strategy\DefaultConsumptionStrategy;
use app\utils\LoggerHelper;
/**
* 轮询策略工厂
*
* 职责:
* - 根据配置创建对应的轮询策略实例
* - 支持自定义策略类
*/
class PollingStrategyFactory
{
/**
* 创建轮询策略
*
* @param string|array<string, mixed> $strategyConfig 策略配置(字符串为策略类名,数组包含 class 和 config
* @return PollingStrategyInterface 策略实例
* @throws \InvalidArgumentException 无效的策略配置
*/
public static function create(string|array $strategyConfig): PollingStrategyInterface
{
// 如果配置是字符串,则作为策略类名
if (is_string($strategyConfig)) {
$className = $strategyConfig;
$strategyConfig = ['class' => $className];
}
// 获取策略类名
$className = $strategyConfig['class'] ?? null;
if (!$className) {
// 如果没有指定策略,使用默认策略
$className = DefaultConsumptionStrategy::class;
}
// 验证类是否存在
if (!class_exists($className)) {
throw new \InvalidArgumentException("策略类不存在: {$className}");
}
// 验证类是否实现了接口
if (!is_subclass_of($className, PollingStrategyInterface::class)) {
throw new \InvalidArgumentException("策略类必须实现 PollingStrategyInterface: {$className}");
}
// 创建策略实例
try {
$strategy = new $className();
LoggerHelper::logBusiness('polling_strategy_created', [
'class' => $className,
]);
return $strategy;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'PollingStrategyFactory',
'action' => 'create',
'class' => $className,
]);
throw new \RuntimeException("无法创建策略实例: {$className}", 0, $e);
}
}
}

View File

@@ -0,0 +1,54 @@
<?php
namespace app\service\DataSource;
/**
* 轮询策略接口
*
* 职责:
* - 定义自定义轮询业务逻辑的接口
* - 每个数据源可配置独立的轮询策略
* - 支持自定义查询、转换、验证逻辑
*/
interface PollingStrategyInterface
{
/**
* 执行轮询查询
*
* @param DataSourceAdapterInterface $adapter 数据源适配器
* @param array<string, mixed> $config 数据源配置
* @param array<string, mixed> $lastSyncInfo 上次同步信息(包含 last_sync_time、last_sync_id 等)
* @return array<array<string, mixed>> 查询结果数组(原始数据)
*/
public function poll(
DataSourceAdapterInterface $adapter,
array $config,
array $lastSyncInfo = []
): array;
/**
* 数据转换
*
* @param array<array<string, mixed>> $rawData 原始数据
* @param array<string, mixed> $config 数据源配置
* @return array<array<string, mixed>> 转换后的数据(标准格式)
*/
public function transform(array $rawData, array $config): array;
/**
* 数据验证
*
* @param array<string, mixed> $record 单条记录
* @param array<string, mixed> $config 数据源配置
* @return bool 是否通过验证
*/
public function validate(array $record, array $config): bool;
/**
* 获取策略名称
*
* @return string 策略名称
*/
public function getName(): string;
}

View File

@@ -0,0 +1,197 @@
<?php
namespace app\service\DataSource\Strategy;
use app\service\DataSource\DataSourceAdapterInterface;
use app\service\DataSource\PollingStrategyInterface;
use app\utils\LoggerHelper;
/**
* 默认消费记录轮询策略(示例)
*
* 职责:
* - 提供默认的轮询策略实现示例
* - 展示如何实现自定义业务逻辑
* - 可根据实际需求扩展或替换
*/
class DefaultConsumptionStrategy implements PollingStrategyInterface
{
/**
* 执行轮询查询
*
* @param DataSourceAdapterInterface $adapter 数据源适配器
* @param array<string, mixed> $config 数据源配置
* @param array<string, mixed> $lastSyncInfo 上次同步信息
* @return array<array<string, mixed>> 查询结果数组
*/
public function poll(
DataSourceAdapterInterface $adapter,
array $config,
array $lastSyncInfo = []
): array {
// 从配置中获取表名和查询条件
$tableName = $config['table'] ?? 'consumption_records';
$lastSyncTime = $lastSyncInfo['last_sync_time'] ?? null;
$lastSyncId = $lastSyncInfo['last_sync_id'] ?? null;
// 构建 SQL 查询(增量查询)
$sql = "SELECT * FROM `{$tableName}` WHERE 1=1";
$params = [];
// 如果有上次同步时间,只查询新增或更新的记录
if ($lastSyncTime !== null) {
$sql .= " AND (`created_at` > :last_sync_time OR `updated_at` > :last_sync_time)";
$params[':last_sync_time'] = $lastSyncTime;
}
// 如果有上次同步ID用于去重可选
if ($lastSyncId !== null) {
$sql .= " AND `id` > :last_sync_id";
$params[':last_sync_id'] = $lastSyncId;
}
// 按创建时间排序
$sql .= " ORDER BY `created_at` ASC, `id` ASC";
// 执行查询批量查询每次最多1000条
$limit = $config['batch_size'] ?? 1000;
$offset = 0;
$allResults = [];
do {
$batchSql = $sql . " LIMIT {$limit} OFFSET {$offset}";
$results = $adapter->queryBatch($batchSql, $params, $offset, $limit);
if (empty($results)) {
break;
}
$allResults = array_merge($allResults, $results);
$offset += $limit;
// 防止无限循环最多查询10万条
if (count($allResults) >= 100000) {
LoggerHelper::logBusiness('polling_batch_limit_reached', [
'table' => $tableName,
'count' => count($allResults),
]);
break;
}
} while (count($results) === $limit);
LoggerHelper::logBusiness('polling_query_completed', [
'table' => $tableName,
'result_count' => count($allResults),
'last_sync_time' => $lastSyncTime,
]);
return $allResults;
}
/**
* 数据转换
*
* @param array<array<string, mixed>> $rawData 原始数据
* @param array<string, mixed> $config 数据源配置
* @return array<array<string, mixed>> 转换后的数据
*/
public function transform(array $rawData, array $config): array
{
// 字段映射配置(从外部数据库字段映射到标准字段)
$fieldMapping = $config['field_mapping'] ?? [
// 默认映射(如果外部数据库字段名与标准字段名一致,则无需映射)
'id' => 'id',
'user_id' => 'user_id',
'amount' => 'amount',
'store_id' => 'store_id',
'product_id' => 'product_id',
'consume_time' => 'consume_time',
'created_at' => 'created_at',
];
$transformedData = [];
foreach ($rawData as $record) {
$transformed = [];
// 应用字段映射
foreach ($fieldMapping as $standardField => $sourceField) {
if (isset($record[$sourceField])) {
$transformed[$standardField] = $record[$sourceField];
}
}
// 确保必要字段存在
if (!empty($transformed)) {
$transformedData[] = $transformed;
}
}
LoggerHelper::logBusiness('polling_transform_completed', [
'input_count' => count($rawData),
'output_count' => count($transformedData),
]);
return $transformedData;
}
/**
* 数据验证
*
* @param array<string, mixed> $record 单条记录
* @param array<string, mixed> $config 数据源配置
* @return bool 是否通过验证
*/
public function validate(array $record, array $config): bool
{
// 必填字段验证
$requiredFields = $config['required_fields'] ?? ['user_id', 'amount', 'consume_time'];
foreach ($requiredFields as $field) {
if (!isset($record[$field]) || $record[$field] === null || $record[$field] === '') {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => "缺少必填字段: {$field}",
'record' => $record,
]);
return false;
}
}
// 金额验证(必须为正数)
if (isset($record['amount'])) {
$amount = (float)$record['amount'];
if ($amount <= 0) {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => '金额必须大于0',
'amount' => $amount,
]);
return false;
}
}
// 时间格式验证(可选)
if (isset($record['consume_time'])) {
$time = strtotime($record['consume_time']);
if ($time === false) {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => '时间格式无效',
'consume_time' => $record['consume_time'],
]);
return false;
}
}
return true;
}
/**
* 获取策略名称
*
* @return string 策略名称
*/
public function getName(): string
{
return 'default_consumption';
}
}

View File

@@ -0,0 +1,225 @@
<?php
namespace app\service\DataSource\Strategy;
use app\service\DataSource\DataSourceAdapterInterface;
use app\service\DataSource\PollingStrategyInterface;
use app\utils\LoggerHelper;
/**
* MongoDB 消费记录轮询策略
*
* 职责:
* - 提供 MongoDB 专用的轮询策略实现
* - 展示如何实现自定义业务逻辑
*/
class MongoDBConsumptionStrategy implements PollingStrategyInterface
{
/**
* 执行轮询查询
*
* @param DataSourceAdapterInterface $adapter 数据源适配器
* @param array<string, mixed> $config 数据源配置
* @param array<string, mixed> $lastSyncInfo 上次同步信息
* @return array<array<string, mixed>> 查询结果数组
*/
public function poll(
DataSourceAdapterInterface $adapter,
array $config,
array $lastSyncInfo = []
): array {
// 从配置中获取集合名和查询条件
$collectionName = $config['collection'] ?? 'consumption_records';
$lastSyncTime = $lastSyncInfo['last_sync_time'] ?? null;
$lastSyncId = $lastSyncInfo['last_sync_id'] ?? null;
// 构建 MongoDB 查询过滤器
$filter = [];
// 如果有上次同步时间,只查询新增或更新的记录
if ($lastSyncTime !== null) {
$lastSyncTimestamp = is_numeric($lastSyncTime) ? (int)$lastSyncTime : strtotime($lastSyncTime);
$lastSyncDate = new \MongoDB\BSON\UTCDateTime($lastSyncTimestamp * 1000);
$filter['$or'] = [
['created_at' => ['$gt' => $lastSyncDate]],
['updated_at' => ['$gt' => $lastSyncDate]],
];
}
// 如果有上次同步ID用于去重可选
if ($lastSyncId !== null) {
$filter['_id'] = ['$gt' => $lastSyncId];
}
// 查询选项
$options = [
'sort' => ['created_at' => 1, '_id' => 1], // 按创建时间和ID排序
];
// 执行查询批量查询每次最多1000条
$limit = $config['batch_size'] ?? 1000;
$offset = 0;
$allResults = [];
do {
// MongoDB 适配器的 queryBatch 方法签名queryBatch(string $sql, array $params = [], int $offset = 0, int $limit = 1000)
// 对于 MongoDB$sql 是集合名,$params 包含 'filter' 和 'options'
$results = $adapter->queryBatch($collectionName, [
'filter' => $filter,
'options' => $options,
], $offset, $limit);
if (empty($results)) {
break;
}
$allResults = array_merge($allResults, $results);
$offset += $limit;
// 防止无限循环最多查询10万条
if (count($allResults) >= 100000) {
LoggerHelper::logBusiness('polling_batch_limit_reached', [
'collection' => $collectionName,
'count' => count($allResults),
]);
break;
}
} while (count($results) === $limit);
LoggerHelper::logBusiness('polling_query_completed', [
'collection' => $collectionName,
'result_count' => count($allResults),
'last_sync_time' => $lastSyncTime,
]);
return $allResults;
}
/**
* 数据转换
*
* @param array<array<string, mixed>> $rawData 原始数据
* @param array<string, mixed> $config 数据源配置
* @return array<array<string, mixed>> 转换后的数据
*/
public function transform(array $rawData, array $config): array
{
// 字段映射配置(从外部数据库字段映射到标准字段)
$fieldMapping = $config['field_mapping'] ?? [
// 默认映射MongoDB 使用 _id需要转换为 id
'_id' => 'id',
'user_id' => 'user_id',
'amount' => 'amount',
'store_id' => 'store_id',
'product_id' => 'product_id',
'consume_time' => 'consume_time',
'created_at' => 'created_at',
];
$transformedData = [];
foreach ($rawData as $record) {
$transformed = [];
// 处理 MongoDB 的 _id 字段(转换为字符串)
if (isset($record['_id'])) {
if (is_object($record['_id']) && method_exists($record['_id'], '__toString')) {
$record['id'] = (string)$record['_id'];
} else {
$record['id'] = (string)$record['_id'];
}
}
// 处理 MongoDB 的日期字段UTCDateTime 转换为字符串)
foreach (['created_at', 'updated_at', 'consume_time'] as $dateField) {
if (isset($record[$dateField])) {
if (is_object($record[$dateField]) && method_exists($record[$dateField], 'toDateTime')) {
$record[$dateField] = $record[$dateField]->toDateTime()->format('Y-m-d H:i:s');
} elseif (is_object($record[$dateField]) && method_exists($record[$dateField], '__toString')) {
$record[$dateField] = (string)$record[$dateField];
}
}
}
// 应用字段映射
foreach ($fieldMapping as $standardField => $sourceField) {
if (isset($record[$sourceField])) {
$transformed[$standardField] = $record[$sourceField];
}
}
// 确保必要字段存在
if (!empty($transformed)) {
$transformedData[] = $transformed;
}
}
LoggerHelper::logBusiness('polling_transform_completed', [
'input_count' => count($rawData),
'output_count' => count($transformedData),
]);
return $transformedData;
}
/**
* 数据验证
*
* @param array<string, mixed> $record 单条记录
* @param array<string, mixed> $config 数据源配置
* @return bool 是否通过验证
*/
public function validate(array $record, array $config): bool
{
// 必填字段验证
$requiredFields = $config['required_fields'] ?? ['user_id', 'amount', 'consume_time'];
foreach ($requiredFields as $field) {
if (!isset($record[$field]) || $record[$field] === null || $record[$field] === '') {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => "缺少必填字段: {$field}",
'record' => $record,
]);
return false;
}
}
// 金额验证(必须为正数)
if (isset($record['amount'])) {
$amount = (float)$record['amount'];
if ($amount <= 0) {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => '金额必须大于0',
'amount' => $amount,
]);
return false;
}
}
// 时间格式验证(可选)
if (isset($record['consume_time'])) {
$time = strtotime($record['consume_time']);
if ($time === false) {
LoggerHelper::logBusiness('polling_validation_failed', [
'reason' => '时间格式无效',
'consume_time' => $record['consume_time'],
]);
return false;
}
}
return true;
}
/**
* 获取策略名称
*
* @return string 策略名称
*/
public function getName(): string
{
return 'mongodb_consumption';
}
}

View File

@@ -0,0 +1,498 @@
<?php
namespace app\service;
use app\repository\DataSourceRepository;
use app\service\DataSource\DataSourceAdapterFactory;
use app\utils\LoggerHelper;
use MongoDB\Client;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 数据源服务
*
* 职责:
* - 管理数据源的CRUD操作
* - 验证数据源连接
* - 提供数据源配置
*/
class DataSourceService
{
public function __construct(
protected DataSourceRepository $repository
) {
}
/**
* 创建数据源
*
* @param array<string, mixed> $data
* @return DataSourceRepository
* @throws \Exception
*/
public function createDataSource(array $data): DataSourceRepository
{
// 生成ID
if (empty($data['data_source_id'])) {
$data['data_source_id'] = UuidGenerator::uuid4()->toString();
}
// 验证必填字段
$requiredFields = ['name', 'type', 'host', 'port', 'database'];
foreach ($requiredFields as $field) {
if (empty($data[$field])) {
throw new \InvalidArgumentException("缺少必填字段: {$field}");
}
}
// 验证类型
$allowedTypes = ['mongodb', 'mysql', 'postgresql'];
if (!in_array(strtolower($data['type']), $allowedTypes)) {
throw new \InvalidArgumentException("不支持的数据源类型: {$data['type']}");
}
// 验证ID唯一性
$existing = $this->repository->newQuery()
->where('data_source_id', $data['data_source_id'])
->first();
if ($existing) {
throw new \InvalidArgumentException("数据源ID已存在: {$data['data_source_id']}");
}
// 验证名称唯一性
$existingByName = $this->repository->newQuery()
->where('name', $data['name'])
->first();
if ($existingByName) {
throw new \InvalidArgumentException("数据源名称已存在: {$data['name']}");
}
// 设置默认值
$data['status'] = $data['status'] ?? 1; // 1:启用, 0:禁用
$data['options'] = $data['options'] ?? [];
$data['is_tag_engine'] = $data['is_tag_engine'] ?? false; // 默认不是标签引擎数据库
// 创建数据源
$dataSource = new DataSourceRepository($data);
$dataSource->save();
// 如果设置为标签引擎数据库,自动将其他数据源设置为 false确保只有一个
if (!empty($data['is_tag_engine'])) {
// 将所有其他数据源的 is_tag_engine 设置为 false
$this->repository->newQuery()
->where('data_source_id', '!=', $dataSource->data_source_id)
->update(['is_tag_engine' => false]);
LoggerHelper::logBusiness('tag_engine_set', [
'data_source_id' => $dataSource->data_source_id,
'action' => 'create',
]);
}
LoggerHelper::logBusiness('data_source_created', [
'data_source_id' => $dataSource->data_source_id,
'name' => $dataSource->name,
'type' => $dataSource->type,
]);
return $dataSource;
}
/**
* 更新数据源
*
* @param string $dataSourceId
* @param array<string, mixed> $data
* @return bool
*/
public function updateDataSource(string $dataSourceId, array $data): bool
{
$dataSource = $this->repository->find($dataSourceId);
if (!$dataSource) {
throw new \InvalidArgumentException("数据源不存在: {$dataSourceId}");
}
// 如果更新名称,验证唯一性
if (isset($data['name']) && $data['name'] !== $dataSource->name) {
$existing = $this->repository->newQuery()
->where('name', $data['name'])
->where('data_source_id', '!=', $dataSourceId)
->first();
if ($existing) {
throw new \InvalidArgumentException("数据源名称已存在: {$data['name']}");
}
}
// 如果设置为标签引擎数据库,自动将其他数据源设置为 false确保只有一个
if (isset($data['is_tag_engine']) && !empty($data['is_tag_engine'])) {
// 将所有其他数据源的 is_tag_engine 设置为 false
$this->repository->newQuery()
->where('data_source_id', '!=', $dataSourceId)
->update(['is_tag_engine' => false]);
LoggerHelper::logBusiness('tag_engine_set', [
'data_source_id' => $dataSourceId,
'action' => 'update',
]);
}
// 更新数据
$dataSource->fill($data);
$result = $dataSource->save();
if ($result) {
LoggerHelper::logBusiness('data_source_updated', [
'data_source_id' => $dataSourceId,
]);
}
return $result;
}
/**
* 删除数据源
*
* @param string $dataSourceId
* @return bool
*/
public function deleteDataSource(string $dataSourceId): bool
{
$dataSource = $this->repository->find($dataSourceId);
if (!$dataSource) {
throw new \InvalidArgumentException("数据源不存在: {$dataSourceId}");
}
// TODO: 检查是否有任务在使用此数据源
// 可以查询 DataCollectionTask 中是否有引用此数据源
$result = $dataSource->delete();
if ($result) {
LoggerHelper::logBusiness('data_source_deleted', [
'data_source_id' => $dataSourceId,
]);
}
return $result;
}
/**
* 获取数据源列表
*
* @param array<string, mixed> $filters
* @return array{list: array, total: int}
*/
public function getDataSourceList(array $filters = []): array
{
try {
$query = $this->repository->newQuery();
// 筛选条件
if (isset($filters['type'])) {
$query->where('type', $filters['type']);
}
if (isset($filters['status'])) {
$query->where('status', $filters['status']);
}
if (isset($filters['name'])) {
$query->where('name', 'like', '%' . $filters['name'] . '%');
}
// 排序
$query->orderBy('created_at', 'desc');
// 分页
$page = (int)($filters['page'] ?? 1);
$pageSize = (int)($filters['page_size'] ?? 20);
$total = $query->count();
$list = $query->skip(($page - 1) * $pageSize)
->take($pageSize)
->get()
->map(function ($item) {
// 不返回密码
$data = $item->toArray();
unset($data['password']);
return $data;
})
->toArray();
return [
'list' => $list,
'total' => $total,
];
} catch (\MongoDB\Driver\Exception\Exception $e) {
// MongoDB 连接错误
LoggerHelper::logError($e, [
'component' => 'DataSourceService',
'action' => 'getDataSourceList',
]);
throw new \RuntimeException('无法连接到 MongoDB 数据库,请检查数据库服务是否正常运行', 500, $e);
} catch (\Exception $e) {
LoggerHelper::logError($e, [
'component' => 'DataSourceService',
'action' => 'getDataSourceList',
]);
throw $e;
}
}
/**
* 获取数据源详情(不包含密码)
*
* @param string $dataSourceId
* @return array<string, mixed>|null
*/
public function getDataSourceDetail(string $dataSourceId): ?array
{
$dataSource = $this->repository->find($dataSourceId);
if (!$dataSource) {
return null;
}
$data = $dataSource->toArray();
unset($data['password']);
return $data;
}
/**
* 获取数据源详情(包含密码,用于连接)
*
* @param string $dataSourceId
* @return array<string, mixed>|null
*/
public function getDataSourceConfig(string $dataSourceId): ?array
{
$dataSource = $this->repository->find($dataSourceId);
if (!$dataSource) {
return null;
}
if ($dataSource->status != 1) {
throw new \RuntimeException("数据源已禁用: {$dataSourceId}");
}
return $dataSource->toConfigArray();
}
/**
* 测试数据源连接
*
* @param array<string, mixed> $config
* @return bool
*/
public function testConnection(array $config): bool
{
try {
$type = strtolower($config['type'] ?? '');
// MongoDB特殊处理
if ($type === 'mongodb') {
// 使用 MongoDBHelper 创建客户端统一DSN构建逻辑
$client = \app\utils\MongoDBHelper::createClient($config, [
'connectTimeoutMS' => 3000,
'socketTimeoutMS' => 5000,
]);
// 尝试列出数据库来测试连接
$client->listDatabases();
return true;
}
// 其他类型使用适配器
$adapter = DataSourceAdapterFactory::create($type, $config);
$connected = $adapter->isConnected();
$adapter->disconnect();
return $connected;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSourceService',
'action' => 'testConnection',
]);
return false;
}
}
/**
* 获取所有启用的数据源用于替代config('data_sources'),从数据库读取)
*
* @return array<string, array> 以data_source_id为key的配置数组
*/
public function getAllEnabledDataSources(): array
{
$dataSources = $this->repository->newQuery()
->where('status', 1)
->get();
$result = [];
foreach ($dataSources as $ds) {
$result[$ds->data_source_id] = $ds->toConfigArray();
}
return $result;
}
/**
* 根据数据源ID获取配置从数据库读取
*
* 支持两种查询方式:
* 1. 通过 data_source_id (UUID) 查询
* 2. 通过 name 字段查询(兼容配置文件中的 key如 sync_mongodb, tag_mongodb
*
* @param string $dataSourceId 数据源ID或名称
* @return array<string, mixed>|null 数据源配置不存在或禁用时返回null
*/
public function getDataSourceConfigById(string $dataSourceId): ?array
{
// \Workerman\Worker::safeEcho("[DataSourceService] 查询数据源配置: data_source_id={$dataSourceId}\n");
// 先尝试通过 data_source_id 查询UUID 格式)
$dataSource = $this->repository->newQuery()
->where('data_source_id', $dataSourceId)
->where('status', 1)
->first();
if ($dataSource) {
// \Workerman\Worker::safeEcho("[DataSourceService] ✓ 通过 data_source_id 查询成功: name={$dataSource->name}\n");
return $dataSource->toConfigArray();
}
// 如果通过 data_source_id 查不到,尝试通过 name 字段查询(兼容配置文件中的 key
// \Workerman\Worker::safeEcho("[DataSourceService] 通过 data_source_id 未找到,尝试通过 name 查询\n");
// 处理配置文件中的常见 key 映射
// 注意:这些映射需要根据实际数据库中的 name 字段值来调整
$nameMapping = [
'sync_mongodb' => '本地大数据库', // 根据实际数据库中的名称调整
'tag_mongodb' => '主数据库', // 标签引擎数据库is_tag_engine=true
'kr_mongodb' => '卡若的主机', // 卡若数据库
];
$searchName = $nameMapping[$dataSourceId] ?? null;
if ($searchName) {
// \Workerman\Worker::safeEcho("[DataSourceService] 使用映射名称查询: {$dataSourceId} -> {$searchName}\n");
// 使用映射的名称查询
$dataSource = $this->repository->newQuery()
->where('name', $searchName)
->where('status', 1)
->first();
if ($dataSource) {
// \Workerman\Worker::safeEcho("[DataSourceService] ✓ 通过映射名称查询成功: name={$dataSource->name}, data_source_id={$dataSource->data_source_id}\n");
return $dataSource->toConfigArray();
}
}
// 如果还是查不到,尝试直接使用 dataSourceId 作为 name 查询
// \Workerman\Worker::safeEcho("[DataSourceService] 尝试直接使用 dataSourceId 作为 name 查询: {$dataSourceId}\n");
$dataSource = $this->repository->newQuery()
->where('name', $dataSourceId)
->where('status', 1)
->first();
if ($dataSource) {
// \Workerman\Worker::safeEcho("[DataSourceService] ✓ 通过 name 直接查询成功: name={$dataSource->name}\n");
return $dataSource->toConfigArray();
}
// 如果还是查不到,对于 tag_mongodb尝试查询 is_tag_engine=true 的数据源
if ($dataSourceId === 'tag_mongodb') {
// \Workerman\Worker::safeEcho("[DataSourceService] 对于 tag_mongodb尝试查询 is_tag_engine=true 的数据源\n");
$dataSource = $this->repository->newQuery()
->where('is_tag_engine', true)
->where('status', 1)
->first();
if ($dataSource) {
// \Workerman\Worker::safeEcho("[DataSourceService] ✓ 通过 is_tag_engine 查询成功: name={$dataSource->name}, data_source_id={$dataSource->data_source_id}\n");
return $dataSource->toConfigArray();
}
}
// \Workerman\Worker::safeEcho("[DataSourceService] ✗ 未找到数据源配置: data_source_id={$dataSourceId}\n");
return null;
}
/**
* 获取标签引擎数据库配置is_tag_engine = true的数据源
*
* @return array<string, mixed>|null 标签引擎数据库配置未找到时返回null
*/
public function getTagEngineDataSourceConfig(): ?array
{
$dataSource = $this->repository->newQuery()
->where('is_tag_engine', true)
->where('status', 1)
->first();
if (!$dataSource) {
return null;
}
return $dataSource->toConfigArray();
}
/**
* 获取标签引擎数据库的data_source_id
*
* @return string|null 标签引擎数据库的data_source_id未找到时返回null
*/
public function getTagEngineDataSourceId(): ?string
{
$dataSource = $this->repository->newQuery()
->where('is_tag_engine', true)
->where('status', 1)
->first();
return $dataSource ? $dataSource->data_source_id : null;
}
/**
* 验证标签引擎数据库配置是否存在
*
* @return bool 是否存在标签引擎数据库
*/
public function hasTagEngineDataSource(): bool
{
$count = $this->repository->newQuery()
->where('is_tag_engine', true)
->where('status', 1)
->count();
return $count > 0;
}
/**
* 获取所有标签引擎数据库(理论上应该只有一个,但允许有多个)
*
* @return array 标签引擎数据库列表
*/
public function getAllTagEngineDataSources(): array
{
$dataSources = $this->repository->newQuery()
->where('is_tag_engine', true)
->where('status', 1)
->get();
$result = [];
foreach ($dataSources as $ds) {
$data = $ds->toArray();
unset($data['password']); // 不返回密码
$result[] = $data;
}
return $result;
}
}

View File

@@ -0,0 +1,242 @@
<?php
namespace app\service;
use app\repository\ConsumptionRecordRepository;
use app\repository\UserProfileRepository;
use app\utils\QueueService;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid;
/**
* 数据同步服务
*
* 职责:
* - 消费消息队列中的数据同步消息
* - 批量写入 MongoDBconsumption_records
* - 更新用户统计user_profile
* - 触发标签计算(推送消息到标签计算队列)
*/
class DataSyncService
{
public function __construct(
protected ConsumptionRecordRepository $consumptionRecordRepository,
protected UserProfileRepository $userProfileRepository
) {
}
/**
* 同步数据到 MongoDB
*
* @param array<string, mixed> $messageData 消息数据(包含 source_id、data 等)
* @return array<string, mixed> 同步结果
*/
public function syncData(array $messageData): array
{
$sourceId = $messageData['source_id'] ?? 'unknown';
$data = $messageData['data'] ?? [];
$count = count($data);
if (empty($data)) {
LoggerHelper::logBusiness('data_sync_empty', [
'source_id' => $sourceId,
]);
return [
'success' => true,
'synced_count' => 0,
'skipped_count' => 0,
];
}
LoggerHelper::logBusiness('data_sync_service_started', [
'source_id' => $sourceId,
'data_count' => $count,
]);
$syncedCount = 0;
$skippedCount = 0;
$userIds = [];
// 批量写入消费记录
foreach ($data as $record) {
try {
// 数据验证
if (!$this->validateRecord($record)) {
$skippedCount++;
continue;
}
// 确保有 record_id
if (empty($record['record_id'])) {
$record['record_id'] = (string)Uuid::uuid4();
}
// 写入消费记录(使用 Eloquent Model 方式)
$consumptionRecord = new ConsumptionRecordRepository();
$consumptionRecord->record_id = $record['record_id'] ?? (string)Uuid::uuid4();
$consumptionRecord->user_id = $record['user_id'];
$consumptionRecord->consume_time = new \DateTimeImmutable($record['consume_time']);
$consumptionRecord->amount = (float)($record['amount'] ?? 0);
$consumptionRecord->actual_amount = (float)($record['actual_amount'] ?? $record['amount'] ?? 0);
$consumptionRecord->currency = $record['currency'] ?? 'CNY';
$consumptionRecord->store_id = $record['store_id'] ?? '';
$consumptionRecord->status = $record['status'] ?? 0;
$consumptionRecord->create_time = new \DateTimeImmutable('now');
$consumptionRecord->save();
$syncedCount++;
// 收集用户ID用于后续批量更新统计
$userId = $record['user_id'] ?? null;
if ($userId) {
$userIds[] = $userId;
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncService',
'action' => 'syncData',
'source_id' => $sourceId,
'record' => $record,
]);
$skippedCount++;
}
}
// 批量更新用户统计(去重)
$uniqueUserIds = array_unique($userIds);
foreach ($uniqueUserIds as $userId) {
try {
$this->updateUserStatistics($userId);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncService',
'action' => 'updateUserStatistics',
'user_id' => $userId,
]);
}
}
// 触发标签计算(为每个用户推送消息)
$tagCalculationCount = 0;
foreach ($uniqueUserIds as $userId) {
try {
$message = [
'user_id' => $userId,
'tag_ids' => null, // null 表示计算所有 real_time 标签
'trigger_type' => 'data_sync',
'source_id' => $sourceId,
'timestamp' => time(),
];
if (QueueService::pushTagCalculation($message)) {
$tagCalculationCount++;
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'DataSyncService',
'action' => 'triggerTagCalculation',
'user_id' => $userId,
]);
}
}
$result = [
'success' => true,
'synced_count' => $syncedCount,
'skipped_count' => $skippedCount,
'user_count' => count($uniqueUserIds),
'tag_calculation_triggered' => $tagCalculationCount,
];
LoggerHelper::logBusiness('data_sync_service_completed', array_merge([
'source_id' => $sourceId,
], $result));
return $result;
}
/**
* 验证记录
*
* @param array<string, mixed> $record 记录数据
* @return bool 是否通过验证
*/
private function validateRecord(array $record): bool
{
// 必填字段验证
$requiredFields = ['user_id', 'amount', 'consume_time'];
foreach ($requiredFields as $field) {
if (!isset($record[$field]) || $record[$field] === null || $record[$field] === '') {
return false;
}
}
// 金额验证
$amount = (float)($record['amount'] ?? 0);
if ($amount <= 0) {
return false;
}
// 时间格式验证
$consumeTime = $record['consume_time'] ?? '';
if (strtotime($consumeTime) === false) {
return false;
}
return true;
}
/**
* 更新用户统计
*
* @param string $userId 用户ID
* @return void
*/
private function updateUserStatistics(string $userId): void
{
// 获取用户的所有消费记录(用于重新计算统计)
// 这里简化处理,只更新最近的数据
// 实际场景中,可以增量更新或全量重新计算
// 查询用户最近的消费记录(使用 Eloquent 查询)
$records = ConsumptionRecordRepository::where('user_id', $userId)
->orderBy('consume_time', 'desc')
->limit(1000)
->get()
->toArray();
if (empty($records)) {
return;
}
// 计算统计值
$totalAmount = 0;
$totalCount = count($records);
$lastConsumeTime = null;
foreach ($records as $record) {
$amount = (float)($record['amount'] ?? 0);
$totalAmount += $amount;
$consumeTime = $record['consume_time'] ?? null;
if ($consumeTime) {
$time = strtotime($consumeTime);
if ($time && ($lastConsumeTime === null || $time > $lastConsumeTime)) {
$lastConsumeTime = $time;
}
}
}
// 更新用户档案(使用 increaseStats 方法,但这里需要全量更新)
// 简化处理:直接更新统计字段
$user = $this->userProfileRepository->findByUserId($userId);
if ($user) {
$user->total_amount = $totalAmount;
$user->total_count = $totalCount;
if ($lastConsumeTime) {
$user->last_consume_time = new \DateTimeImmutable('@' . $lastConsumeTime);
}
$user->save();
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,312 @@
<?php
namespace app\service;
use app\repository\UserProfileRepository;
use app\repository\UserPhoneRelationRepository;
use app\service\UserPhoneService;
use app\utils\EncryptionHelper;
use app\utils\IdCardHelper;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 身份解析服务
*
* 职责:
* - 根据手机号解析person_iduser_id
* - 如果找不到,创建临时人
* - 支持身份证绑定,将临时人转为正式人
* - 处理多手机号到同一人的映射
*/
class IdentifierService
{
public function __construct(
protected UserProfileRepository $userProfileRepository,
protected UserPhoneService $userPhoneService
) {
}
/**
* 根据手机号解析用户IDperson_id
*
* 流程:
* 1. 查询手机号关联表找到指定时间点有效的user_id
* 2. 如果找不到,创建临时人并建立关联
*
* @param string $phoneNumber 手机号
* @param \DateTimeInterface|null $atTime 查询时间点(默认为当前时间)
* @return string user_idperson_id
*/
public function resolvePersonIdByPhone(string $phoneNumber, ?\DateTimeInterface $atTime = null): string
{
// 检查手机号是否为空
$trimmedPhone = trim($phoneNumber);
if (empty($trimmedPhone)) {
// 如果手机号为空,创建一个没有手机号的临时用户
$userId = $this->createTemporaryPerson(null, $atTime);
LoggerHelper::logBusiness('temporary_person_created_no_phone', [
'user_id' => $userId,
'note' => '手机号为空,创建无手机号的临时用户',
]);
return $userId;
}
// 1. 先查询手机号关联表(使用指定的时间点)
$userId = $this->userPhoneService->findUserByPhone($trimmedPhone, $atTime);
if ($userId !== null) {
LoggerHelper::logBusiness('person_resolved_by_phone', [
'phone_number' => $trimmedPhone,
'user_id' => $userId,
'source' => 'existing_relation',
'at_time' => $atTime ? $atTime->format('Y-m-d H:i:s') : null,
]);
return $userId;
}
// 2. 如果找不到创建临时人使用atTime作为生效时间
$userId = $this->createTemporaryPerson($trimmedPhone, $atTime);
LoggerHelper::logBusiness('temporary_person_created', [
'phone_number' => $trimmedPhone,
'user_id' => $userId,
'effective_time' => $atTime ? $atTime->format('Y-m-d H:i:s') : null,
]);
return $userId;
}
/**
* 根据身份证解析用户IDperson_id
*
* @param string $idCard 身份证号
* @return string|null user_idperson_id如果不存在返回null
*/
public function resolvePersonIdByIdCard(string $idCard): ?string
{
$idCardHash = EncryptionHelper::hash($idCard);
$user = $this->userProfileRepository->findByIdCardHash($idCardHash);
if ($user) {
LoggerHelper::logBusiness('person_resolved_by_id_card', [
'id_card_hash' => $idCardHash,
'user_id' => $user->user_id,
]);
return $user->user_id;
}
return null;
}
/**
* 绑定身份证到用户(将临时人转为正式人,或创建正式人)
*
* @param string $userId 用户ID
* @param string $idCard 身份证号
* @return bool 是否成功
* @throws \InvalidArgumentException
*/
public function bindIdCardToPerson(string $userId, string $idCard): bool
{
$idCardHash = EncryptionHelper::hash($idCard);
$idCardEncrypted = EncryptionHelper::encrypt($idCard);
// 检查该身份证是否已被其他用户使用
$existingUser = $this->userProfileRepository->findByIdCardHash($idCardHash);
if ($existingUser && $existingUser->user_id !== $userId) {
throw new \InvalidArgumentException("身份证号已被其他用户使用user_id: {$existingUser->user_id}");
}
// 更新用户信息
$user = $this->userProfileRepository->findByUserId($userId);
if (!$user) {
throw new \InvalidArgumentException("用户不存在: {$userId}");
}
// 如果用户已经是正式人且身份证匹配,无需更新
if (!$user->is_temporary && $user->id_card_hash === $idCardHash) {
return true;
}
// 更新身份证信息并标记为正式人
$user->id_card_hash = $idCardHash;
$user->id_card_encrypted = $idCardEncrypted;
$user->id_card_type = '身份证';
$user->is_temporary = false;
// 从身份证号中自动提取基础信息(如果字段为空才更新)
$idCardInfo = IdCardHelper::extractInfo($idCard);
if ($idCardInfo['birthday'] !== null && $user->birthday === null) {
$user->birthday = $idCardInfo['birthday'];
}
// 只有当性别解析成功且当前值为 null 时才更新0 也被认为是未设置)
if ($idCardInfo['gender'] > 0 && ($user->gender === null || $user->gender === 0)) {
$user->gender = $idCardInfo['gender'];
}
$user->update_time = new \DateTimeImmutable('now');
$user->save();
LoggerHelper::logBusiness('id_card_bound_to_person', [
'user_id' => $userId,
'id_card_hash' => $idCardHash,
'was_temporary' => $user->is_temporary ?? true,
]);
return true;
}
/**
* 创建临时人
*
* @param string|null $phoneNumber 手机号(可选,用于建立关联)
* @param \DateTimeInterface|null $effectiveTime 生效时间(用于手机关联,默认当前时间)
* @return string user_id
*/
private function createTemporaryPerson(?string $phoneNumber = null, ?\DateTimeInterface $effectiveTime = null): string
{
$now = new \DateTimeImmutable('now');
$userId = UuidGenerator::uuid4()->toString();
// 创建临时人记录
$user = new UserProfileRepository();
$user->user_id = $userId;
$user->is_temporary = true;
$user->status = 0;
$user->total_amount = 0;
$user->total_count = 0;
$user->create_time = $now;
$user->update_time = $now;
$user->save();
// 如果有手机号建立关联使用effectiveTime作为生效时间
// 检查手机号不为空null 或空字符串都跳过)
if ($phoneNumber !== null && trim($phoneNumber) !== '') {
try {
$trimmedPhone = trim($phoneNumber);
$this->userPhoneService->addPhoneToUser($userId, $trimmedPhone, [
'source' => 'auto_created',
'type' => 'personal',
'effective_time' => $effectiveTime ?? $now,
]);
LoggerHelper::logBusiness('phone_relation_created_success', [
'user_id' => $userId,
'phone_number' => $trimmedPhone,
'effective_time' => ($effectiveTime ?? $now)->format('Y-m-d H:i:s'),
]);
} catch (\Throwable $e) {
// 手机号关联失败不影响用户创建,只记录详细的错误日志
LoggerHelper::logError($e, [
'component' => 'IdentifierService',
'action' => 'createTemporaryPerson',
'user_id' => $userId,
'phone_number' => $phoneNumber,
'phone_number_length' => strlen($phoneNumber),
'error_message' => $e->getMessage(),
'error_type' => get_class($e),
]);
// 同时记录业务日志,便于排查
LoggerHelper::logBusiness('phone_relation_create_failed', [
'user_id' => $userId,
'phone_number' => $phoneNumber,
'error_message' => $e->getMessage(),
'note' => '用户已创建,但手机关联失败',
]);
}
} elseif ($phoneNumber !== null && trim($phoneNumber) === '') {
// 手机号是空字符串,记录日志
LoggerHelper::logBusiness('phone_relation_skipped_empty', [
'user_id' => $userId,
'note' => '手机号为空字符串,跳过关联创建',
]);
}
return $userId;
}
/**
* 根据手机号或身份证解析用户ID
*
* 优先级:身份证 > 手机号
*
* @param string|null $phoneNumber 手机号
* @param string|null $idCard 身份证号
* @param \DateTimeInterface|null $atTime 查询时间点(用于手机号查询,默认为当前时间)
* @return string user_id
*/
public function resolvePersonId(?string $phoneNumber = null, ?string $idCard = null, ?\DateTimeInterface $atTime = null): string
{
$atTime = $atTime ?? new \DateTimeImmutable('now');
// 优先使用身份证
if ($idCard !== null && !empty($idCard)) {
$userId = $this->resolvePersonIdByIdCard($idCard);
if ($userId !== null) {
// 如果身份证存在,但提供了手机号,确保手机号关联到该用户
if ($phoneNumber !== null && !empty($phoneNumber)) {
// 在atTime时间点查询手机号关联
$existingUserId = $this->userPhoneService->findUserByPhone($phoneNumber, $atTime);
if ($existingUserId === null) {
// 手机号未关联建立关联使用atTime作为生效时间
$this->userPhoneService->addPhoneToUser($userId, $phoneNumber, [
'source' => 'id_card_resolved',
'type' => 'personal',
'effective_time' => $atTime,
]);
} elseif ($existingUserId !== $userId) {
// 手机号已关联到其他用户需要合并由PersonMergeService处理
LoggerHelper::logBusiness('phone_bound_to_different_person', [
'phone_number' => $phoneNumber,
'existing_user_id' => $existingUserId,
'id_card_user_id' => $userId,
'at_time' => $atTime->format('Y-m-d H:i:s'),
]);
}
}
return $userId;
} else {
// 身份证不存在,但有身份证信息,创建一个临时用户并绑定身份证(使其成为正式用户)
$userId = $this->createTemporaryPerson($phoneNumber, $atTime);
try {
$this->bindIdCardToPerson($userId, $idCard);
} catch (\Throwable $e) {
// 绑定失败不影响返回user_id
LoggerHelper::logError($e, [
'component' => 'IdentifierService',
'action' => 'resolvePersonId',
'user_id' => $userId,
]);
}
return $userId;
}
}
// 使用手机号传入atTime
if ($phoneNumber !== null && !empty($phoneNumber)) {
$userId = $this->resolvePersonIdByPhone($phoneNumber, $atTime);
// 如果同时提供了身份证,绑定身份证
if ($idCard !== null && !empty($idCard)) {
try {
$this->bindIdCardToPerson($userId, $idCard);
} catch (\Throwable $e) {
// 绑定失败不影响返回user_id
LoggerHelper::logError($e, [
'component' => 'IdentifierService',
'action' => 'resolvePersonId',
'user_id' => $userId,
]);
}
}
return $userId;
}
// 都没有提供,创建临时人
return $this->createTemporaryPerson(null, $atTime);
}
}

View File

@@ -0,0 +1,497 @@
<?php
namespace app\service;
use app\repository\UserProfileRepository;
use app\repository\UserTagRepository;
use app\repository\UserPhoneRelationRepository;
use app\repository\ConsumptionRecordRepository;
use app\service\TagService;
use app\service\UserPhoneService;
use app\utils\QueueService;
use app\utils\LoggerHelper;
/**
* 身份合并服务
*
* 职责:
* - 合并临时人到正式人
* - 合并多个用户到同一人(基于身份证)
* - 合并标签、统计数据、手机号关联等
*/
class PersonMergeService
{
public function __construct(
protected UserProfileRepository $userProfileRepository,
protected UserTagRepository $userTagRepository,
protected UserPhoneService $userPhoneService,
protected TagService $tagService
) {
}
/**
* 合并临时人到正式人
*
* 场景:手机号发现了对应的身份证号
*
* @param string $tempUserId 临时人user_id
* @param string $idCard 身份证号
* @return string 正式人的user_id
* @throws \InvalidArgumentException
*/
public function mergeTemporaryToFormal(string $tempUserId, string $idCard): string
{
$tempUser = $this->userProfileRepository->findByUserId($tempUserId);
if (!$tempUser) {
throw new \InvalidArgumentException("临时人不存在: {$tempUserId}");
}
if (!$tempUser->is_temporary) {
throw new \InvalidArgumentException("用户不是临时人: {$tempUserId}");
}
$idCardHash = \app\utils\EncryptionHelper::hash($idCard);
// 查找该身份证是否已有正式人
$formalUser = $this->userProfileRepository->findByIdCardHash($idCardHash);
if ($formalUser) {
// 情况1身份证已存在合并临时人到正式人
if ($formalUser->user_id === $tempUserId) {
// 已经是同一个人,只需标记为正式人(传入原始身份证号以提取信息)
$this->userProfileRepository->markAsFormal($tempUserId, $idCardHash, \app\utils\EncryptionHelper::encrypt($idCard), $idCard);
return $tempUserId;
}
// 合并到已存在的正式人
$this->mergeUsers($tempUserId, $formalUser->user_id);
return $formalUser->user_id;
} else {
// 情况2身份证不存在将临时人转为正式人传入原始身份证号以提取信息
$this->userProfileRepository->markAsFormal($tempUserId, $idCardHash, \app\utils\EncryptionHelper::encrypt($idCard), $idCard);
$tempUser->id_card_type = '身份证';
$tempUser->save();
LoggerHelper::logBusiness('temporary_person_converted_to_formal', [
'user_id' => $tempUserId,
'id_card_hash' => $idCardHash,
]);
// 重新计算标签
$this->recalculateTags($tempUserId);
return $tempUserId;
}
}
/**
* 合并两个用户将sourceUserId合并到targetUserId
*
* @param string $sourceUserId 源用户ID将被合并的用户
* @param string $targetUserId 目标用户ID保留的用户
* @return bool 是否成功
*/
public function mergeUsers(string $sourceUserId, string $targetUserId): bool
{
if ($sourceUserId === $targetUserId) {
return true;
}
$sourceUser = $this->userProfileRepository->findByUserId($sourceUserId);
$targetUser = $this->userProfileRepository->findByUserId($targetUserId);
if (!$sourceUser || !$targetUser) {
throw new \InvalidArgumentException("用户不存在: source={$sourceUserId}, target={$targetUserId}");
}
LoggerHelper::logBusiness('person_merge_started', [
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
]);
try {
// 1. 合并统计数据
$this->mergeStatistics($sourceUser, $targetUser);
// 2. 合并手机号关联
$this->mergePhoneRelations($sourceUserId, $targetUserId);
// 3. 合并标签
$this->mergeTags($sourceUserId, $targetUserId);
// 4. 合并消费记录更新user_id
$this->mergeConsumptionRecords($sourceUserId, $targetUserId);
// 5. 记录合并历史
$this->recordMergeHistory($sourceUserId, $targetUserId);
// 6. 标记源用户为已合并
$sourceUser->status = 1; // 标记为已删除/已合并
$sourceUser->merged_from_user_id = $targetUserId; // 记录合并到的目标用户ID
$sourceUser->update_time = new \DateTimeImmutable('now');
$sourceUser->save();
// 7. 更新目标用户的标签更新时间
$targetUser->tags_update_time = new \DateTimeImmutable('now');
$targetUser->update_time = new \DateTimeImmutable('now');
$targetUser->save();
LoggerHelper::logBusiness('person_merge_completed', [
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
]);
// 8. 重新计算目标用户的标签
$this->recalculateTags($targetUserId);
return true;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'PersonMergeService',
'action' => 'mergeUsers',
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
]);
throw $e;
}
}
/**
* 合并统计数据
*
* @param UserProfileRepository $sourceUser
* @param UserProfileRepository $targetUser
*/
private function mergeStatistics(UserProfileRepository $sourceUser, UserProfileRepository $targetUser): void
{
// 合并总金额和总次数
$targetUser->total_amount = (float)($targetUser->total_amount ?? 0) + (float)($sourceUser->total_amount ?? 0);
$targetUser->total_count = (int)($targetUser->total_count ?? 0) + (int)($sourceUser->total_count ?? 0);
// 取更晚的最后消费时间
if ($sourceUser->last_consume_time &&
(!$targetUser->last_consume_time || $sourceUser->last_consume_time > $targetUser->last_consume_time)) {
$targetUser->last_consume_time = $sourceUser->last_consume_time;
}
$targetUser->save();
}
/**
* 合并手机号关联
*
* @param string $sourceUserId
* @param string $targetUserId
*/
private function mergePhoneRelations(string $sourceUserId, string $targetUserId): void
{
// 获取源用户的所有手机号
$sourcePhones = $this->userPhoneService->getUserPhoneNumbers($sourceUserId, false);
foreach ($sourcePhones as $phoneNumber) {
try {
// 将手机号关联到目标用户
$this->userPhoneService->addPhoneToUser($targetUserId, $phoneNumber, [
'source' => 'person_merge',
'type' => 'personal',
]);
// 失效源用户的手机号关联
$this->userPhoneService->removePhoneFromUser($sourceUserId, $phoneNumber);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'PersonMergeService',
'action' => 'mergePhoneRelations',
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
'phone_number' => $phoneNumber,
]);
}
}
}
/**
* 合并标签
*
* 智能合并策略:
* 1. 如果目标用户没有该标签,直接复制源用户的标签
* 2. 如果目标用户已有该标签,根据标签类型和定义决定合并策略:
* - 数值型标签number根据标签定义的聚合方式sum/max/min/avg合并
* - 布尔型标签boolean取 OR任一为true则为true
* - 字符串型标签string保留目标用户的值不覆盖
* - 枚举型标签:保留目标用户的值(不覆盖)
* 3. 置信度取两者中的较高值
*
* @param string $sourceUserId
* @param string $targetUserId
*/
private function mergeTags(string $sourceUserId, string $targetUserId): void
{
$sourceTags = $this->userTagRepository->newQuery()
->where('user_id', $sourceUserId)
->get();
// 获取标签定义,用于判断合并策略
$tagDefinitionRepo = new \app\repository\TagDefinitionRepository();
foreach ($sourceTags as $sourceTag) {
// 检查目标用户是否已有该标签
$targetTag = $this->userTagRepository->newQuery()
->where('user_id', $targetUserId)
->where('tag_id', $sourceTag->tag_id)
->first();
if (!$targetTag) {
// 目标用户没有该标签,复制源用户的标签
$newTag = new UserTagRepository();
$newTag->user_id = $targetUserId;
$newTag->tag_id = $sourceTag->tag_id;
$newTag->tag_value = $sourceTag->tag_value;
$newTag->tag_value_type = $sourceTag->tag_value_type;
$newTag->confidence = $sourceTag->confidence;
$newTag->effective_time = $sourceTag->effective_time;
$newTag->expire_time = $sourceTag->expire_time;
$newTag->create_time = new \DateTimeImmutable('now');
$newTag->update_time = new \DateTimeImmutable('now');
$newTag->save();
} else {
// 目标用户已有标签,根据类型智能合并
$mergedValue = $this->mergeTagValue(
$sourceTag,
$targetTag,
$tagDefinitionRepo->newQuery()->where('tag_id', $sourceTag->tag_id)->first()
);
if ($mergedValue !== null) {
$targetTag->tag_value = $mergedValue;
$targetTag->confidence = max((float)$sourceTag->confidence, (float)$targetTag->confidence);
$targetTag->update_time = new \DateTimeImmutable('now');
$targetTag->save();
}
}
}
// 删除源用户的标签
$this->userTagRepository->newQuery()
->where('user_id', $sourceUserId)
->delete();
}
/**
* 合并标签值
*
* @param UserTagRepository $sourceTag 源标签
* @param UserTagRepository $targetTag 目标标签
* @param \app\repository\TagDefinitionRepository|null $tagDef 标签定义(可选)
* @return string|null 合并后的标签值如果不需要更新返回null
*/
private function mergeTagValue(
UserTagRepository $sourceTag,
UserTagRepository $targetTag,
?\app\repository\TagDefinitionRepository $tagDef = null
): ?string {
$sourceValue = $sourceTag->tag_value;
$targetValue = $targetTag->tag_value;
$sourceType = $sourceTag->tag_value_type;
$targetType = $targetTag->tag_value_type;
// 如果类型不一致,保留目标值
if ($sourceType !== $targetType) {
return null;
}
// 根据类型合并
switch ($targetType) {
case 'number':
// 数值型:根据标签定义的聚合方式合并
$aggregation = null;
if ($tagDef && isset($tagDef->rule_config)) {
$ruleConfig = is_string($tagDef->rule_config)
? json_decode($tagDef->rule_config, true)
: $tagDef->rule_config;
$aggregation = $ruleConfig['aggregation'] ?? 'sum';
} else {
$aggregation = 'sum'; // 默认累加
}
$sourceNum = (float)$sourceValue;
$targetNum = (float)$targetValue;
return match($aggregation) {
'sum' => (string)($sourceNum + $targetNum),
'max' => (string)max($sourceNum, $targetNum),
'min' => (string)min($sourceNum, $targetNum),
'avg' => (string)(($sourceNum + $targetNum) / 2),
default => (string)($sourceNum + $targetNum), // 默认累加
};
case 'boolean':
// 布尔型:取 OR任一为true则为true
$sourceBool = $sourceValue === 'true' || $sourceValue === '1';
$targetBool = $targetValue === 'true' || $targetValue === '1';
return ($sourceBool || $targetBool) ? 'true' : 'false';
case 'string':
case 'json':
default:
// 字符串型、JSON型等保留目标值不覆盖
return null;
}
}
/**
* 合并消费记录
*
* @param string $sourceUserId
* @param string $targetUserId
*/
private function mergeConsumptionRecords(string $sourceUserId, string $targetUserId): void
{
// 更新消费记录的user_id
ConsumptionRecordRepository::where('user_id', $sourceUserId)
->update(['user_id' => $targetUserId]);
}
/**
* 重新计算用户标签
*
* @param string $userId
*/
private function recalculateTags(string $userId): void
{
try {
// 异步触发标签计算
QueueService::pushTagCalculation([
'user_id' => $userId,
'tag_ids' => null, // 计算所有标签
'trigger_type' => 'person_merge',
'timestamp' => time(),
]);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'PersonMergeService',
'action' => 'recalculateTags',
'user_id' => $userId,
]);
}
}
/**
* 根据手机号发现身份证后,合并相关用户
*
* 这是场景4的实现如果某个手机号发现了对应的身份证号
* 查询该身份下是否有标签,如果有就会将对应的这个身份证号的所有标签重新计算同步。
*
* @param string $phoneNumber 手机号
* @param string $idCard 身份证号
* @return string 正式人的user_id
*/
public function mergePhoneToIdCard(string $phoneNumber, string $idCard): string
{
// 1. 查找手机号对应的用户
$phoneUserId = $this->userPhoneService->findUserByPhone($phoneNumber);
// 2. 查找身份证对应的用户
$idCardHash = \app\utils\EncryptionHelper::hash($idCard);
$idCardUser = $this->userProfileRepository->findByIdCardHash($idCardHash);
if ($idCardUser && $phoneUserId && $idCardUser->user_id === $phoneUserId) {
// 已经是同一个人,只需确保是正式人(传入原始身份证号以提取信息)
if ($idCardUser->is_temporary) {
$this->userProfileRepository->markAsFormal($phoneUserId, $idCardHash, \app\utils\EncryptionHelper::encrypt($idCard), $idCard);
}
$this->recalculateTags($phoneUserId);
return $phoneUserId;
}
if ($idCardUser && $phoneUserId && $idCardUser->user_id !== $phoneUserId) {
// 身份证和手机号对应不同用户,需要合并
$this->mergeUsers($phoneUserId, $idCardUser->user_id);
$this->recalculateTags($idCardUser->user_id);
return $idCardUser->user_id;
}
if ($idCardUser && !$phoneUserId) {
// 身份证存在,但手机号未关联,建立关联
$this->userPhoneService->addPhoneToUser($idCardUser->user_id, $phoneNumber, [
'source' => 'id_card_discovered',
'type' => 'personal',
]);
$this->recalculateTags($idCardUser->user_id);
return $idCardUser->user_id;
}
if (!$idCardUser && $phoneUserId) {
// 手机号存在,但身份证不存在,将临时人转为正式人
return $this->mergeTemporaryToFormal($phoneUserId, $idCard);
}
// 都不存在,创建正式人
$userId = \Ramsey\Uuid\Uuid::uuid4()->toString();
$now = new \DateTimeImmutable('now');
// 从身份证号中自动提取基础信息
$idCardInfo = \app\utils\IdCardHelper::extractInfo($idCard);
$user = new UserProfileRepository();
$user->user_id = $userId;
$user->id_card_hash = $idCardHash;
$user->id_card_encrypted = \app\utils\EncryptionHelper::encrypt($idCard);
$user->id_card_type = '身份证';
$user->is_temporary = false;
$user->status = 0;
$user->total_amount = 0;
$user->total_count = 0;
$user->birthday = $idCardInfo['birthday']; // 可能为 null
$user->gender = $idCardInfo['gender'] > 0 ? $idCardInfo['gender'] : null; // 解析失败则为 null
$user->create_time = $now;
$user->update_time = $now;
$user->save();
$this->userPhoneService->addPhoneToUser($userId, $phoneNumber, [
'source' => 'new_created',
'type' => 'personal',
]);
return $userId;
}
/**
* 记录合并历史
*
* @param string $sourceUserId 源用户ID
* @param string $targetUserId 目标用户ID
*/
private function recordMergeHistory(string $sourceUserId, string $targetUserId): void
{
try {
$sourceUser = $this->userProfileRepository->findByUserId($sourceUserId);
$targetUser = $this->userProfileRepository->findByUserId($targetUserId);
if (!$sourceUser || !$targetUser) {
return;
}
// 记录合并信息到日志(可以扩展为独立的合并历史表)
LoggerHelper::logBusiness('person_merge_history', [
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
'source_is_temporary' => $sourceUser->is_temporary ?? true,
'target_is_temporary' => $targetUser->is_temporary ?? false,
'source_id_card_hash' => $sourceUser->id_card_hash ?? null,
'target_id_card_hash' => $targetUser->id_card_hash ?? null,
'merge_time' => date('Y-m-d H:i:s'),
]);
} catch (\Throwable $e) {
// 记录历史失败不影响合并流程
LoggerHelper::logError($e, [
'component' => 'PersonMergeService',
'action' => 'recordMergeHistory',
'source_user_id' => $sourceUserId,
'target_user_id' => $targetUserId,
]);
}
}
}

View File

@@ -0,0 +1,164 @@
<?php
namespace app\service;
use app\repository\StoreRepository;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 门店服务
*
* 职责:
* - 创建门店
* - 查询门店信息
* - 根据门店名称获取或创建门店ID
*/
class StoreService
{
public function __construct(
protected StoreRepository $storeRepository
) {
}
/**
* 根据门店名称获取或创建门店ID
*
* @param string $storeName 门店名称
* @param string|null $source 数据源标识(用于生成默认门店编码)
* @param array<string, mixed> $extraData 额外的门店信息
* @return string 门店ID
*/
public function getOrCreateStoreByName(
string $storeName,
?string $source = null,
array $extraData = []
): string {
// 1. 先查找是否已存在同名门店(正常状态)
$existingStore = $this->storeRepository->findByStoreName($storeName);
if ($existingStore) {
LoggerHelper::logBusiness('store_found_by_name', [
'store_name' => $storeName,
'store_id' => $existingStore->store_id,
]);
return $existingStore->store_id;
}
// 2. 如果不存在,创建新门店
$storeId = $this->createStore($storeName, $source, $extraData);
LoggerHelper::logBusiness('store_created_by_name', [
'store_name' => $storeName,
'store_id' => $storeId,
'source' => $source,
]);
return $storeId;
}
/**
* 创建门店
*
* @param string $storeName 门店名称
* @param string|null $source 数据源标识
* @param array<string, mixed> $extraData 额外的门店信息
* @return string 门店ID
*/
public function createStore(string $storeName, ?string $source = null, array $extraData = []): string
{
$now = new \DateTimeImmutable('now');
$storeId = UuidGenerator::uuid4()->toString();
// 生成门店编码如果提供了store_code则使用否则自动生成
$storeCode = $extraData['store_code'] ?? $this->generateStoreCode($storeName, $source);
// 检查门店编码是否已存在
$existingStore = $this->storeRepository->findByStoreCode($storeCode);
if ($existingStore) {
// 如果编码已存在,重新生成
$storeCode = $this->generateStoreCode($storeName, $source, true);
}
// 创建门店记录
$store = new StoreRepository();
$store->store_id = $storeId;
$store->store_code = $storeCode;
$store->store_name = $storeName;
$store->store_type = $extraData['store_type'] ?? '线上店'; // 默认线上店
$store->store_level = $extraData['store_level'] ?? null;
$store->industry_id = $extraData['industry_id'] ?? 'default'; // 默认行业ID后续可配置
$store->industry_detail_id = $extraData['industry_detail_id'] ?? null;
$store->store_address = $extraData['store_address'] ?? null;
$store->store_province = $extraData['store_province'] ?? null;
$store->store_city = $extraData['store_city'] ?? null;
$store->store_district = $extraData['store_district'] ?? null;
$store->store_business_area = $extraData['store_business_area'] ?? null;
$store->store_longitude = isset($extraData['store_longitude']) ? (float)$extraData['store_longitude'] : null;
$store->store_latitude = isset($extraData['store_latitude']) ? (float)$extraData['store_latitude'] : null;
$store->store_phone = $extraData['store_phone'] ?? null;
$store->status = 0; // 0-正常
$store->create_time = $now;
$store->update_time = $now;
$store->save();
LoggerHelper::logBusiness('store_created', [
'store_id' => $storeId,
'store_name' => $storeName,
'store_code' => $storeCode,
'store_type' => $store->store_type,
]);
return $storeId;
}
/**
* 生成门店编码
*
* @param string $storeName 门店名称
* @param string|null $source 数据源标识
* @param bool $addTimestamp 是否添加时间戳(用于避免重复)
* @return string 门店编码
*/
private function generateStoreCode(string $storeName, ?string $source = null, bool $addTimestamp = false): string
{
// 清理门店名称,移除特殊字符,保留中英文和数字
$cleanedName = preg_replace('/[^\p{L}\p{N}]/u', '', $storeName);
// 如果名称过长截取前20个字符
if (mb_strlen($cleanedName) > 20) {
$cleanedName = mb_substr($cleanedName, 0, 20);
}
// 如果名称为空,使用默认值
if (empty($cleanedName)) {
$cleanedName = 'STORE';
}
// 生成编码:{source}_{cleaned_name}_{hash}
$hash = substr(md5($storeName . ($source ?? '')), 0, 8);
$code = strtoupper(($source ? $source . '_' : '') . $cleanedName . '_' . $hash);
// 如果需要添加时间戳(用于避免重复)
if ($addTimestamp) {
$code .= '_' . time();
}
return $code;
}
/**
* 根据门店ID获取门店信息
*
* @param string $storeId 门店ID
* @return StoreRepository|null
*/
public function getStoreById(string $storeId): ?StoreRepository
{
return $this->storeRepository->newQuery()
->where('store_id', $storeId)
->first();
}
}

View File

@@ -0,0 +1,226 @@
<?php
namespace app\service;
use app\repository\TagDefinitionRepository;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 标签初始化服务
*
* 用于预置初始标签定义
*/
class TagInitService
{
public function __construct(
protected TagDefinitionRepository $tagDefinitionRepository
) {
}
/**
* 初始化基础标签定义
*
* 创建几个简单的标签示例,用于测试标签计算引擎
*/
public function initBasicTags(): void
{
$now = new \DateTimeImmutable('now');
// 标签1高消费用户总消费金额 >= 5000
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'high_consumer',
'tag_name' => '高消费用户',
'category' => '消费能力',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'total_amount',
'operator' => '>=',
'value' => 5000,
],
],
'tag_value' => 'high',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 1,
'description' => '总消费金额大于等于5000的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
// 标签2活跃用户消费次数 >= 10
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'active_user',
'tag_name' => '活跃用户',
'category' => '活跃度',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'total_count',
'operator' => '>=',
'value' => 10,
],
],
'tag_value' => 'active',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 2,
'description' => '消费次数大于等于10次的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
// 标签3中消费用户总消费金额 >= 1000 且 < 5000
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'medium_consumer',
'tag_name' => '中消费用户',
'category' => '消费能力',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'total_amount',
'operator' => '>=',
'value' => 1000,
],
[
'field' => 'total_amount',
'operator' => '<',
'value' => 5000,
],
],
'tag_value' => 'medium',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 3,
'description' => '总消费金额在1000-5000之间的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
// 标签4低消费用户总消费金额 < 1000
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'low_consumer',
'tag_name' => '低消费用户',
'category' => '消费能力',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'total_amount',
'operator' => '<',
'value' => 1000,
],
],
'tag_value' => 'low',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 4,
'description' => '总消费金额小于1000的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
// 标签5新用户消费次数 < 3
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'new_user',
'tag_name' => '新用户',
'category' => '活跃度',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'total_count',
'operator' => '<',
'value' => 3,
],
],
'tag_value' => 'new',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 5,
'description' => '消费次数小于3次的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
// 标签6沉睡用户最后消费时间超过90天
$this->createTagIfNotExists([
'tag_id' => UuidGenerator::uuid4()->toString(),
'tag_code' => 'dormant_user',
'tag_name' => '沉睡用户',
'category' => '活跃度',
'rule_type' => 'simple',
'rule_config' => [
'rule_type' => 'simple',
'conditions' => [
[
'field' => 'last_consume_time',
'operator' => '<',
'value' => time() - 90 * 24 * 3600, // 90天前的时间戳
],
],
'tag_value' => 'dormant',
'confidence' => 1.0,
],
'update_frequency' => 'real_time',
'priority' => 6,
'description' => '最后消费时间超过90天的用户',
'status' => 0,
'version' => 1,
'create_time' => $now,
'update_time' => $now,
]);
}
/**
* 如果标签不存在则创建
*
* @param array<string, mixed> $tagData
*/
private function createTagIfNotExists(array $tagData): void
{
$existing = $this->tagDefinitionRepository->newQuery()
->where('tag_code', $tagData['tag_code'])
->first();
if (!$existing) {
$tag = new TagDefinitionRepository();
foreach ($tagData as $key => $value) {
$tag->$key = $value;
}
$tag->save();
echo "已创建标签: {$tagData['tag_name']} ({$tagData['tag_code']})\n";
} else {
echo "标签已存在: {$tagData['tag_name']} ({$tagData['tag_code']})\n";
}
}
}

View File

@@ -0,0 +1,96 @@
<?php
namespace app\service\TagRuleEngine;
/**
* 简单标签规则引擎
*
* 支持基础条件判断,用于计算简单标签(如:总消费金额等级、消费频次等级等)
*/
class SimpleRuleEngine
{
/**
* 计算标签值
*
* @param array<string, mixed> $ruleConfig 规则配置(从 tag_definitions.rule_config 解析)
* @param array<string, mixed> $userData 用户数据(从 user_profile 获取)
* @return array{value: mixed, confidence: float} 返回标签值和置信度
*/
public function calculate(array $ruleConfig, array $userData): array
{
if (!isset($ruleConfig['rule_type']) || $ruleConfig['rule_type'] !== 'simple') {
throw new \InvalidArgumentException('规则类型必须是 simple');
}
if (!isset($ruleConfig['conditions']) || !is_array($ruleConfig['conditions'])) {
throw new \InvalidArgumentException('规则配置中缺少 conditions');
}
// 执行所有条件判断
$allMatch = true;
foreach ($ruleConfig['conditions'] as $condition) {
if (!$this->evaluateCondition($condition, $userData)) {
$allMatch = false;
break;
}
}
// 如果所有条件都满足,返回标签值
if ($allMatch) {
// 简单标签:如果满足条件,标签值为 true 或指定的值
$tagValue = $ruleConfig['tag_value'] ?? true;
$confidence = $ruleConfig['confidence'] ?? 1.0;
return [
'value' => $tagValue,
'confidence' => (float)$confidence,
];
}
// 条件不满足,返回 false
return [
'value' => false,
'confidence' => 0.0,
];
}
/**
* 评估单个条件
*
* @param array<string, mixed> $condition 条件配置:{field, operator, value}
* @param array<string, mixed> $userData 用户数据
* @return bool
*/
private function evaluateCondition(array $condition, array $userData): bool
{
if (!isset($condition['field']) || !isset($condition['operator']) || !isset($condition['value'])) {
throw new \InvalidArgumentException('条件配置不完整:需要 field, operator, value');
}
$field = $condition['field'];
$operator = $condition['operator'];
$expectedValue = $condition['value'];
// 从用户数据中获取字段值
if (!isset($userData[$field])) {
// 字段不存在,根据运算符判断(例如 > 0 时,不存在视为 0
$actualValue = 0;
} else {
$actualValue = $userData[$field];
}
// 根据运算符进行比较
return match ($operator) {
'>' => $actualValue > $expectedValue,
'>=' => $actualValue >= $expectedValue,
'<' => $actualValue < $expectedValue,
'<=' => $actualValue <= $expectedValue,
'=' => $actualValue == $expectedValue,
'!=' => $actualValue != $expectedValue,
'in' => in_array($actualValue, (array)$expectedValue),
'not_in' => !in_array($actualValue, (array)$expectedValue),
default => throw new \InvalidArgumentException("不支持的运算符: {$operator}"),
};
}
}

View File

@@ -0,0 +1,587 @@
<?php
namespace app\service;
use app\repository\TagDefinitionRepository;
use app\repository\UserProfileRepository;
use app\repository\UserTagRepository;
use app\repository\TagHistoryRepository;
use app\service\TagRuleEngine\SimpleRuleEngine;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 标签服务
*
* 职责:
* - 根据用户数据计算标签值
* - 更新用户标签
* - 记录标签变更历史
*/
class TagService
{
public function __construct(
protected TagDefinitionRepository $tagDefinitionRepository,
protected UserProfileRepository $userProfileRepository,
protected UserTagRepository $userTagRepository,
protected TagHistoryRepository $tagHistoryRepository,
protected SimpleRuleEngine $ruleEngine
) {
}
/**
* 为指定用户计算并更新标签
*
* @param string $userId 用户ID
* @param array<string>|null $tagIds 要计算的标签ID列表null 表示计算所有启用且更新频率为 real_time 的标签)
* @return array<string, mixed> 返回更新的标签信息
*/
public function calculateTags(string $userId, ?array $tagIds = null): array
{
// 获取用户数据
$user = $this->userProfileRepository->findByUserId($userId);
if (!$user) {
throw new \InvalidArgumentException("用户不存在: {$userId}");
}
// 准备用户数据(用于规则引擎计算)
$userData = [
'total_amount' => (float)($user->total_amount ?? 0),
'total_count' => (int)($user->total_count ?? 0),
'last_consume_time' => $user->last_consume_time ? $user->last_consume_time->getTimestamp() : 0,
];
// 获取要计算的标签定义
$tagDefinitions = $this->getTagDefinitions($tagIds);
$updatedTags = [];
$now = new \DateTimeImmutable('now');
foreach ($tagDefinitions as $tagDef) {
try {
// 解析规则配置
$ruleConfig = is_string($tagDef->rule_config)
? json_decode($tagDef->rule_config, true)
: $tagDef->rule_config;
if (!$ruleConfig) {
continue;
}
// 根据规则类型选择计算引擎
if ($ruleConfig['rule_type'] === 'simple') {
$result = $this->ruleEngine->calculate($ruleConfig, $userData);
} else {
// 其他规则类型pipeline/custom暂不支持
continue;
}
// 获取旧标签值(用于历史记录)
$oldTag = $this->userTagRepository->newQuery()
->where('user_id', $userId)
->where('tag_id', $tagDef->tag_id)
->first();
$oldValue = $oldTag ? $oldTag->tag_value : null;
// 更新或创建标签
if ($oldTag) {
$oldTag->tag_value = $this->formatTagValue($result['value']);
$oldTag->tag_value_type = $this->getTagValueType($result['value']);
$oldTag->confidence = $result['confidence'];
$oldTag->update_time = $now;
$oldTag->save();
$userTag = $oldTag;
} else {
$userTag = new UserTagRepository();
$userTag->user_id = $userId;
$userTag->tag_id = $tagDef->tag_id;
$userTag->tag_value = $this->formatTagValue($result['value']);
$userTag->tag_value_type = $this->getTagValueType($result['value']);
$userTag->confidence = $result['confidence'];
$userTag->effective_time = $now;
$userTag->create_time = $now;
$userTag->update_time = $now;
$userTag->save();
}
// 记录标签变更历史(仅当值发生变化时)
if ($oldValue !== $userTag->tag_value) {
$this->recordTagHistory($userId, $tagDef->tag_id, $oldValue, $userTag->tag_value, $now);
}
$updatedTags[] = [
'tag_id' => $tagDef->tag_id,
'tag_code' => $tagDef->tag_code,
'tag_name' => $tagDef->tag_name,
'value' => $userTag->tag_value,
'confidence' => $userTag->confidence,
];
// 记录标签计算日志
LoggerHelper::logTagCalculation($userId, $tagDef->tag_id, [
'tag_code' => $tagDef->tag_code,
'value' => $userTag->tag_value,
'confidence' => $userTag->confidence,
]);
} catch (\Throwable $e) {
// 记录错误但继续处理其他标签
LoggerHelper::logError($e, [
'user_id' => $userId,
'tag_id' => $tagDef->tag_id ?? null,
'tag_code' => $tagDef->tag_code ?? null,
]);
}
}
// 更新用户的标签更新时间
$user->tags_update_time = $now;
$user->save();
return $updatedTags;
}
/**
* 获取标签定义列表
*
* @param array<string>|null $tagIds
* @return \Illuminate\Database\Eloquent\Collection
*/
private function getTagDefinitions(?array $tagIds = null)
{
$query = $this->tagDefinitionRepository->newQuery()
->where('status', 0); // 只获取启用的标签
if ($tagIds !== null) {
$query->whereIn('tag_id', $tagIds);
} else {
// 默认只计算实时更新的标签
$query->where('update_frequency', 'real_time');
}
return $query->get();
}
/**
* 格式化标签值
*
* @param mixed $value
* @return string
*/
private function formatTagValue($value): string
{
if (is_bool($value)) {
return $value ? 'true' : 'false';
}
if (is_array($value) || is_object($value)) {
return json_encode($value);
}
return (string)$value;
}
/**
* 获取标签值类型
*
* @param mixed $value
* @return string
*/
private function getTagValueType($value): string
{
if (is_bool($value)) {
return 'boolean';
}
if (is_int($value) || is_float($value)) {
return 'number';
}
if (is_array($value) || is_object($value)) {
return 'json';
}
return 'string';
}
/**
* 记录标签变更历史
*
* @param string $userId
* @param string $tagId
* @param mixed $oldValue
* @param string $newValue
* @param \DateTimeInterface $changeTime
*/
private function recordTagHistory(string $userId, string $tagId, $oldValue, string $newValue, \DateTimeInterface $changeTime): void
{
$history = new TagHistoryRepository();
$history->history_id = UuidGenerator::uuid4()->toString();
$history->user_id = $userId;
$history->tag_id = $tagId;
$history->old_value = $oldValue !== null ? (string)$oldValue : null;
$history->new_value = $newValue;
$history->change_reason = 'auto_calculate';
$history->change_time = $changeTime;
$history->operator = 'system';
$history->save();
}
/**
* 根据标签筛选用户
*
* @param array<string, mixed> $conditions 查询条件数组,每个条件包含:
* - tag_code: 标签编码(必填)
* - operator: 操作符(=, !=, >, >=, <, <=, in, not_in必填
* - value: 标签值(必填)
* @param string $logic 多个条件之间的逻辑关系AND 或 OR默认 AND
* @param int $page 页码从1开始
* @param int $pageSize 每页数量
* @param bool $includeUserInfo 是否包含用户基本信息
* @return array<string, mixed> 返回符合条件的用户列表
*/
public function filterUsersByTags(
array $conditions,
string $logic = 'AND',
int $page = 1,
int $pageSize = 20,
bool $includeUserInfo = false
): array {
if (empty($conditions)) {
return [
'users' => [],
'total' => 0,
'page' => $page,
'page_size' => $pageSize,
];
}
// 1. 根据 tag_code 获取 tag_id 列表
$tagCodes = array_column($conditions, 'tag_code');
$tagDefinitions = $this->tagDefinitionRepository->newQuery()
->whereIn('tag_code', $tagCodes)
->get()
->keyBy('tag_code');
$tagIdMap = [];
foreach ($tagDefinitions as $tagDef) {
$tagIdMap[$tagDef->tag_code] = $tagDef->tag_id;
}
// 验证所有 tag_code 都存在
$missingTags = array_diff($tagCodes, array_keys($tagIdMap));
if (!empty($missingTags)) {
throw new \InvalidArgumentException('标签编码不存在: ' . implode(', ', $missingTags));
}
// 2. 根据逻辑类型处理查询
if (strtoupper($logic) === 'OR') {
// OR 逻辑:使用 orWhere查询满足任一条件的用户
$query = $this->userTagRepository->newQuery();
$query->where(function ($q) use ($conditions, $tagIdMap) {
$first = true;
foreach ($conditions as $condition) {
$tagId = $tagIdMap[$condition['tag_code']];
$operator = $condition['operator'] ?? '=';
$value = $condition['value'];
$formattedValue = $this->formatTagValue($value);
if ($first) {
$this->applyTagCondition($q, $tagId, $operator, $formattedValue, $value);
$first = false;
} else {
$q->orWhere(function ($subQ) use ($tagId, $operator, $formattedValue, $value) {
$this->applyTagCondition($subQ, $tagId, $operator, $formattedValue, $value);
});
}
}
});
// 分页查询
$total = $query->count();
$userTags = $query->skip(($page - 1) * $pageSize)
->take($pageSize)
->get();
// 提取 user_id 列表
$userIds = $userTags->pluck('user_id')->unique()->toArray();
} else {
// AND 逻辑:所有条件都必须满足
// 由于每个标签是独立的记录,需要分别查询每个条件,然后取交集
$userIdsSets = [];
foreach ($conditions as $condition) {
$tagId = $tagIdMap[$condition['tag_code']];
$tagDef = $tagDefinitions->get($condition['tag_code']);
$operator = $condition['operator'] ?? '=';
$value = $condition['value'];
$formattedValue = $this->formatTagValue($value);
// 为每个条件单独查询满足条件的 user_id先从标签表查询
$subQuery = $this->userTagRepository->newQuery();
$this->applyTagCondition($subQuery, $tagId, $operator, $formattedValue, $value);
$tagUserIds = $subQuery->pluck('user_id')->unique()->toArray();
// 如果标签表中没有符合条件的记录,且标签定义中有规则,则基于规则从用户档案表筛选
// 这样可以处理用户还没有计算标签的情况
if ($tagDef && $tagDef->rule_type === 'simple') {
$ruleConfig = is_string($tagDef->rule_config)
? json_decode($tagDef->rule_config, true)
: $tagDef->rule_config;
if ($ruleConfig && isset($ruleConfig['tag_value']) && $ruleConfig['tag_value'] === $value) {
// 基于规则从用户档案表筛选
$profileQuery = $this->userProfileRepository->newQuery();
$this->applyRuleToProfileQuery($profileQuery, $ruleConfig);
$profileUserIds = $profileQuery->pluck('user_id')->unique()->toArray();
// 合并标签表和用户档案表的查询结果(去重)
$tagUserIds = array_unique(array_merge($tagUserIds, $profileUserIds));
}
}
$userIdsSets[] = $tagUserIds;
}
// 取交集所有条件都满足的用户ID
if (empty($userIdsSets)) {
$userIds = [];
} else {
$userIds = $userIdsSets[0];
for ($i = 1; $i < count($userIdsSets); $i++) {
$userIds = array_intersect($userIds, $userIdsSets[$i]);
}
}
// 如果没有满足所有条件的用户,直接返回空结果
if (empty($userIds)) {
return [
'users' => [],
'total' => 0,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => 0,
];
}
// 分页处理
$total = count($userIds);
$offset = ($page - 1) * $pageSize;
$userIds = array_slice($userIds, $offset, $pageSize);
}
// 5. 如果需要用户信息,则关联查询
$users = [];
if ($includeUserInfo && !empty($userIds)) {
$userProfiles = $this->userProfileRepository->newQuery()
->whereIn('user_id', $userIds)
->get()
->keyBy('user_id');
foreach ($userIds as $userId) {
$userProfile = $userProfiles->get($userId);
if ($userProfile) {
$users[] = [
'user_id' => $userId,
'name' => $userProfile->name ?? null,
'phone' => $userProfile->phone ?? null,
'total_amount' => $userProfile->total_amount ?? 0,
'total_count' => $userProfile->total_count ?? 0,
'last_consume_time' => $userProfile->last_consume_time ?? null,
];
} else {
$users[] = [
'user_id' => $userId,
];
}
}
} else {
foreach ($userIds as $userId) {
$users[] = ['user_id' => $userId];
}
}
return [
'users' => $users,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => (int)ceil($total / $pageSize),
];
}
/**
* 应用标签查询条件到查询构建器
*
* @param \Illuminate\Database\Eloquent\Builder $query 查询构建器
* @param string $tagId 标签ID
* @param string $operator 操作符
* @param string $formattedValue 格式化后的标签值
* @param mixed $originalValue 原始标签值(用于 in/not_in
*/
private function applyTagCondition($query, string $tagId, string $operator, string $formattedValue, $originalValue): void
{
$query->where('tag_id', $tagId);
switch ($operator) {
case '=':
case '==':
$query->where('tag_value', $formattedValue);
break;
case '!=':
case '<>':
$query->where('tag_value', '!=', $formattedValue);
break;
case '>':
$query->where('tag_value', '>', $formattedValue);
break;
case '>=':
$query->where('tag_value', '>=', $formattedValue);
break;
case '<':
$query->where('tag_value', '<', $formattedValue);
break;
case '<=':
$query->where('tag_value', '<=', $formattedValue);
break;
case 'in':
if (!is_array($originalValue)) {
throw new \InvalidArgumentException('in 操作符的值必须是数组');
}
$query->whereIn('tag_value', array_map([$this, 'formatTagValue'], $originalValue));
break;
case 'not_in':
if (!is_array($originalValue)) {
throw new \InvalidArgumentException('not_in 操作符的值必须是数组');
}
$query->whereNotIn('tag_value', array_map([$this, 'formatTagValue'], $originalValue));
break;
default:
throw new \InvalidArgumentException("不支持的操作符: {$operator}");
}
}
/**
* 将标签规则应用到用户档案查询
*
* @param \Illuminate\Database\Eloquent\Builder $query 查询构建器
* @param array<string, mixed> $ruleConfig 规则配置
*/
private function applyRuleToProfileQuery($query, array $ruleConfig): void
{
if (!isset($ruleConfig['conditions']) || !is_array($ruleConfig['conditions'])) {
return;
}
foreach ($ruleConfig['conditions'] as $condition) {
if (!isset($condition['field']) || !isset($condition['operator']) || !isset($condition['value'])) {
continue;
}
$field = $condition['field'];
$operator = $condition['operator'];
$value = $condition['value'];
// 将规则条件转换为用户档案表的查询条件
switch ($operator) {
case '>':
$query->where($field, '>', $value);
break;
case '>=':
$query->where($field, '>=', $value);
break;
case '<':
$query->where($field, '<', $value);
break;
case '<=':
$query->where($field, '<=', $value);
break;
case '=':
case '==':
$query->where($field, $value);
break;
case '!=':
case '<>':
$query->where($field, '!=', $value);
break;
case 'in':
if (is_array($value)) {
$query->whereIn($field, $value);
}
break;
case 'not_in':
if (is_array($value)) {
$query->whereNotIn($field, $value);
}
break;
}
}
// 只查询未删除的用户
$query->where('status', 0);
}
/**
* 获取指定用户的标签列表
*
* @param string $userId
* @return array<int, array<string, mixed>>
*/
public function getUserTags(string $userId): array
{
$userTags = $this->userTagRepository->newQuery()
->where('user_id', $userId)
->get();
$result = [];
foreach ($userTags as $userTag) {
$tagDef = $this->tagDefinitionRepository->newQuery()
->where('tag_id', $userTag->tag_id)
->first();
$result[] = [
'tag_id' => $userTag->tag_id,
'tag_code' => $tagDef ? $tagDef->tag_code : null,
'tag_name' => $tagDef ? $tagDef->tag_name : null,
'category' => $tagDef ? $tagDef->category : null,
'tag_value' => $userTag->tag_value,
'tag_value_type' => $userTag->tag_value_type,
'confidence' => $userTag->confidence,
'effective_time' => $userTag->effective_time,
'expire_time' => $userTag->expire_time,
'update_time' => $userTag->update_time,
];
}
return $result;
}
/**
* 删除用户的指定标签
*
* @param string $userId 用户ID
* @param string $tagId 标签ID
* @return bool 是否删除成功
*/
public function deleteUserTag(string $userId, string $tagId): bool
{
$userTag = $this->userTagRepository->newQuery()
->where('user_id', $userId)
->where('tag_id', $tagId)
->first();
if (!$userTag) {
return false;
}
$oldValue = $userTag->tag_value;
// 删除标签
$userTag->delete();
// 记录历史
$now = new \DateTimeImmutable('now');
$this->recordTagHistory($userId, $tagId, $oldValue, null, $now, 'tag_deleted');
LoggerHelper::logBusiness('tag_deleted', [
'user_id' => $userId,
'tag_id' => $tagId,
]);
return true;
}
}

View File

@@ -0,0 +1,331 @@
<?php
namespace app\service;
use app\repository\TagTaskRepository;
use app\repository\TagTaskExecutionRepository;
use app\repository\UserProfileRepository;
use app\repository\TagDefinitionRepository;
use app\service\TagService;
use app\utils\LoggerHelper;
use app\utils\RedisHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 标签任务执行器
*
* 职责:
* - 执行标签计算任务
* - 批量遍历用户数据打标签
* - 更新任务进度和统计信息
*/
class TagTaskExecutor
{
public function __construct(
protected TagTaskRepository $taskRepository,
protected TagTaskExecutionRepository $executionRepository,
protected UserProfileRepository $userProfileRepository,
protected TagDefinitionRepository $tagDefinitionRepository,
protected TagService $tagService
) {
}
/**
* 执行标签任务
*
* @param string $taskId 任务ID
* @return void
*/
public function execute(string $taskId): void
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
// 创建执行记录
$executionId = UuidGenerator::uuid4()->toString();
$execution = $this->executionRepository->create([
'execution_id' => $executionId,
'task_id' => $taskId,
'started_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
'status' => 'running',
'processed_users' => 0,
'success_count' => 0,
'error_count' => 0,
]);
try {
// 获取用户列表
$userIds = $this->getUserIds($task);
$totalUsers = count($userIds);
// 更新任务进度
$this->updateTaskProgress($taskId, [
'total_users' => $totalUsers,
'processed_users' => 0,
'success_count' => 0,
'error_count' => 0,
'percentage' => 0,
]);
// 获取目标标签ID列表
$targetTagIds = $task->target_tag_ids ?? null;
// 批量处理用户
$batchSize = $task->config['batch_size'] ?? 100;
$processedCount = 0;
$successCount = 0;
$errorCount = 0;
foreach (array_chunk($userIds, $batchSize) as $batch) {
// 检查任务状态(是否被暂停或停止)
if (!$this->checkTaskStatus($taskId)) {
LoggerHelper::logBusiness('tag_task_paused_or_stopped', [
'task_id' => $taskId,
'execution_id' => $executionId,
'processed' => $processedCount,
]);
break;
}
// 批量处理用户
foreach ($batch as $userId) {
try {
// 计算用户标签
$this->tagService->calculateTags($userId, $targetTagIds);
$successCount++;
} catch (\Exception $e) {
$errorCount++;
LoggerHelper::logError($e, [
'component' => 'TagTaskExecutor',
'action' => 'calculateTags',
'task_id' => $taskId,
'user_id' => $userId,
]);
// 根据错误处理策略决定是否继续
$errorHandling = $task->config['error_handling'] ?? 'skip';
if ($errorHandling === 'stop') {
throw $e;
}
}
$processedCount++;
// 每处理一定数量更新一次进度
if ($processedCount % 10 === 0) {
$this->updateTaskProgress($taskId, [
'processed_users' => $processedCount,
'success_count' => $successCount,
'error_count' => $errorCount,
'percentage' => $totalUsers > 0 ? round(($processedCount / $totalUsers) * 100, 2) : 0,
]);
// 更新执行记录
$this->executionRepository->where('execution_id', $executionId)->update([
'processed_users' => $processedCount,
'success_count' => $successCount,
'error_count' => $errorCount,
]);
}
}
}
// 更新最终进度
$this->updateTaskProgress($taskId, [
'processed_users' => $processedCount,
'success_count' => $successCount,
'error_count' => $errorCount,
'percentage' => $totalUsers > 0 ? round(($processedCount / $totalUsers) * 100, 2) : 100,
]);
// 更新执行记录为完成
$this->executionRepository->where('execution_id', $executionId)->update([
'status' => 'completed',
'finished_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
'processed_users' => $processedCount,
'success_count' => $successCount,
'error_count' => $errorCount,
]);
// 更新任务统计
$this->updateTaskStatistics($taskId, $successCount, $errorCount);
LoggerHelper::logBusiness('tag_task_execution_completed', [
'task_id' => $taskId,
'execution_id' => $executionId,
'total_users' => $totalUsers,
'processed' => $processedCount,
'success' => $successCount,
'error' => $errorCount,
]);
} catch (\Throwable $e) {
// 更新执行记录为失败
$this->executionRepository->where('execution_id', $executionId)->update([
'status' => 'failed',
'finished_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
'error_message' => $e->getMessage(),
]);
// 更新任务状态为错误
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'error',
'progress.status' => 'error',
'progress.last_error' => $e->getMessage(),
]);
LoggerHelper::logError($e, [
'component' => 'TagTaskExecutor',
'action' => 'execute',
'task_id' => $taskId,
'execution_id' => $executionId,
]);
throw $e;
}
}
/**
* 获取用户ID列表
*
* @param mixed $task 任务对象
* @return array<string> 用户ID列表
*/
private function getUserIds($task): array
{
$userScope = $task->user_scope ?? ['type' => 'all'];
$scopeType = $userScope['type'] ?? 'all';
switch ($scopeType) {
case 'all':
// 获取所有用户
$users = $this->userProfileRepository->newQuery()
->where('status', 0) // 只获取正常状态的用户
->get();
return $users->pluck('user_id')->toArray();
case 'list':
// 指定用户列表
return $userScope['user_ids'] ?? [];
case 'filter':
// 按条件筛选
// 这里可以扩展支持更复杂的筛选条件
$query = $this->userProfileRepository->newQuery()
->where('status', 0);
// 可以添加更多筛选条件
if (isset($userScope['conditions']) && is_array($userScope['conditions'])) {
foreach ($userScope['conditions'] as $condition) {
$field = $condition['field'] ?? '';
$operator = $condition['operator'] ?? '=';
$value = $condition['value'] ?? null;
if (empty($field)) {
continue;
}
switch ($operator) {
case '>':
$query->where($field, '>', $value);
break;
case '>=':
$query->where($field, '>=', $value);
break;
case '<':
$query->where($field, '<', $value);
break;
case '<=':
$query->where($field, '<=', $value);
break;
case '=':
$query->where($field, $value);
break;
case '!=':
$query->where($field, '!=', $value);
break;
case 'in':
if (is_array($value)) {
$query->whereIn($field, $value);
}
break;
}
}
}
$users = $query->get();
return $users->pluck('user_id')->toArray();
default:
throw new \InvalidArgumentException("不支持的用户范围类型: {$scopeType}");
}
}
/**
* 更新任务进度
*/
private function updateTaskProgress(string $taskId, array $progress): void
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
return;
}
$currentProgress = $task->progress ?? [];
$currentProgress = array_merge($currentProgress, $progress);
$currentProgress['status'] = 'running';
$this->taskRepository->where('task_id', $taskId)->update([
'progress' => $currentProgress,
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
]);
}
/**
* 更新任务统计
*/
private function updateTaskStatistics(string $taskId, int $successCount, int $errorCount): void
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
return;
}
$statistics = $task->statistics ?? [];
$statistics['total_executions'] = ($statistics['total_executions'] ?? 0) + 1;
$statistics['success_executions'] = ($statistics['success_executions'] ?? 0) + ($errorCount === 0 ? 1 : 0);
$statistics['failed_executions'] = ($statistics['failed_executions'] ?? 0) + ($errorCount > 0 ? 1 : 0);
$statistics['last_run_time'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
$this->taskRepository->where('task_id', $taskId)->update([
'statistics' => $statistics,
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
]);
}
/**
* 检查任务状态
*/
private function checkTaskStatus(string $taskId): bool
{
// 检查Redis标志
if (RedisHelper::exists("tag_task:{$taskId}:pause")) {
return false;
}
if (RedisHelper::exists("tag_task:{$taskId}:stop")) {
return false;
}
// 检查数据库状态
$task = $this->taskRepository->find($taskId);
if ($task && in_array($task->status, ['paused', 'stopped', 'error'])) {
return false;
}
return true;
}
}

View File

@@ -0,0 +1,283 @@
<?php
namespace app\service;
use app\repository\TagTaskRepository;
use app\repository\TagTaskExecutionRepository;
use app\repository\UserProfileRepository;
use app\service\TagService;
use app\utils\LoggerHelper;
use app\utils\RedisHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 标签任务管理服务
*
* 职责:
* - 创建、更新、删除标签任务
* - 管理任务状态(启动、暂停、停止)
* - 执行标签计算任务
* - 追踪任务进度和统计信息
*/
class TagTaskService
{
public function __construct(
protected TagTaskRepository $taskRepository,
protected TagTaskExecutionRepository $executionRepository,
protected UserProfileRepository $userProfileRepository,
protected TagService $tagService
) {
}
/**
* 创建标签任务
*
* @param array<string, mixed> $taskData 任务数据
* @return array<string, mixed> 创建的任务信息
*/
public function createTask(array $taskData): array
{
$taskId = UuidGenerator::uuid4()->toString();
$task = [
'task_id' => $taskId,
'name' => $taskData['name'] ?? '未命名标签任务',
'description' => $taskData['description'] ?? '',
'task_type' => $taskData['task_type'] ?? 'full',
'target_tag_ids' => $taskData['target_tag_ids'] ?? [],
'user_scope' => $taskData['user_scope'] ?? ['type' => 'all'],
'schedule' => $taskData['schedule'] ?? [
'enabled' => false,
'cron' => null,
],
'config' => $taskData['config'] ?? [
'concurrency' => 10,
'batch_size' => 100,
'error_handling' => 'skip',
],
'status' => 'pending',
'progress' => [
'total_users' => 0,
'processed_users' => 0,
'success_count' => 0,
'error_count' => 0,
'percentage' => 0,
],
'statistics' => [
'total_executions' => 0,
'success_executions' => 0,
'failed_executions' => 0,
'last_run_time' => null,
],
'created_by' => $taskData['created_by'] ?? 'system',
'created_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
];
$this->taskRepository->create($task);
LoggerHelper::logBusiness('tag_task_created', [
'task_id' => $taskId,
'task_name' => $task['name'],
]);
return $task;
}
/**
* 更新任务
*/
public function updateTask(string $taskId, array $taskData): bool
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
if ($task->status === 'running') {
$allowedFields = ['name', 'description', 'schedule'];
$taskData = array_intersect_key($taskData, array_flip($allowedFields));
}
$taskData['updated_at'] = new \MongoDB\BSON\UTCDateTime(time() * 1000);
return $this->taskRepository->where('task_id', $taskId)->update($taskData) > 0;
}
/**
* 删除任务
*/
public function deleteTask(string $taskId): bool
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
if ($task->status === 'running') {
$this->stopTask($taskId);
}
return $this->taskRepository->where('task_id', $taskId)->delete() > 0;
}
/**
* 启动任务
*/
public function startTask(string $taskId): bool
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
if ($task->status === 'running') {
throw new \RuntimeException("任务已在运行中: {$taskId}");
}
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'running',
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
]);
// 设置Redis标志通知调度器启动任务
RedisHelper::set("tag_task:{$taskId}:start", '1', 3600);
LoggerHelper::logBusiness('tag_task_started', [
'task_id' => $taskId,
]);
return true;
}
/**
* 暂停任务
*/
public function pauseTask(string $taskId): bool
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
if ($task->status !== 'running') {
throw new \RuntimeException("任务未在运行中: {$taskId}");
}
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'paused',
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
]);
RedisHelper::set("tag_task:{$taskId}:pause", '1', 3600);
return true;
}
/**
* 停止任务
*/
public function stopTask(string $taskId): bool
{
$task = $this->taskRepository->find($taskId);
if (!$task) {
throw new \InvalidArgumentException("任务不存在: {$taskId}");
}
$this->taskRepository->where('task_id', $taskId)->update([
'status' => 'stopped',
'updated_at' => new \MongoDB\BSON\UTCDateTime(time() * 1000),
]);
RedisHelper::set("tag_task:{$taskId}:stop", '1', 3600);
return true;
}
/**
* 获取任务列表
*/
public function getTaskList(array $filters = [], int $page = 1, int $pageSize = 20): array
{
$query = $this->taskRepository->query();
if (isset($filters['status'])) {
$query->where('status', $filters['status']);
}
if (isset($filters['task_type'])) {
$query->where('task_type', $filters['task_type']);
}
if (isset($filters['name'])) {
$query->where('name', 'like', '%' . $filters['name'] . '%');
}
$total = $query->count();
$tasks = $query->orderBy('created_at', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get()
->toArray();
return [
'tasks' => $tasks,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => ceil($total / $pageSize),
];
}
/**
* 获取任务详情
*/
public function getTask(string $taskId): ?array
{
$task = $this->taskRepository->find($taskId);
return $task ? $task->toArray() : null;
}
/**
* 获取任务执行记录
*/
public function getExecutions(string $taskId, int $page = 1, int $pageSize = 20): array
{
$query = $this->executionRepository->query()->where('task_id', $taskId);
$total = $query->count();
$executions = $query->orderBy('started_at', 'desc')
->skip(($page - 1) * $pageSize)
->take($pageSize)
->get()
->toArray();
return [
'executions' => $executions,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => ceil($total / $pageSize),
];
}
/**
* 执行任务(供调度器调用)
*/
public function executeTask(string $taskId): void
{
$executor = new \app\service\TagTaskExecutor(
$this->taskRepository,
$this->executionRepository,
$this->userProfileRepository,
new \app\repository\TagDefinitionRepository(),
$this->tagService
);
$executor->execute($taskId);
}
}

View File

@@ -0,0 +1,537 @@
<?php
namespace app\service;
use app\repository\UserPhoneRelationRepository;
use app\utils\EncryptionHelper;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 用户手机号服务
*
* 职责:
* - 管理用户与手机号的关联关系
* - 处理手机号的历史记录(支持手机号回收后重新分配)
* - 根据手机号查找当前用户
* - 获取用户的所有手机号
*/
class UserPhoneService
{
public function __construct(
protected UserPhoneRelationRepository $phoneRelationRepository
) {
}
/**
* 为用户添加手机号
*
* @param string $userId 用户ID
* @param string $phoneNumber 手机号
* @param array<string, mixed> $options 可选参数
* - type: 手机号类型personal/work/backup/other
* - is_verified: 是否已验证
* - effective_time: 生效时间(默认当前时间)
* - expire_time: 失效时间默认null表示当前有效
* - source: 来源registration/update/manual/import
* @return string 关联ID
* @throws \InvalidArgumentException
*/
public function addPhoneToUser(string $userId, string $phoneNumber, array $options = []): string
{
\Workerman\Worker::safeEcho("\n");
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n");
\Workerman\Worker::safeEcho("[UserPhoneService::addPhoneToUser] 【断点1-方法入口】开始执行\n");
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n");
\Workerman\Worker::safeEcho("【断点1】原始传入参数:\n");
\Workerman\Worker::safeEcho(" - userId: {$userId}\n");
\Workerman\Worker::safeEcho(" - phoneNumber: {$phoneNumber}\n");
\Workerman\Worker::safeEcho(" - options: " . json_encode($options, JSON_UNESCAPED_UNICODE) . "\n");
\Workerman\Worker::safeEcho("\n【断点2-参数处理】开始处理参数\n");
$phoneNumber = trim($phoneNumber);
\Workerman\Worker::safeEcho(" - trim后phoneNumber: {$phoneNumber}\n");
// 检查手机号是否为空
if (empty($phoneNumber)) {
\Workerman\Worker::safeEcho("【断点2】❌ 手机号为空,抛出异常\n");
throw new \InvalidArgumentException('手机号不能为空');
}
// 过滤非数字字符
$originalPhone = $phoneNumber;
\Workerman\Worker::safeEcho("\n【断点3-过滤处理】开始过滤非数字字符\n");
$phoneNumber = $this->filterPhoneNumber($phoneNumber);
\Workerman\Worker::safeEcho(" - 原始手机号: {$originalPhone}\n");
\Workerman\Worker::safeEcho(" - 过滤后手机号: {$phoneNumber}\n");
\Workerman\Worker::safeEcho(" - 过滤后长度: " . strlen($phoneNumber) . "\n");
// 检查过滤后是否为空
if (empty($phoneNumber)) {
\Workerman\Worker::safeEcho("【断点3】❌ 手机号过滤后为空,抛出异常\n");
throw new \InvalidArgumentException("手机号过滤后为空: {$originalPhone}");
}
\Workerman\Worker::safeEcho("\n【断点4-格式验证】开始验证手机号格式\n");
// 验证手机号格式(过滤后的手机号)
$isValid = $this->validatePhoneNumber($phoneNumber);
\Workerman\Worker::safeEcho(" - 验证结果: " . ($isValid ? '通过 ✓' : '失败 ✗') . "\n");
if (!$isValid) {
\Workerman\Worker::safeEcho("【断点4】❌ 手机号格式验证失败,抛出异常\n");
\Workerman\Worker::safeEcho(" - 验证规则: /^1[3-9]\\d{9}$/\n");
\Workerman\Worker::safeEcho(" - 实际值: {$phoneNumber}\n");
\Workerman\Worker::safeEcho(" - 长度: " . strlen($phoneNumber) . "\n");
throw new \InvalidArgumentException("手机号格式不正确: {$originalPhone} -> {$phoneNumber} (长度: " . strlen($phoneNumber) . ")");
}
\Workerman\Worker::safeEcho("【断点4】✓ 格式验证通过\n");
\Workerman\Worker::safeEcho("\n【断点5-哈希计算】开始计算手机号哈希\n");
$phoneHash = EncryptionHelper::hash($phoneNumber);
$now = new \DateTimeImmutable('now');
$effectiveTime = $options['effective_time'] ?? $now;
\Workerman\Worker::safeEcho(" - phoneHash: {$phoneHash}\n");
\Workerman\Worker::safeEcho(" - effectiveTime: " . $effectiveTime->format('Y-m-d H:i:s') . "\n");
\Workerman\Worker::safeEcho("\n【断点6-冲突检查】检查是否存在冲突关联\n");
// 检查该手机号在effectiveTime是否已有有效关联
// 使用effectiveTime作为查询时间点查找是否有冲突的关联
$existingActive = $this->phoneRelationRepository->findActiveByPhoneHash($phoneHash, $effectiveTime);
\Workerman\Worker::safeEcho(" - 查询结果: " . ($existingActive ? "找到冲突关联 (user_id: {$existingActive->user_id})" : "无冲突") . "\n");
if ($existingActive && $existingActive->user_id !== $userId) {
\Workerman\Worker::safeEcho("【断点6】⚠ 发现冲突,需要失效旧关联\n");
// 如果手机号在effectiveTime已被其他用户使用需要先失效旧关联
// 过期时间设置为新关联的effectiveTime保证时间连续避免间隙
$existingActive->expire_time = $effectiveTime;
$existingActive->is_active = false;
$existingActive->update_time = $now;
$existingActive->save();
\Workerman\Worker::safeEcho(" - 旧关联已失效expire_time: " . $effectiveTime->format('Y-m-d H:i:s') . "\n");
LoggerHelper::logBusiness('phone_relation_expired_due_to_conflict', [
'phone_number' => $phoneNumber,
'old_user_id' => $existingActive->user_id,
'new_user_id' => $userId,
'expire_time' => $effectiveTime->format('Y-m-d H:i:s'),
'effective_time' => $effectiveTime->format('Y-m-d H:i:s'),
]);
} else {
\Workerman\Worker::safeEcho("【断点6】✓ 无冲突,继续创建新关联\n");
}
\Workerman\Worker::safeEcho("\n【断点7-数据准备】开始准备要保存的数据\n");
try {
\Workerman\Worker::safeEcho(" [7.1] 创建 UserPhoneRelationRepository 对象...\n");
// 创建新关联
$relation = new UserPhoneRelationRepository();
\Workerman\Worker::safeEcho(" [7.1] ✓ 对象创建成功\n");
\Workerman\Worker::safeEcho(" [7.2] 设置 relation_id...\n");
$relation->relation_id = UuidGenerator::uuid4()->toString();
\Workerman\Worker::safeEcho(" [7.2] ✓ relation_id = {$relation->relation_id}\n");
\Workerman\Worker::safeEcho(" [7.3] 设置 phone_number...\n");
$relation->phone_number = $phoneNumber;
\Workerman\Worker::safeEcho(" [7.3] ✓ phone_number = {$relation->phone_number}\n");
\Workerman\Worker::safeEcho(" [7.4] 设置 phone_hash...\n");
$relation->phone_hash = $phoneHash;
\Workerman\Worker::safeEcho(" [7.4] ✓ phone_hash = {$relation->phone_hash}\n");
\Workerman\Worker::safeEcho(" [7.5] 设置 user_id...\n");
$relation->user_id = $userId;
\Workerman\Worker::safeEcho(" [7.5] ✓ user_id = {$relation->user_id}\n");
\Workerman\Worker::safeEcho(" [7.6] 设置 effective_time...\n");
\Workerman\Worker::safeEcho(" - effectiveTime类型: " . get_class($effectiveTime) . "\n");
\Workerman\Worker::safeEcho(" - effectiveTime值: " . $effectiveTime->format('Y-m-d H:i:s') . "\n");
$relation->effective_time = $effectiveTime;
\Workerman\Worker::safeEcho(" [7.6] ✓ effective_time 设置完成\n");
\Workerman\Worker::safeEcho(" [7.7] 设置 expire_time...\n");
$expireTimeValue = $options['expire_time'] ?? null;
\Workerman\Worker::safeEcho(" - expireTime值: " . ($expireTimeValue ? (is_object($expireTimeValue) ? $expireTimeValue->format('Y-m-d H:i:s') : $expireTimeValue) : 'null') . "\n");
$relation->expire_time = $expireTimeValue;
\Workerman\Worker::safeEcho(" [7.7] ✓ expire_time 设置完成\n");
\Workerman\Worker::safeEcho(" [7.8] 设置 is_active...\n");
// 如果 expire_time 为 null 或不存在,则 is_active 为 true
$isActiveValue = ($options['expire_time'] ?? null) === null;
\Workerman\Worker::safeEcho(" - isActive值: " . ($isActiveValue ? 'true' : 'false') . "\n");
$relation->is_active = $isActiveValue;
\Workerman\Worker::safeEcho(" [7.8] ✓ is_active 设置完成\n");
\Workerman\Worker::safeEcho(" [7.9] 设置 type...\n");
$typeValue = $options['type'] ?? 'personal';
\Workerman\Worker::safeEcho(" - type值: {$typeValue}\n");
$relation->type = $typeValue;
\Workerman\Worker::safeEcho(" [7.9] ✓ type 设置完成\n");
\Workerman\Worker::safeEcho(" [7.10] 设置 is_verified...\n");
$isVerifiedValue = $options['is_verified'] ?? false;
\Workerman\Worker::safeEcho(" - isVerified值: " . ($isVerifiedValue ? 'true' : 'false') . "\n");
$relation->is_verified = $isVerifiedValue;
\Workerman\Worker::safeEcho(" [7.10] ✓ is_verified 设置完成\n");
\Workerman\Worker::safeEcho(" [7.11] 设置 source...\n");
$sourceValue = $options['source'] ?? 'manual';
\Workerman\Worker::safeEcho(" - source值: {$sourceValue}\n");
$relation->source = $sourceValue;
\Workerman\Worker::safeEcho(" [7.11] ✓ source 设置完成\n");
\Workerman\Worker::safeEcho(" [7.12] 设置 create_time...\n");
\Workerman\Worker::safeEcho(" - now类型: " . get_class($now) . "\n");
\Workerman\Worker::safeEcho(" - now值: " . $now->format('Y-m-d H:i:s') . "\n");
$relation->create_time = $now;
\Workerman\Worker::safeEcho(" [7.12] ✓ create_time 设置完成\n");
\Workerman\Worker::safeEcho(" [7.13] 设置 update_time...\n");
$relation->update_time = $now;
\Workerman\Worker::safeEcho(" [7.13] ✓ update_time 设置完成\n");
\Workerman\Worker::safeEcho(" [7.14] ✓ 所有属性设置完成,准备打印数据详情\n");
} catch (\Throwable $e) {
\Workerman\Worker::safeEcho("\n【断点7】❌ 数据准备过程中发生异常!\n");
\Workerman\Worker::safeEcho(" - 错误信息: " . $e->getMessage() . "\n");
\Workerman\Worker::safeEcho(" - 错误类型: " . get_class($e) . "\n");
\Workerman\Worker::safeEcho(" - 文件: " . $e->getFile() . ":" . $e->getLine() . "\n");
\Workerman\Worker::safeEcho(" - 堆栈跟踪:\n");
$trace = $e->getTraceAsString();
$traceLines = explode("\n", $trace);
foreach (array_slice($traceLines, 0, 10) as $line) {
\Workerman\Worker::safeEcho(" " . $line . "\n");
}
throw $e;
}
\Workerman\Worker::safeEcho("【断点7】准备保存的数据详情:\n");
\Workerman\Worker::safeEcho(" - relation_id: {$relation->relation_id}\n");
\Workerman\Worker::safeEcho(" - phone_number: {$relation->phone_number}\n");
\Workerman\Worker::safeEcho(" - phone_hash: {$relation->phone_hash}\n");
\Workerman\Worker::safeEcho(" - user_id: {$relation->user_id}\n");
\Workerman\Worker::safeEcho(" - effective_time: " . ($relation->effective_time ? $relation->effective_time->format('Y-m-d H:i:s') : 'null') . "\n");
\Workerman\Worker::safeEcho(" - expire_time: " . ($relation->expire_time ? $relation->expire_time->format('Y-m-d H:i:s') : 'null') . "\n");
\Workerman\Worker::safeEcho(" - is_active: " . ($relation->is_active ? 'true' : 'false') . "\n");
\Workerman\Worker::safeEcho(" - type: {$relation->type}\n");
\Workerman\Worker::safeEcho(" - is_verified: " . ($relation->is_verified ? 'true' : 'false') . "\n");
\Workerman\Worker::safeEcho(" - source: {$relation->source}\n");
\Workerman\Worker::safeEcho(" - create_time: " . ($relation->create_time ? $relation->create_time->format('Y-m-d H:i:s') : 'null') . "\n");
\Workerman\Worker::safeEcho(" - update_time: " . ($relation->update_time ? $relation->update_time->format('Y-m-d H:i:s') : 'null') . "\n");
\Workerman\Worker::safeEcho("\n【断点8-数据库配置检查】检查数据库配置\n");
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n");
try {
// 获取表名
$tableName = $relation->getTable();
\Workerman\Worker::safeEcho(" ✓ 目标表名: {$tableName}\n");
// 获取连接名
$connectionName = $relation->getConnectionName();
\Workerman\Worker::safeEcho(" ✓ 数据库连接名: {$connectionName}\n");
// 获取连接对象
$connection = $relation->getConnection();
\Workerman\Worker::safeEcho(" ✓ 连接对象获取成功\n");
// 获取数据库名
$databaseName = $connection->getDatabaseName();
\Workerman\Worker::safeEcho(" ✓ 数据库名: {$databaseName}\n");
// 获取配置信息
$config = config('database.connections.' . $connectionName, []);
\Workerman\Worker::safeEcho("\n 数据库配置详情:\n");
\Workerman\Worker::safeEcho(" - driver: " . ($config['driver'] ?? 'unknown') . "\n");
\Workerman\Worker::safeEcho(" - dsn: " . ($config['dsn'] ?? 'unknown') . "\n");
\Workerman\Worker::safeEcho(" - database: " . ($config['database'] ?? 'unknown') . "\n");
\Workerman\Worker::safeEcho(" - username: " . (isset($config['username']) ? $config['username'] : 'null') . "\n");
\Workerman\Worker::safeEcho(" - has_password: " . (isset($config['password']) ? 'yes' : 'no') . "\n");
// 尝试获取MongoDB客户端信息
try {
$mongoClient = $connection->getMongoClient();
if ($mongoClient) {
\Workerman\Worker::safeEcho(" - MongoDB客户端: 已获取 ✓\n");
}
} catch (\Throwable $e) {
\Workerman\Worker::safeEcho(" - MongoDB客户端获取失败: " . $e->getMessage() . "\n");
}
// 测试连接
try {
$testCollection = $connection->getCollection($tableName);
\Workerman\Worker::safeEcho(" - 集合对象获取: 成功 ✓\n");
\Workerman\Worker::safeEcho(" - 集合名: {$tableName}\n");
} catch (\Throwable $e) {
\Workerman\Worker::safeEcho(" - 集合对象获取失败: " . $e->getMessage() . "\n");
}
\Workerman\Worker::safeEcho("\n 最终写入目标:\n");
\Workerman\Worker::safeEcho(" - 数据库: {$databaseName}\n");
\Workerman\Worker::safeEcho(" - 集合: {$tableName}\n");
\Workerman\Worker::safeEcho(" - 连接: {$connectionName}\n");
\Workerman\Worker::safeEcho(" - 连接状态: 已连接 ✓\n");
} catch (\Throwable $e) {
\Workerman\Worker::safeEcho(" ❌ 数据库配置检查失败!\n");
\Workerman\Worker::safeEcho(" - 错误信息: " . $e->getMessage() . "\n");
\Workerman\Worker::safeEcho(" - 错误类型: " . get_class($e) . "\n");
\Workerman\Worker::safeEcho(" - 文件: " . $e->getFile() . ":" . $e->getLine() . "\n");
\Workerman\Worker::safeEcho(" - 堆栈: " . $e->getTraceAsString() . "\n");
}
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n");
\Workerman\Worker::safeEcho("\n【断点9-执行保存】开始执行 save() 操作\n");
\Workerman\Worker::safeEcho(" - 调用: \$relation->save()\n");
// 执行保存
try {
$saveResult = $relation->save();
\Workerman\Worker::safeEcho("【断点9】save() 执行完成\n");
\Workerman\Worker::safeEcho(" - save() 返回值: " . ($saveResult ? 'true ✓' : 'false ✗') . "\n");
if (!$saveResult) {
\Workerman\Worker::safeEcho(" - ❌ 警告save() 返回 false数据可能未保存\n");
}
\Workerman\Worker::safeEcho("\n【断点10-保存后验证】验证数据是否真的写入数据库\n");
\Workerman\Worker::safeEcho(" - 查询条件: relation_id = {$relation->relation_id}\n");
// 验证是否真的保存成功(尝试查询)
$savedRelation = $this->phoneRelationRepository->findByRelationId($relation->relation_id);
if ($savedRelation) {
\Workerman\Worker::safeEcho(" - ✅ 验证成功:查询到保存的数据\n");
\Workerman\Worker::safeEcho(" - 查询到的 relation_id: {$savedRelation->relation_id}\n");
\Workerman\Worker::safeEcho(" - 查询到的 user_id: {$savedRelation->user_id}\n");
\Workerman\Worker::safeEcho(" - 查询到的 phone_number: {$savedRelation->phone_number}\n");
} else {
\Workerman\Worker::safeEcho(" - ❌ 验证失败save()返回true但查询不到数据\n");
\Workerman\Worker::safeEcho(" - 可能原因:\n");
\Workerman\Worker::safeEcho(" 1. MongoDB写入确认问题w=0模式\n");
\Workerman\Worker::safeEcho(" 2. 数据库连接问题\n");
\Workerman\Worker::safeEcho(" 3. 事务未提交\n");
\Workerman\Worker::safeEcho(" 4. 写入延迟\n");
}
} catch (\Throwable $e) {
\Workerman\Worker::safeEcho("\n【断点9】❌ 保存过程中发生异常!\n");
\Workerman\Worker::safeEcho(" - 错误信息: " . $e->getMessage() . "\n");
\Workerman\Worker::safeEcho(" - 错误类型: " . get_class($e) . "\n");
\Workerman\Worker::safeEcho(" - 文件: " . $e->getFile() . ":" . $e->getLine() . "\n");
\Workerman\Worker::safeEcho(" - 堆栈跟踪:\n");
$trace = $e->getTraceAsString();
$traceLines = explode("\n", $trace);
foreach (array_slice($traceLines, 0, 5) as $line) {
\Workerman\Worker::safeEcho(" " . $line . "\n");
}
throw $e;
}
\Workerman\Worker::safeEcho("\n【断点11-日志记录】记录业务日志\n");
LoggerHelper::logBusiness('phone_relation_created', [
'relation_id' => $relation->relation_id,
'user_id' => $userId,
'phone_number' => $phoneNumber,
'type' => $relation->type,
'effective_time' => $effectiveTime->format('Y-m-d H:i:s'),
]);
\Workerman\Worker::safeEcho("【断点11】✓ 业务日志已记录\n");
\Workerman\Worker::safeEcho("\n【断点12-方法返回】准备返回结果\n");
\Workerman\Worker::safeEcho(" - 返回 relation_id: {$relation->relation_id}\n");
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n");
\Workerman\Worker::safeEcho("[UserPhoneService::addPhoneToUser] ✅ 方法执行完成\n");
\Workerman\Worker::safeEcho("━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n");
return $relation->relation_id;
}
/**
* 移除用户的手机号(失效关联)
*
* @param string $userId 用户ID
* @param string $phoneNumber 手机号
* @param \DateTimeInterface|null $expireTime 过期时间(默认当前时间)
* @return bool 是否成功
*/
public function removePhoneFromUser(string $userId, string $phoneNumber, ?\DateTimeInterface $expireTime = null): bool
{
// 过滤非数字字符
$phoneNumber = $this->filterPhoneNumber(trim($phoneNumber));
if (empty($phoneNumber)) {
return false;
}
$phoneHash = EncryptionHelper::hash($phoneNumber);
$expireTime = $expireTime ?? new \DateTimeImmutable('now');
$relations = $this->phoneRelationRepository->newQuery()
->where('phone_hash', $phoneHash)
->where('user_id', $userId)
->where('is_active', true)
->where(function($q) use ($expireTime) {
$q->whereNull('expire_time')
->orWhere('expire_time', '>', $expireTime);
})
->get();
if ($relations->isEmpty()) {
return false;
}
foreach ($relations as $relation) {
$relation->expire_time = $expireTime;
$relation->is_active = false;
$relation->update_time = new \DateTimeImmutable('now');
$relation->save();
}
LoggerHelper::logBusiness('phone_relation_removed', [
'user_id' => $userId,
'phone_number' => $phoneNumber,
'expire_time' => $expireTime->format('Y-m-d H:i:s'),
]);
return true;
}
/**
* 根据手机号查找当前用户
*
* @param string $phoneNumber 手机号
* @param \DateTimeInterface|null $atTime 查询时间点(默认为当前时间)
* @return string|null 用户ID
*/
public function findUserByPhone(string $phoneNumber, ?\DateTimeInterface $atTime = null): ?string
{
// 过滤非数字字符
$phoneNumber = $this->filterPhoneNumber(trim($phoneNumber));
if (empty($phoneNumber)) {
return null;
}
$phoneHash = EncryptionHelper::hash($phoneNumber);
$relation = $this->phoneRelationRepository->findActiveByPhoneHash($phoneHash, $atTime);
return $relation ? $relation->user_id : null;
}
/**
* 获取用户的所有手机号
*
* @param string $userId 用户ID
* @param bool $includeHistory 是否包含历史记录
* @return array<array<string, mixed>> 手机号列表
*/
public function getUserPhones(string $userId, bool $includeHistory = false): array
{
$relations = $this->phoneRelationRepository->findByUserId($userId, $includeHistory);
return array_map(function($relation) {
return [
'phone_number' => $relation->phone_number,
'type' => $relation->type,
'is_verified' => $relation->is_verified,
'effective_time' => $relation->effective_time,
'expire_time' => $relation->expire_time,
'is_active' => $relation->is_active,
'source' => $relation->source,
];
}, $relations);
}
/**
* 获取用户的所有手机号号码(仅号码列表)
*
* @param string $userId 用户ID
* @param bool $includeHistory 是否包含历史记录
* @return array<string> 手机号列表
*/
public function getUserPhoneNumbers(string $userId, bool $includeHistory = false): array
{
$relations = $this->phoneRelationRepository->findByUserId($userId, $includeHistory);
return array_map(function($relation) {
return $relation->phone_number;
}, $relations);
}
/**
* 获取手机号的历史关联记录
*
* @param string $phoneNumber 手机号
* @return array<array<string, mixed>> 历史关联记录
*/
public function getPhoneHistory(string $phoneNumber): array
{
// 过滤非数字字符
$phoneNumber = $this->filterPhoneNumber(trim($phoneNumber));
if (empty($phoneNumber)) {
return [];
}
$phoneHash = EncryptionHelper::hash($phoneNumber);
$relations = $this->phoneRelationRepository->findHistoryByPhoneHash($phoneHash);
return array_map(function($relation) {
return [
'relation_id' => $relation->relation_id,
'user_id' => $relation->user_id,
'effective_time' => $relation->effective_time,
'expire_time' => $relation->expire_time,
'is_active' => $relation->is_active,
'type' => $relation->type,
'is_verified' => $relation->is_verified,
'source' => $relation->source,
];
}, $relations);
}
/**
* 检查手机号是否已被使用(当前有效)
*
* @param string $phoneNumber 手机号
* @return bool
*/
public function isPhoneInUse(string $phoneNumber): bool
{
// 过滤非数字字符
$phoneNumber = $this->filterPhoneNumber(trim($phoneNumber));
if (empty($phoneNumber)) {
return false;
}
$phoneHash = EncryptionHelper::hash($phoneNumber);
$relation = $this->phoneRelationRepository->findActiveByPhoneHash($phoneHash);
return $relation !== null;
}
/**
* 过滤手机号中的非数字字符
*
* @param string $phoneNumber 原始手机号
* @return string 过滤后的手机号(只包含数字)
*/
protected function filterPhoneNumber(string $phoneNumber): string
{
// 移除所有非数字字符
return preg_replace('/\D/', '', $phoneNumber);
}
/**
* 验证手机号格式(内部使用,假设已经过滤过非数字字符)
*
* @param string $phoneNumber 已过滤的手机号(只包含数字)
* @return bool
*/
protected function validatePhoneNumber(string $phoneNumber): bool
{
// 中国大陆手机号11位数字以1开头
return preg_match('/^1[3-9]\d{9}$/', $phoneNumber) === 1;
}
}

View File

@@ -0,0 +1,395 @@
<?php
namespace app\service;
use app\repository\UserProfileRepository;
use app\utils\EncryptionHelper;
use app\utils\IdCardHelper;
use app\utils\LoggerHelper;
use Ramsey\Uuid\Uuid as UuidGenerator;
/**
* 用户服务
*
* 职责:
* - 创建用户(包含身份证加密)
* - 查询用户信息(支持解密身份证)
* - 根据身份证哈希匹配用户
*/
class UserService
{
public function __construct(
protected UserProfileRepository $userProfileRepository
) {
}
/**
* 创建用户
*
* @param array<string, mixed> $data 用户数据
* @return array<string, mixed> 创建的用户信息
* @throws \InvalidArgumentException
*/
public function createUser(array $data): array
{
// 验证必填字段
if (empty($data['id_card'])) {
throw new \InvalidArgumentException('身份证号不能为空');
}
$idCard = trim($data['id_card']);
$idCardType = $data['id_card_type'] ?? '身份证';
// 验证身份证格式(简单验证)
if ($idCardType === '身份证' && !$this->validateIdCard($idCard)) {
throw new \InvalidArgumentException('身份证号格式不正确');
}
// 检查是否已存在(通过身份证哈希)
$idCardHash = EncryptionHelper::hash($idCard);
$existingUser = $this->userProfileRepository->newQuery()
->where('id_card_hash', $idCardHash)
->first();
if ($existingUser) {
throw new \InvalidArgumentException('该身份证号已存在user_id: ' . $existingUser->user_id);
}
// 加密身份证
$idCardEncrypted = EncryptionHelper::encrypt($idCard);
// 生成用户ID
$userId = $data['user_id'] ?? UuidGenerator::uuid4()->toString();
$now = new \DateTimeImmutable('now');
// 从身份证号中自动提取基础信息(如果未提供)
$idCardInfo = IdCardHelper::extractInfo($idCard);
$gender = isset($data['gender']) ? (int)$data['gender'] : ($idCardInfo['gender'] > 0 ? $idCardInfo['gender'] : null);
$birthday = isset($data['birthday']) ? new \DateTimeImmutable($data['birthday']) : $idCardInfo['birthday'];
// 创建用户记录
$user = new UserProfileRepository();
$user->user_id = $userId;
$user->id_card_hash = $idCardHash;
$user->id_card_encrypted = $idCardEncrypted;
$user->id_card_type = $idCardType;
$user->name = $data['name'] ?? null;
$user->phone = $data['phone'] ?? null;
$user->address = $data['address'] ?? null;
$user->email = $data['email'] ?? null;
$user->gender = $gender;
$user->birthday = $birthday;
$user->total_amount = isset($data['total_amount']) ? (float)$data['total_amount'] : 0;
$user->total_count = isset($data['total_count']) ? (int)$data['total_count'] : 0;
$user->last_consume_time = isset($data['last_consume_time']) ? new \DateTimeImmutable($data['last_consume_time']) : null;
$user->status = isset($data['status']) ? (int)$data['status'] : 0;
$user->create_time = $now;
$user->update_time = $now;
$user->save();
LoggerHelper::logBusiness('user_created', [
'user_id' => $userId,
'name' => $user->name,
'id_card_type' => $idCardType,
]);
return [
'user_id' => $userId,
'name' => $user->name,
'phone' => $user->phone,
'id_card_type' => $idCardType,
'create_time' => $user->create_time,
];
}
/**
* 根据 user_id 获取用户信息
*
* @param string $userId 用户ID
* @param bool $decryptIdCard 是否解密身份证(需要权限控制)
* @return array<string, mixed>|null 用户信息
*/
public function getUserById(string $userId, bool $decryptIdCard = false): ?array
{
$user = $this->userProfileRepository->findByUserId($userId);
if (!$user) {
return null;
}
$result = [
'user_id' => $user->user_id,
'name' => $user->name,
'phone' => $user->phone,
'address' => $user->address,
'email' => $user->email,
'gender' => $user->gender,
'birthday' => $user->birthday,
'id_card_type' => $user->id_card_type,
'total_amount' => $user->total_amount,
'total_count' => $user->total_count,
'last_consume_time' => $user->last_consume_time,
'tags_update_time' => $user->tags_update_time,
'status' => $user->status,
'create_time' => $user->create_time,
'update_time' => $user->update_time,
];
// 如果需要解密身份证(需要权限控制)
if ($decryptIdCard) {
try {
$result['id_card'] = EncryptionHelper::decrypt($user->id_card_encrypted);
} catch (\Throwable $e) {
LoggerHelper::logError($e, ['user_id' => $userId, 'action' => 'decrypt_id_card']);
$result['id_card'] = null;
$result['decrypt_error'] = '解密失败';
}
} else {
// 返回脱敏的身份证
$result['id_card_encrypted'] = $user->id_card_encrypted;
}
return $result;
}
/**
* 根据身份证号查找用户(通过哈希匹配)
*
* @param string $idCard 身份证号
* @return array<string, mixed>|null 用户信息
*/
public function findUserByIdCard(string $idCard): ?array
{
$idCardHash = EncryptionHelper::hash($idCard);
$user = $this->userProfileRepository->newQuery()
->where('id_card_hash', $idCardHash)
->first();
if (!$user) {
return null;
}
return $this->getUserById($user->user_id, false);
}
/**
* 更新用户信息
*
* @param string $userId 用户ID
* @param array<string, mixed> $data 要更新的用户数据
* @return array<string, mixed> 更新后的用户信息
* @throws \InvalidArgumentException
*/
public function updateUser(string $userId, array $data): array
{
$user = $this->userProfileRepository->findByUserId($userId);
if (!$user) {
throw new \InvalidArgumentException("用户不存在: {$userId}");
}
$now = new \DateTimeImmutable('now');
// 更新允许修改的字段
if (isset($data['name'])) {
$user->name = $data['name'];
}
if (isset($data['phone'])) {
$user->phone = $data['phone'];
}
if (isset($data['email'])) {
$user->email = $data['email'];
}
if (isset($data['address'])) {
$user->address = $data['address'];
}
if (isset($data['gender'])) {
$user->gender = (int)$data['gender'];
}
if (isset($data['birthday'])) {
$user->birthday = new \DateTimeImmutable($data['birthday']);
}
if (isset($data['status'])) {
$user->status = (int)$data['status'];
}
$user->update_time = $now;
$user->save();
LoggerHelper::logBusiness('user_updated', [
'user_id' => $userId,
'updated_fields' => array_keys($data),
]);
return $this->getUserById($userId, false);
}
/**
* 删除用户(软删除,设置状态为禁用)
*
* @param string $userId 用户ID
* @return bool 是否删除成功
* @throws \InvalidArgumentException
*/
public function deleteUser(string $userId): bool
{
$user = $this->userProfileRepository->findByUserId($userId);
if (!$user) {
throw new \InvalidArgumentException("用户不存在: {$userId}");
}
// 软删除:设置状态为禁用
$user->status = 1; // 1 表示禁用
$user->update_time = new \DateTimeImmutable('now');
$user->save();
LoggerHelper::logBusiness('user_deleted', [
'user_id' => $userId,
]);
return true;
}
/**
* 搜索用户(支持多种条件组合)
*
* @param array<string, mixed> $conditions 搜索条件
* - name: 姓名(模糊搜索)
* - phone: 手机号(精确或模糊)
* - email: 邮箱(精确或模糊)
* - id_card: 身份证号(精确匹配)
* - gender: 性别0-未知1-男2-女)
* - status: 状态0-正常1-禁用)
* - min_total_amount: 最小总消费金额
* - max_total_amount: 最大总消费金额
* - min_total_count: 最小消费次数
* - max_total_count: 最大消费次数
* @param int $page 页码从1开始
* @param int $pageSize 每页数量
* @return array<string, mixed> 返回用户列表和分页信息
*/
public function searchUsers(array $conditions, int $page = 1, int $pageSize = 20): array
{
$query = $this->userProfileRepository->newQuery();
// 姓名模糊搜索MongoDB 使用正则表达式)
if (!empty($conditions['name'])) {
$namePattern = preg_quote($conditions['name'], '/');
$query->where('name', 'regex', "/{$namePattern}/i");
}
// 手机号搜索(支持精确和模糊)
if (!empty($conditions['phone'])) {
if (isset($conditions['phone_exact']) && $conditions['phone_exact']) {
// 精确匹配
$query->where('phone', $conditions['phone']);
} else {
// 模糊匹配MongoDB 使用正则表达式)
$phonePattern = preg_quote($conditions['phone'], '/');
$query->where('phone', 'regex', "/{$phonePattern}/i");
}
}
// 邮箱搜索(支持精确和模糊)
if (!empty($conditions['email'])) {
if (isset($conditions['email_exact']) && $conditions['email_exact']) {
// 精确匹配
$query->where('email', $conditions['email']);
} else {
// 模糊匹配MongoDB 使用正则表达式)
$emailPattern = preg_quote($conditions['email'], '/');
$query->where('email', 'regex', "/{$emailPattern}/i");
}
}
// 如果指定了 user_ids限制搜索范围
if (!empty($conditions['user_ids']) && is_array($conditions['user_ids'])) {
$query->whereIn('user_id', $conditions['user_ids']);
}
// 身份证号精确匹配(通过哈希)
if (!empty($conditions['id_card'])) {
$idCardHash = EncryptionHelper::hash($conditions['id_card']);
$query->where('id_card_hash', $idCardHash);
}
// 性别筛选
if (isset($conditions['gender']) && $conditions['gender'] !== '') {
$query->where('gender', (int)$conditions['gender']);
}
// 状态筛选
if (isset($conditions['status']) && $conditions['status'] !== '') {
$query->where('status', (int)$conditions['status']);
}
// 总消费金额范围
if (isset($conditions['min_total_amount'])) {
$query->where('total_amount', '>=', (float)$conditions['min_total_amount']);
}
if (isset($conditions['max_total_amount'])) {
$query->where('total_amount', '<=', (float)$conditions['max_total_amount']);
}
// 消费次数范围
if (isset($conditions['min_total_count'])) {
$query->where('total_count', '>=', (int)$conditions['min_total_count']);
}
if (isset($conditions['max_total_count'])) {
$query->where('total_count', '<=', (int)$conditions['max_total_count']);
}
// 分页
$total = $query->count();
$users = $query->skip(($page - 1) * $pageSize)
->take($pageSize)
->orderBy('create_time', 'desc')
->get();
// 转换为数组格式
$result = [];
foreach ($users as $user) {
$result[] = [
'user_id' => $user->user_id,
'name' => $user->name,
'phone' => $user->phone,
'email' => $user->email,
'address' => $user->address,
'gender' => $user->gender,
'birthday' => $user->birthday,
'id_card_type' => $user->id_card_type,
'total_amount' => $user->total_amount,
'total_count' => $user->total_count,
'last_consume_time' => $user->last_consume_time,
'tags_update_time' => $user->tags_update_time,
'status' => $user->status,
'create_time' => $user->create_time,
'update_time' => $user->update_time,
];
}
return [
'users' => $result,
'total' => $total,
'page' => $page,
'page_size' => $pageSize,
'total_pages' => (int)ceil($total / $pageSize),
];
}
/**
* 验证身份证号格式(简单验证)
*
* @param string $idCard 身份证号
* @return bool
*/
protected function validateIdCard(string $idCard): bool
{
// 15位或18位数字最后一位可能是X
return preg_match('/^(\d{15}|\d{17}[\dXx])$/', $idCard) === 1;
}
}

View File

@@ -0,0 +1,137 @@
<?php
namespace app\utils;
/**
* API 响应辅助工具类
*
* 提供统一的 API 响应格式
*/
class ApiResponseHelper
{
/**
* 判断是否为开发环境
*
* @return bool
*/
protected static function isDevelopment(): bool
{
return config('app.debug', false) || env('APP_ENV', 'production') === 'development';
}
/**
* 成功响应
*
* @param mixed $data 响应数据
* @param string $message 响应消息
* @param int $httpCode HTTP状态码
* @return \support\Response
*/
public static function success($data = null, string $message = 'ok', int $httpCode = 200): \support\Response
{
$response = [
'code' => 0,
'msg' => $message,
];
if ($data !== null) {
$response['data'] = $data;
}
return json($response, $httpCode);
}
/**
* 错误响应
*
* @param string $message 错误消息
* @param int $code 错误码业务错误码非HTTP状态码
* @param int $httpCode HTTP状态码
* @param array<string, mixed> $extra 额外信息
* @return \support\Response
*/
public static function error(
string $message,
int $code = 400,
int $httpCode = 400,
array $extra = []
): \support\Response {
$response = [
'code' => $code,
'msg' => $message,
];
// 开发环境可以返回更多调试信息
if (self::isDevelopment() && !empty($extra)) {
$response = array_merge($response, $extra);
}
return json($response, $httpCode);
}
/**
* 异常响应
*
* @param \Throwable $exception 异常对象
* @param int $httpCode HTTP状态码
* @return \support\Response
*/
public static function exception(\Throwable $exception, int $httpCode = 500): \support\Response
{
// 记录错误日志
LoggerHelper::logError($exception);
$code = 500;
$message = '内部服务器错误';
// 根据异常类型设置错误码和消息
if ($exception instanceof \InvalidArgumentException) {
$code = 400;
$message = $exception->getMessage();
} elseif ($exception instanceof \RuntimeException) {
$code = 500;
$message = $exception->getMessage();
}
$response = [
'code' => $code,
'msg' => $message,
];
// 开发环境返回详细错误信息
if (self::isDevelopment()) {
$response['debug'] = [
'message' => $exception->getMessage(),
'file' => $exception->getFile(),
'line' => $exception->getLine(),
'trace' => $exception->getTraceAsString(),
];
}
return json($response, $httpCode);
}
/**
* 验证错误响应
*
* @param array<string, string> $errors 验证错误列表
* @return \support\Response
*/
public static function validationError(array $errors): \support\Response
{
$message = '参数验证失败';
if (!empty($errors)) {
$firstError = reset($errors);
$message = is_array($firstError) ? $firstError[0] : $firstError;
}
$response = [
'code' => 400,
'msg' => $message,
'errors' => $errors,
];
return json($response, 400);
}
}

View File

@@ -0,0 +1,143 @@
<?php
namespace app\utils;
/**
* 数据脱敏工具类
*
* 用于在接口返回和日志中脱敏敏感信息
*/
class DataMaskingHelper
{
/**
* 脱敏身份证号
*
* @param string|null $idCard 身份证号
* @return string 脱敏后的身份证号110101********1234
*/
public static function maskIdCard(?string $idCard): string
{
if (empty($idCard)) {
return '';
}
$config = config('encryption.masking.id_card', []);
$prefixLength = $config['prefix_length'] ?? 6;
$suffixLength = $config['suffix_length'] ?? 4;
$maskChar = $config['mask_char'] ?? '*';
$length = mb_strlen($idCard);
if ($length <= $prefixLength + $suffixLength) {
// 如果长度不足以脱敏,返回全部用*替代
return str_repeat($maskChar, $length);
}
$prefix = mb_substr($idCard, 0, $prefixLength);
$suffix = mb_substr($idCard, -$suffixLength);
$maskLength = $length - $prefixLength - $suffixLength;
return $prefix . str_repeat($maskChar, $maskLength) . $suffix;
}
/**
* 脱敏手机号
*
* @param string|null $phone 手机号
* @return string 脱敏后的手机号138****5678
*/
public static function maskPhone(?string $phone): string
{
if (empty($phone)) {
return '';
}
$config = config('encryption.masking.phone', []);
$prefixLength = $config['prefix_length'] ?? 3;
$suffixLength = $config['suffix_length'] ?? 4;
$maskChar = $config['mask_char'] ?? '*';
$length = mb_strlen($phone);
if ($length <= $prefixLength + $suffixLength) {
return str_repeat($maskChar, $length);
}
$prefix = mb_substr($phone, 0, $prefixLength);
$suffix = mb_substr($phone, -$suffixLength);
$maskLength = $length - $prefixLength - $suffixLength;
return $prefix . str_repeat($maskChar, $maskLength) . $suffix;
}
/**
* 脱敏邮箱
*
* @param string|null $email 邮箱
* @return string 脱敏后的邮箱ab***@example.com
*/
public static function maskEmail(?string $email): string
{
if (empty($email)) {
return '';
}
$config = config('encryption.masking.email', []);
$prefixLength = $config['prefix_length'] ?? 2;
$maskChar = $config['mask_char'] ?? '*';
$atPos = mb_strpos($email, '@');
if ($atPos === false) {
// 如果没有@符号,按普通字符串处理
$length = mb_strlen($email);
if ($length <= $prefixLength) {
return str_repeat($maskChar, $length);
}
$prefix = mb_substr($email, 0, $prefixLength);
return $prefix . str_repeat($maskChar, $length - $prefixLength);
}
$localPart = mb_substr($email, 0, $atPos);
$domain = mb_substr($email, $atPos);
$localLength = mb_strlen($localPart);
if ($localLength <= $prefixLength) {
$maskedLocal = str_repeat($maskChar, $localLength);
} else {
$prefix = mb_substr($localPart, 0, $prefixLength);
$maskedLocal = $prefix . str_repeat($maskChar, $localLength - $prefixLength);
}
return $maskedLocal . $domain;
}
/**
* 脱敏数组中的敏感字段
*
* @param array<string, mixed> $data 数据数组
* @param array<string> $sensitiveFields 敏感字段列表(如:['id_card', 'phone', 'email']
* @return array<string, mixed> 脱敏后的数组
*/
public static function maskArray(array $data, array $sensitiveFields = ['id_card', 'id_card_encrypted', 'phone', 'email']): array
{
$masked = $data;
foreach ($sensitiveFields as $field) {
if (isset($masked[$field]) && is_string($masked[$field])) {
switch ($field) {
case 'id_card':
case 'id_card_encrypted':
$masked[$field] = self::maskIdCard($masked[$field]);
break;
case 'phone':
$masked[$field] = self::maskPhone($masked[$field]);
break;
case 'email':
$masked[$field] = self::maskEmail($masked[$field]);
break;
}
}
}
return $masked;
}
}

View File

@@ -0,0 +1,141 @@
<?php
namespace app\utils;
/**
* 加密工具类
*
* 提供身份证等敏感数据的加密、解密和哈希功能
*/
class EncryptionHelper
{
/**
* 加密字符串(使用 AES-256-CBC
*
* @param string $plaintext 明文
* @return string 加密后的密文base64编码包含IV
* @throws \RuntimeException
*/
public static function encrypt(string $plaintext): string
{
if (empty($plaintext)) {
return '';
}
$config = config('encryption', []);
$keyString = $config['aes']['key'] ?? '';
$cipher = $config['aes']['cipher'] ?? 'AES-256-CBC';
$ivLength = $config['aes']['iv_length'] ?? 16;
if (empty($keyString)) {
throw new \InvalidArgumentException('加密密钥配置错误,密钥不能为空');
}
// 使用 SHA256 哈希处理密钥确保密钥长度为32字节AES-256需要256位密钥
// 即使原始密钥长度不够,哈希后也会得到固定长度的密钥
$key = substr(hash('sha256', $keyString), 0, 32);
// 生成随机IV
$iv = openssl_random_pseudo_bytes($ivLength);
if ($iv === false) {
throw new \RuntimeException('无法生成随机IV');
}
// 加密
$encrypted = openssl_encrypt($plaintext, $cipher, $key, OPENSSL_RAW_DATA, $iv);
if ($encrypted === false) {
throw new \RuntimeException('加密失败: ' . openssl_error_string());
}
// 将IV和密文组合然后base64编码
return base64_encode($iv . $encrypted);
}
/**
* 解密字符串
*
* @param string $ciphertext 密文base64编码包含IV
* @return string 解密后的明文
* @throws \RuntimeException
*/
public static function decrypt(string $ciphertext): string
{
if (empty($ciphertext)) {
return '';
}
$config = config('encryption', []);
$keyString = $config['aes']['key'] ?? '';
$cipher = $config['aes']['cipher'] ?? 'AES-256-CBC';
$ivLength = $config['aes']['iv_length'] ?? 16;
if (empty($keyString)) {
throw new \InvalidArgumentException('加密密钥配置错误,密钥不能为空');
}
// 使用 SHA256 哈希处理密钥确保密钥长度为32字节
// 即使原始密钥长度不够,哈希后也会得到固定长度的密钥
$key = substr(hash('sha256', $keyString), 0, 32);
// 解码base64
$data = base64_decode($ciphertext, true);
if ($data === false) {
throw new \RuntimeException('密文格式错误base64解码失败');
}
// 提取IV和密文
if (strlen($data) < $ivLength) {
throw new \RuntimeException('密文格式错误(长度不足)');
}
$iv = substr($data, 0, $ivLength);
$encrypted = substr($data, $ivLength);
// 解密
$decrypted = openssl_decrypt($encrypted, $cipher, $key, OPENSSL_RAW_DATA, $iv);
if ($decrypted === false) {
throw new \RuntimeException('解密失败: ' . openssl_error_string());
}
return $decrypted;
}
/**
* 计算字符串的哈希值(用于身份证匹配)
*
* @param string $plaintext 明文
* @return string 哈希值hex编码
*/
public static function hash(string $plaintext): string
{
if (empty($plaintext)) {
return '';
}
$config = config('encryption', []);
$algorithm = $config['hash']['algorithm'] ?? 'sha256';
$useSalt = $config['hash']['use_salt'] ?? false;
$salt = $config['hash']['salt'] ?? '';
$data = $plaintext;
if ($useSalt && !empty($salt)) {
$data = $salt . $plaintext;
}
return hash($algorithm, $data);
}
/**
* 验证明文是否匹配哈希值
*
* @param string $plaintext 明文
* @param string $hash 哈希值
* @return bool
*/
public static function verifyHash(string $plaintext, string $hash): bool
{
$calculatedHash = self::hash($plaintext);
return hash_equals($calculatedHash, $hash);
}
}

View File

@@ -0,0 +1,102 @@
<?php
namespace app\utils;
/**
* 身份证工具类
*
* 职责:
* - 从身份证号中提取出生日期
* - 从身份证号中提取性别
* - 验证身份证号格式
*/
class IdCardHelper
{
/**
* 从身份证号中提取出生日期
*
* @param string $idCard 身份证号15位或18位
* @return \DateTimeImmutable|null 出生日期解析失败返回null
*/
public static function extractBirthday(string $idCard): ?\DateTimeImmutable
{
$idCard = trim($idCard);
$length = strlen($idCard);
if ($length === 18) {
// 18位身份证第7-14位是出生日期YYYYMMDD
$birthDateStr = substr($idCard, 6, 8);
$year = (int)substr($birthDateStr, 0, 4);
$month = (int)substr($birthDateStr, 4, 2);
$day = (int)substr($birthDateStr, 6, 2);
} elseif ($length === 15) {
// 15位身份证第7-12位是出生日期YYMMDD
$birthDateStr = substr($idCard, 6, 6);
$year = (int)substr($birthDateStr, 0, 2);
$month = (int)substr($birthDateStr, 2, 2);
$day = (int)substr($birthDateStr, 4, 2);
// 15位身份证的年份需要加上1900或2000
// 通常出生年份在1900-2000之间如果大于当前年份的后两位则加1900否则加2000
$currentYearLastTwo = (int)date('y');
if ($year > $currentYearLastTwo) {
$year += 1900;
} else {
$year += 2000;
}
} else {
return null;
}
// 验证日期是否有效
if (!checkdate($month, $day, $year)) {
return null;
}
try {
return new \DateTimeImmutable(sprintf('%04d-%02d-%02d', $year, $month, $day));
} catch (\Throwable $e) {
return null;
}
}
/**
* 从身份证号中提取性别
*
* @param string $idCard 身份证号15位或18位
* @return int 性别1=男2=女0=未知
*/
public static function extractGender(string $idCard): int
{
$idCard = trim($idCard);
$length = strlen($idCard);
if ($length === 18) {
// 18位身份证第17位索引16是性别码
$genderCode = (int)substr($idCard, 16, 1);
} elseif ($length === 15) {
// 15位身份证第15位索引14是性别码
$genderCode = (int)substr($idCard, 14, 1);
} else {
return 0; // 未知
}
// 奇数表示男性,偶数表示女性
return ($genderCode % 2 === 1) ? 1 : 2;
}
/**
* 从身份证号中提取所有可解析的信息
*
* @param string $idCard 身份证号
* @return array<string, mixed> 包含 birthday 和 gender 的数组
*/
public static function extractInfo(string $idCard): array
{
return [
'birthday' => self::extractBirthday($idCard),
'gender' => self::extractGender($idCard),
];
}
}

View File

@@ -0,0 +1,137 @@
<?php
namespace app\utils;
use Monolog\Processor\ProcessorInterface;
/**
* 日志脱敏处理器
*
* 自动对日志中的敏感信息进行脱敏处理
* 兼容 Monolog 2.x使用 array 格式)
*/
class LogMaskingProcessor implements ProcessorInterface
{
/**
* 敏感字段列表
*
* @var array<string>
*/
protected array $sensitiveFields = [
'id_card',
'id_card_encrypted',
'id_card_hash',
'phone',
'email',
'password',
'token',
'secret',
];
/**
* 处理日志记录,对敏感信息进行脱敏
*
* @param array<string, mixed> $record Monolog 2.x 格式的日志记录数组
* @return array<string, mixed> 处理后的日志记录数组
*/
public function __invoke(array $record): array
{
// 处理 context 中的敏感信息
if (isset($record['context']) && is_array($record['context'])) {
$record['context'] = $this->maskArray($record['context']);
}
// 处理 extra 中的敏感信息
if (isset($record['extra']) && is_array($record['extra'])) {
$record['extra'] = $this->maskArray($record['extra']);
}
// 对消息本身也进行脱敏(如果包含敏感信息)
if (isset($record['message']) && is_string($record['message'])) {
$record['message'] = $this->maskString($record['message']);
}
return $record;
}
/**
* 脱敏数组中的敏感字段
*
* @param array<string, mixed> $data
* @return array<string, mixed>
*/
protected function maskArray(array $data): array
{
$masked = [];
foreach ($data as $key => $value) {
$lowerKey = strtolower($key);
// 检查字段名是否包含敏感关键词
$isSensitive = false;
foreach ($this->sensitiveFields as $field) {
if (strpos($lowerKey, $field) !== false) {
$isSensitive = true;
break;
}
}
if ($isSensitive && is_string($value)) {
// 根据字段类型选择脱敏方法
if (strpos($lowerKey, 'phone') !== false) {
$masked[$key] = DataMaskingHelper::maskPhone($value);
} elseif (strpos($lowerKey, 'email') !== false) {
$masked[$key] = DataMaskingHelper::maskEmail($value);
} elseif (strpos($lowerKey, 'id_card') !== false) {
$masked[$key] = DataMaskingHelper::maskIdCard($value);
} else {
// 其他敏感字段,用*替代
$masked[$key] = str_repeat('*', min(strlen($value), 20));
}
} elseif (is_array($value)) {
$masked[$key] = $this->maskArray($value);
} else {
$masked[$key] = $value;
}
}
return $masked;
}
/**
* 脱敏字符串中的敏感信息(简单模式,匹配常见格式)
*
* @param string $message
* @return string
*/
protected function maskString(string $message): string
{
// 匹配身份证号18位或15位数字
$message = preg_replace_callback(
'/\b\d{15}(\d{3})?[Xx]?\b/',
function ($matches) {
return DataMaskingHelper::maskIdCard($matches[0]);
},
$message
);
// 匹配手机号11位数字1开头
$message = preg_replace_callback(
'/\b1[3-9]\d{9}\b/',
function ($matches) {
return DataMaskingHelper::maskPhone($matches[0]);
},
$message
);
// 匹配邮箱
$message = preg_replace_callback(
'/\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/',
function ($matches) {
return DataMaskingHelper::maskEmail($matches[0]);
},
$message
);
return $message;
}
}

View File

@@ -0,0 +1,155 @@
<?php
namespace app\utils;
use Monolog\Logger;
/**
* 日志辅助工具类
*
* 提供结构化的日志记录方法
*/
class LoggerHelper
{
/**
* 记录请求日志
*
* @param string $method HTTP方法
* @param string $path 请求路径
* @param array<string, mixed> $params 请求参数
* @param float|null $duration 请求耗时(秒)
*/
public static function logRequest(string $method, string $path, array $params = [], ?float $duration = null): void
{
$logger = \support\Log::channel('default');
$context = [
'type' => 'request',
'method' => $method,
'path' => $path,
'params' => $params,
];
if ($duration !== null) {
$context['duration'] = round($duration * 1000, 2) . 'ms';
}
$logger->info("请求: {$method} {$path}", $context);
}
/**
* 记录业务日志
*
* @param string $action 操作名称
* @param array<string, mixed> $context 上下文信息
* @param string $level 日志级别info/warning/error
*/
public static function logBusiness(string $action, array $context = [], string $level = 'info'): void
{
$logger = \support\Log::channel('default');
$context['type'] = 'business';
$context['action'] = $action;
$logger->$level("业务操作: {$action}", $context);
}
/**
* 记录标签计算日志
*
* @param string $userId 用户ID
* @param string $tagId 标签ID
* @param array<string, mixed> $result 计算结果
* @param float|null $duration 计算耗时(秒)
*/
public static function logTagCalculation(string $userId, string $tagId, array $result, ?float $duration = null): void
{
$logger = \support\Log::channel('default');
$context = [
'type' => 'tag_calculation',
'user_id' => $userId,
'tag_id' => $tagId,
'result' => $result,
];
if ($duration !== null) {
$context['duration'] = round($duration * 1000, 2) . 'ms';
}
$logger->info("标签计算: user_id={$userId}, tag_id={$tagId}", $context);
}
/**
* 记录错误日志
*
* @param \Throwable $exception 异常对象
* @param array<string, mixed> $context 额外上下文
*/
public static function logError(\Throwable $exception, array $context = []): void
{
$logger = \support\Log::channel('default');
$context['type'] = 'error';
// 限制 trace 长度,避免内存溢出
$trace = $exception->getTraceAsString();
$originalTraceLength = strlen($trace);
$maxTraceLength = 5000; // 最大 trace 长度(字符数)
// 限制 trace 行数只保留前50行
$traceLines = explode("\n", $trace);
$originalLineCount = count($traceLines);
if ($originalLineCount > 50) {
$traceLines = array_slice($traceLines, 0, 50);
$trace = implode("\n", $traceLines) . "\n... (trace truncated, total lines: {$originalLineCount})";
}
// 限制 trace 字符长度
if (strlen($trace) > $maxTraceLength) {
$trace = substr($trace, 0, $maxTraceLength) . "\n... (trace truncated, total length: {$originalTraceLength} bytes)";
}
$context['exception'] = [
'message' => $exception->getMessage(),
'file' => $exception->getFile(),
'line' => $exception->getLine(),
'trace' => $trace,
'class' => get_class($exception),
];
// 如果上下文数据太大,也进行限制
$contextJson = json_encode($context);
if (strlen($contextJson) > 10000) {
// 如果上下文太大,只保留关键信息
$context = [
'type' => 'error',
'exception' => [
'message' => $exception->getMessage(),
'file' => $exception->getFile(),
'line' => $exception->getLine(),
'class' => get_class($exception),
'trace' => substr($trace, 0, 2000) . '... (truncated)',
],
];
}
$logger->error("异常: {$exception->getMessage()}", $context);
}
/**
* 记录性能日志
*
* @param string $operation 操作名称
* @param float $duration 耗时(秒)
* @param array<string, mixed> $context 上下文信息
*/
public static function logPerformance(string $operation, float $duration, array $context = []): void
{
$logger = \support\Log::channel('default');
$context['type'] = 'performance';
$context['operation'] = $operation;
$context['duration'] = round($duration * 1000, 2) . 'ms';
$level = $duration > 1.0 ? 'warning' : 'info';
$logger->$level("性能: {$operation} 耗时 {$context['duration']}", $context);
}
}

View File

@@ -0,0 +1,55 @@
<?php
namespace app\utils;
use MongoDB\Client;
/**
* MongoDB 连接辅助工具类
*
* 统一 MongoDB DSN 构建和客户端创建逻辑
*/
class MongoDBHelper
{
/**
* 构建 MongoDB DSN 连接字符串
*
* @param array<string, mixed> $config 数据库配置
* @return string DSN 字符串
*/
public static function buildDsn(array $config): string
{
$host = $config['host'] ?? '192.168.1.106';
$port = $config['port'] ?? 27017;
$username = $config['username'] ?? '';
$password = $config['password'] ?? '';
$authSource = $config['auth_source'] ?? 'admin';
if (!empty($username) && !empty($password)) {
return "mongodb://{$username}:{$password}@{$host}:{$port}/{$authSource}";
}
return "mongodb://{$host}:{$port}";
}
/**
* 创建 MongoDB 客户端
*
* @param array<string, mixed> $config 数据库配置
* @param array<string, mixed> $options 额外选项(可选)
* @return Client MongoDB 客户端实例
*/
public static function createClient(array $config, array $options = []): Client
{
$defaultOptions = [
'connectTimeoutMS' => 5000,
'socketTimeoutMS' => 5000,
];
return new Client(
self::buildDsn($config),
array_merge($defaultOptions, $options)
);
}
}

View File

@@ -0,0 +1,247 @@
<?php
namespace app\utils;
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;
use PhpAmqpLib\Channel\AMQPChannel;
use app\utils\LoggerHelper;
/**
* 队列服务封装
*
* 职责:
* - 封装 RabbitMQ 连接和消息推送
* - 提供统一的队列操作接口
*/
class QueueService
{
private static ?AMQPStreamConnection $connection = null;
private static ?AMQPChannel $channel = null;
private static array $config = [];
/**
* 初始化连接(单例模式)
*/
private static function initConnection(): void
{
if (self::$connection !== null && self::$connection->isConnected()) {
return;
}
$config = config('queue.connections.rabbitmq');
self::$config = $config;
try {
self::$connection = new AMQPStreamConnection(
$config['host'],
$config['port'],
$config['user'],
$config['password'],
$config['vhost'],
false, // insist
'AMQPLAIN', // login_method
null, // login_response
'en_US', // locale
$config['timeout'] ?? 10.0, // connection_timeout
$config['timeout'] ?? 10.0, // read_write_timeout
null, // context
false, // keepalive
$config['heartbeat'] ?? 0 // heartbeat
);
self::$channel = self::$connection->channel();
// 声明数据同步交换机
if (isset($config['exchanges']['data_sync'])) {
$exchangeConfig = $config['exchanges']['data_sync'];
self::$channel->exchange_declare(
$exchangeConfig['name'],
$exchangeConfig['type'],
false, // passive
$exchangeConfig['durable'],
$exchangeConfig['auto_delete']
);
}
// 声明标签计算交换机
if (isset($config['exchanges']['tag_calculation'])) {
$exchangeConfig = $config['exchanges']['tag_calculation'];
self::$channel->exchange_declare(
$exchangeConfig['name'],
$exchangeConfig['type'],
false, // passive
$exchangeConfig['durable'],
$exchangeConfig['auto_delete']
);
}
// 声明队列
if (isset($config['queues']['tag_calculation'])) {
$queueConfig = $config['queues']['tag_calculation'];
self::$channel->queue_declare(
$queueConfig['name'],
false, // passive
$queueConfig['durable'],
false, // exclusive
$queueConfig['auto_delete'],
false, // nowait
$queueConfig['arguments'] ?? []
);
// 绑定队列到交换机
if (isset($config['routing_keys']['tag_calculation'])) {
self::$channel->queue_bind(
$queueConfig['name'],
$config['exchanges']['tag_calculation']['name'],
$config['routing_keys']['tag_calculation']
);
}
}
LoggerHelper::logBusiness('queue_connection_established', [
'host' => $config['host'],
'port' => $config['port'],
]);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'QueueService',
'action' => 'initConnection',
]);
throw $e;
}
}
/**
* 推送消息到数据同步队列
*
* @param array<string, mixed> $data 消息数据包含数据源ID、数据记录等
* @return bool 是否推送成功
*/
public static function pushDataSync(array $data): bool
{
try {
self::initConnection();
$config = self::$config;
$messageConfig = config('queue.message', []);
$messageBody = json_encode($data, JSON_UNESCAPED_UNICODE);
$message = new AMQPMessage(
$messageBody,
[
'delivery_mode' => $messageConfig['delivery_mode'] ?? AMQPMessage::DELIVERY_MODE_PERSISTENT,
'content_type' => $messageConfig['content_type'] ?? 'application/json',
]
);
$exchangeName = $config['exchanges']['data_sync']['name'];
$routingKey = $config['routing_keys']['data_sync'];
self::$channel->basic_publish($message, $exchangeName, $routingKey);
LoggerHelper::logBusiness('queue_message_pushed', [
'queue' => 'data_sync',
'data' => $data,
]);
return true;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'QueueService',
'action' => 'pushDataSync',
'data' => $data,
]);
return false;
}
}
/**
* 推送消息到标签计算队列
*
* @param array<string, mixed> $data 消息数据
* @return bool 是否推送成功
*/
public static function pushTagCalculation(array $data): bool
{
try {
self::initConnection();
$config = self::$config;
$messageConfig = config('queue.message', []);
$messageBody = json_encode($data, JSON_UNESCAPED_UNICODE);
$message = new AMQPMessage(
$messageBody,
[
'delivery_mode' => $messageConfig['delivery_mode'] ?? AMQPMessage::DELIVERY_MODE_PERSISTENT,
'content_type' => $messageConfig['content_type'] ?? 'application/json',
]
);
$exchangeName = $config['exchanges']['tag_calculation']['name'];
$routingKey = $config['routing_keys']['tag_calculation'];
self::$channel->basic_publish($message, $exchangeName, $routingKey);
LoggerHelper::logBusiness('queue_message_pushed', [
'queue' => 'tag_calculation',
'data' => $data,
]);
return true;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'QueueService',
'action' => 'pushTagCalculation',
'data' => $data,
]);
return false;
}
}
/**
* 关闭连接
*/
public static function closeConnection(): void
{
try {
if (self::$channel !== null) {
self::$channel->close();
self::$channel = null;
}
if (self::$connection !== null && self::$connection->isConnected()) {
self::$connection->close();
self::$connection = null;
}
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'QueueService',
'action' => 'closeConnection',
]);
}
}
/**
* 获取通道(用于消费者)
*
* @return AMQPChannel
*/
public static function getChannel(): AMQPChannel
{
self::initConnection();
return self::$channel;
}
/**
* 获取连接(用于消费者)
*
* @return AMQPStreamConnection
*/
public static function getConnection(): AMQPStreamConnection
{
self::initConnection();
return self::$connection;
}
}

View File

@@ -0,0 +1,267 @@
<?php
namespace app\utils;
use Predis\Client;
use app\utils\LoggerHelper;
/**
* Redis 工具类
*
* 职责:
* - 封装 Redis 连接和基础操作
* - 提供分布式锁功能
*/
class RedisHelper
{
private static ?Client $client = null;
private static array $config = [];
/**
* 获取 Redis 客户端(单例模式)
*
* @return Client Redis 客户端
*/
public static function getClient(): Client
{
if (self::$client !== null) {
return self::$client;
}
// 从 session 配置中读取 Redis 配置(临时方案,后续可创建独立的 cache.php
$sessionConfig = config('session.config.redis', []);
self::$config = [
'host' => $sessionConfig['host'] ?? getenv('REDIS_HOST') ?: '127.0.0.1',
'port' => (int)($sessionConfig['port'] ?? getenv('REDIS_PORT') ?: 6379),
'password' => $sessionConfig['auth'] ?? getenv('REDIS_PASSWORD') ?: null,
'database' => (int)($sessionConfig['database'] ?? getenv('REDIS_DATABASE') ?: 0),
'timeout' => $sessionConfig['timeout'] ?? 2.0,
];
$parameters = [
'host' => self::$config['host'],
'port' => self::$config['port'],
];
if (!empty(self::$config['password'])) {
$parameters['password'] = self::$config['password'];
}
if (self::$config['database'] > 0) {
$parameters['database'] = self::$config['database'];
}
$options = [
'timeout' => self::$config['timeout'],
];
self::$client = new Client($parameters, $options);
// 测试连接
try {
self::$client->ping();
LoggerHelper::logBusiness('redis_connected', [
'host' => self::$config['host'],
'port' => self::$config['port'],
]);
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'getClient',
]);
throw $e;
}
return self::$client;
}
/**
* 获取分布式锁
*
* @param string $key 锁的键
* @param int $ttl 锁的过期时间(秒)
* @param int $retryTimes 重试次数
* @param int $retryDelay 重试延迟(毫秒)
* @return bool 是否获取成功
*/
public static function acquireLock(string $key, int $ttl = 300, int $retryTimes = 3, int $retryDelay = 1000): bool
{
$client = self::getClient();
$lockKey = "lock:{$key}";
$lockValue = uniqid(gethostname() . '_', true); // 唯一值,用于安全释放锁
for ($i = 0; $i <= $retryTimes; $i++) {
// 尝试获取锁SET key value NX EX ttl
$result = $client->set($lockKey, $lockValue, 'EX', $ttl, 'NX');
if ($result) {
LoggerHelper::logBusiness('redis_lock_acquired', [
'key' => $key,
'ttl' => $ttl,
]);
return true;
}
// 如果还有重试机会,等待后重试
if ($i < $retryTimes) {
usleep($retryDelay * 1000); // 转换为微秒
}
}
LoggerHelper::logBusiness('redis_lock_failed', [
'key' => $key,
'retry_times' => $retryTimes,
]);
return false;
}
/**
* 释放分布式锁
*
* @param string $key 锁的键
* @return bool 是否释放成功
*/
public static function releaseLock(string $key): bool
{
$client = self::getClient();
$lockKey = "lock:{$key}";
try {
$result = $client->del([$lockKey]);
if ($result > 0) {
LoggerHelper::logBusiness('redis_lock_released', [
'key' => $key,
]);
return true;
}
return false;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'releaseLock',
'key' => $key,
]);
return false;
}
}
/**
* 设置键值对
*
* @param string $key 键
* @param mixed $value 值
* @param int|null $ttl 过期时间null 表示不过期
* @return bool 是否设置成功
*/
public static function set(string $key, $value, ?int $ttl = null): bool
{
try {
$client = self::getClient();
$serialized = is_string($value) ? $value : json_encode($value, JSON_UNESCAPED_UNICODE);
if ($ttl !== null) {
$client->setex($key, $ttl, $serialized);
} else {
$client->set($key, $serialized);
}
return true;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'set',
'key' => $key,
]);
return false;
}
}
/**
* 获取键值
*
* @param string $key 键
* @return mixed 值,不存在返回 null
*/
public static function get(string $key)
{
try {
$client = self::getClient();
$value = $client->get($key);
if ($value === null) {
return null;
}
// 尝试 JSON 解码
$decoded = json_decode($value, true);
return $decoded !== null ? $decoded : $value;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'get',
'key' => $key,
]);
return null;
}
}
/**
* 删除键
*
* @param string $key 键
* @return bool 是否删除成功
*/
public static function delete(string $key): bool
{
try {
$client = self::getClient();
$result = $client->del([$key]);
return $result > 0;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'delete',
'key' => $key,
]);
return false;
}
}
/**
* 检查键是否存在
*
* @param string $key 键
* @return bool 是否存在
*/
public static function exists(string $key): bool
{
try {
$client = self::getClient();
$result = $client->exists($key);
return $result > 0;
} catch (\Throwable $e) {
LoggerHelper::logError($e, [
'component' => 'RedisHelper',
'action' => 'exists',
'key' => $key,
]);
return false;
}
}
/**
* 删除键别名兼容del方法
*
* @param string $key 键
* @return bool 是否删除成功
*/
public static function del(string $key): bool
{
return self::delete($key);
}
}

View File

@@ -0,0 +1,74 @@
<?php
/**
* 数据采集任务配置
*
* 说明:
* - 此配置文件仅保留数据库同步任务等系统级任务
* - 其他数据采集任务通过数据库 data_collection_tasks 表进行管理和配置
* - 数据库中的任务支持启动、暂停、删除等操作,由前端界面进行管理
* - 每个任务配置:数据源引用、业务处理类、调度规则、分片配置
* - 数据处理逻辑在业务代码中实现
*/
return [
// 全局配置
'global' => [
// 分布式锁配置
'distributed_lock' => [
'driver' => 'redis',
'ttl' => 300,
'retry_times' => 3,
'retry_delay' => 1000,
],
// 错误处理配置
'error_handling' => [
'max_retries' => 3,
'retry_delay' => 5,
'circuit_breaker' => [
'enabled' => true,
'failure_threshold' => 10,
'recovery_timeout' => 60,
],
],
],
// 采集任务列表(配置文件中的任务)
'tasks' => [
// 数据库同步任务(实时同步源数据库到目标数据库)
// 注意:这是一个系统级任务,通过配置文件管理,不从数据库加载
'database_sync' => [
'name' => '数据库实时同步',
'enabled' => false,
// 源数据库(从 data_sources.php 引用)
'source_data_source' => 'kr_mongodb',
// 目标数据库(从 data_sources.php 引用)
'target_data_source' => 'sync_mongodb',
// 业务处理类
'handler_class' => \app\service\DataCollection\Handler\DatabaseSyncHandler::class,
// 调度配置(数据库同步是持续运行的,不需要定时调度)
'schedule' => [
'cron' => null, // 不需要 Cron启动后持续运行
'enabled' => false, // 禁用定时调度,启动后持续运行
],
// 分片配置(每个 Worker 可以监听不同的数据库)
'sharding' => [
'strategy' => 'by_database', // 按数据库分片
'shard_count' => 1,
],
// 同步状态存储配置
'sync_state' => [
'storage' => 'file', // 使用文件存储进度DatabaseSyncService 使用文件)
'key_prefix' => 'database_sync:',
],
// 注意:业务逻辑相关配置(数据库列表、排除规则、性能配置等)
// 已移到 DatabaseSyncHandler 类中,使用默认值或从独立配置读取
],
],
];

View File

@@ -0,0 +1,63 @@
<?php
/**
* 数据源连接配置
*
* 说明:
* - 所有数据库连接的配置集合
* - 只包含连接信息,不包含业务逻辑
* - 可以被其他配置引用(如任务采集配置)
*/
return [
// 卡若的数据库(爬虫抓取的业务数据库)
'kr_mongodb' => [
'type' => 'mongodb',
'host' => getenv('KR_MONGODB_HOST'),
'port' => (int)getenv('KR_MONGODB_PORT'),
'database' => getenv('KR_MONGODB_DATABASE'),
'username' => getenv('KR_MONGODB_USER'),
'password' => getenv('KR_MONGODB_PASSWORD'),
'auth_source' => getenv('KR_MONGODB_AUTH_SOURCE'),
'options' => [
'ssl' => false,
'connectTimeoutMS' => 3000,
'socketTimeoutMS' => 5000,
'authMechanism' => 'SCRAM-SHA-256',
],
],
// 标签数据库(主机标签数据库)
'tag_mongodb' => [
'type' => 'mongodb',
'host' => getenv('TAG_MONGODB_HOST') ?: '192.168.1.106',
'port' => (int)(getenv('TAG_MONGODB_PORT') ?: 27017),
'database' => getenv('TAG_MONGODB_DATABASE') ?: 'ckb',
'username' => getenv('TAG_MONGODB_USER') ?: 'ckb',
'password' => getenv('TAG_MONGODB_PASSWORD') ?: '123456',
'auth_source' => getenv('TAG_MONGODB_AUTH') ?: 'ckb',
'options' => [
'ssl' => false,
'connectTimeoutMS' => 3000,
'socketTimeoutMS' => 5000,
'authMechanism' => 'SCRAM-SHA-256',
],
],
// 同步目标数据库主机同步KR数据库
'sync_mongodb' => [
'type' => 'mongodb',
'host' => getenv('SYNC_MONGODB_HOST'),
'port' => (int)getenv('SYNC_MONGODB_PORT'),
'database' => getenv('SYNC_MONGODB_DATABASE') ?: 'KR',
'username' => getenv('SYNC_MONGODB_USER'),
'password' => getenv('SYNC_MONGODB_PASS'),
'auth_source' => getenv('SYNC_MONGODB_AUTH'),
'options' => [
'ssl' => false,
'connectTimeoutMS' => 3000,
'socketTimeoutMS' => 5000,
'authMechanism' => 'SCRAM-SHA-256',
],
],
];

View File

@@ -0,0 +1,60 @@
<?php
/**
* 加密配置
*
* 用于身份证等敏感数据的加密和哈希
*/
return [
// AES 加密配置
'aes' => [
// 加密密钥32字节256位
// 注意:生产环境应使用环境变量或密钥管理服务,不要硬编码
// 使用 getenv() 获取环境变量,如果不存在则使用默认值
// 默认密钥至少32字符实际使用时会被 SHA256 哈希处理)
'key' => getenv('ENCRYPTION_AES_KEY') ?: 'your-32-byte-secret-key-here-12345678',
// 加密方法
'cipher' => 'AES-256-CBC',
// IV 长度(字节)
'iv_length' => 16,
],
// 哈希配置
'hash' => [
// 哈希算法(用于身份证哈希)
'algorithm' => 'sha256',
// 是否使用盐值(可选,增强安全性)
'use_salt' => true,
// 盐值(如果启用)
// 使用 getenv() 获取环境变量,如果不存在则使用默认值
'salt' => getenv('ENCRYPTION_HASH_SALT') ?: 'your-hash-salt-here',
],
// 脱敏配置
'masking' => [
// 身份证脱敏规则保留前6位和后4位中间用*替代
'id_card' => [
'prefix_length' => 6,
'suffix_length' => 4,
'mask_char' => '*',
],
// 手机号脱敏规则保留前3位和后4位中间用*替代
'phone' => [
'prefix_length' => 3,
'suffix_length' => 4,
'mask_char' => '*',
],
// 邮箱脱敏规则:保留@前的前2位和@后的域名
'email' => [
'prefix_length' => 2,
'mask_char' => '*',
],
],
];

81
Moncter/config/queue.php Normal file
View File

@@ -0,0 +1,81 @@
<?php
/**
* RabbitMQ 消息队列配置
*
* 用于标签系统的异步消息处理
*/
return [
'default' => 'rabbitmq',
'connections' => [
'rabbitmq' => [
'driver' => 'rabbitmq',
'host' => getenv('RABBITMQ_HOST') ?: '127.0.0.1',
'port' => (int)(getenv('RABBITMQ_PORT') ?: 5672),
'user' => getenv('RABBITMQ_USER') ?: 'guest',
'password' => getenv('RABBITMQ_PASSWORD') ?: 'guest',
'vhost' => getenv('RABBITMQ_VHOST') ?: '/',
'timeout' => 10, // 连接超时时间(秒)
// 队列配置
'queues' => [
// 数据同步队列:外部数据源轮询后推送的数据
'data_sync' => [
'name' => 'data_sync_queue',
'durable' => true, // 队列持久化
'auto_delete' => false,
'arguments' => [],
],
// 标签计算队列:消费记录写入后触发标签计算
'tag_calculation' => [
'name' => 'tag_calculation_queue',
'durable' => true, // 队列持久化
'auto_delete' => false,
'arguments' => [],
],
],
// 交换机配置
'exchanges' => [
'data_sync' => [
'name' => 'data_sync_exchange',
'type' => 'direct',
'durable' => true,
'auto_delete' => false,
],
'tag_calculation' => [
'name' => 'tag_calculation_exchange',
'type' => 'direct',
'durable' => true,
'auto_delete' => false,
],
],
// 路由键配置
'routing_keys' => [
'data_sync' => 'data.sync',
'tag_calculation' => 'tag.calculation',
],
],
],
// 消息配置
'message' => [
'delivery_mode' => 2, // 消息持久化2 = 持久化)
'content_type' => 'application/json',
],
// 消费者配置
'consumer' => [
'data_sync' => [
'prefetch_count' => 10, // 每次处理10条消息批量处理
'no_ack' => false, // 需要确认消息
],
'tag_calculation' => [
'prefetch_count' => 1, // 每次只处理一条消息
'no_ack' => false, // 需要确认消息
],
],
];

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,73 @@
<?php
namespace support\bootstrap;
use Webman\Bootstrap;
use Workerman\Worker;
use MongoDB\Laravel\Connection;
use Illuminate\Database\Capsule\Manager as Capsule;
use Illuminate\Database\ConnectionResolverInterface;
/**
* MongoDB 连接初始化 Bootstrap
*
* 用于初始化 MongoDB Laravel 包的数据库连接管理器
*/
class MongoDB implements Bootstrap
{
public static function start(?Worker $worker): void
{
$dbConfig = config('database', []);
if (isset($dbConfig['connections']['mongodb'])) {
$mongoConfig = $dbConfig['connections']['mongodb'];
// 构建包含认证信息的 DSN
$dsn = $mongoConfig['dsn'];
if (!empty($mongoConfig['username']) && !empty($mongoConfig['password'])) {
if (strpos($dsn, '@') === false) {
$dsn = str_replace(
'mongodb://',
'mongodb://' . urlencode($mongoConfig['username']) . ':' . urlencode($mongoConfig['password']) . '@',
$dsn
);
$dsn .= '/' . $mongoConfig['database'];
if (!empty($mongoConfig['options']['authSource'])) {
$dsn .= '?authSource=' . urlencode($mongoConfig['options']['authSource']);
}
}
}
// 过滤掉空字符串的选项
$options = array_filter($mongoConfig['options'] ?? [], function ($value) {
return $value !== '';
});
// 初始化 MongoDB Laravel 连接管理器
// 创建 Capsule 实例
$capsule = new Capsule();
// 注册 MongoDB 连接工厂
// MongoDB Laravel 包使用 'mongodb' 驱动,需要注册连接解析器
$capsule->getDatabaseManager()->extend('mongodb', function ($config, $name) {
$config['name'] = $name;
return new Connection($config);
});
// 添加 MongoDB 连接
$capsule->addConnection([
'driver' => 'mongodb',
'dsn' => $dsn,
'database' => $mongoConfig['database'],
'options' => $options,
], 'mongodb');
// 设置为全局连接管理器
$capsule->setAsGlobal();
// 启动 Eloquent ORM
$capsule->bootEloquent();
}
}
}