feat: Text check before synchronization (#689)

* fix: icon * fix: web selector * fix: web selector * perf: link sync * dev doc * chomd doc * perf: git intro * 466 intro * intro img * add json editor (#5) * team limit * websync limit * json editor * text editor * perf: search test * change cq value type * doc * intro img --------- Co-authored-by: heheer <[email protected]>
labring · Jan 4, 2024 · 8288290 · 8288290
1 parent c2abbb5
commit 8288290
Show file tree

Hide file tree

Showing 64 changed files with 1,789 additions and 1,489 deletions.
diff --git a/.github/imgs/intro1.png b/.github/imgs/intro1.png
diff --git a/.github/imgs/intro2.png b/.github/imgs/intro2.png
diff --git a/.github/imgs/intro3.png b/.github/imgs/intro3.png
diff --git a/.github/imgs/intro4.png b/.github/imgs/intro4.png
diff --git a/docSite/content/docs/development/intro.md b/docSite/content/docs/development/intro.md
@@ -71,6 +71,8 @@ git clone [email protected]:<github_username>/FastGPT.git
 ### 5. 运行
 
 ```bash
+# 给脚本代码执行权限
+chmod -R +x ./scripts/
 # 代码根目录下执行，会安装根 package、projects 和 packages 内所有依赖
 pnpm i
 # 切换到应用目录
@@ -105,6 +107,10 @@ docker build -t dockername/fastgpt:tag --build-arg name=app --build-arg proxy=ta
 1. 如果你是连接远程的数据库，先检查对应的端口是否开放。
 2. 如果是本地运行的数据库，可尝试`host`改成`localhost`或`127.0.0.1`
 
+### sh ./scripts/postinstall.sh 没权限
+
+FastGPT 在`pnpm i`后会执行`postinstall`脚本，用于自动生成`ChakraUI`的`Type`。如果没有权限，可以先执行`chmod -R +x ./scripts/`，再执行`pnpm i`。
+
 ### 加入社区
 
 遇到困难了吗？有任何问题吗? 加入微信群与开发者和用户保持沟通。

diff --git a/docSite/content/docs/development/qa.md b/docSite/content/docs/development/qa.md
@@ -31,6 +31,10 @@ OneAPI 中没有配置该模型渠道。
 
 页面中是用 stream=true 模式，所以API也需要设置 stream=true 来进行测试。部分模型接口（国产居多）非 Stream 的兼容有点垃圾。
 
+### Incorrect API key provided: sk-xxxx.You can find your api Key at xxx
+
+OneAPI 的 API Key 配置错误，需要修改`OPENAI_API_KEY`环境变量，并重启容器（先 stop 然后 rm 掉，最后再 up -d 运行一次）。可以`exec`进入容器，`env`查看环境变量是否生效。
+
 ## Docker 部署常见问题
 
 ### 如何更新？
@@ -87,3 +91,15 @@ PG 数据库没有连接上/初始化失败，可以查看日志。FastGPT 会
 mongo连接失败，检查
 1. mongo 服务有没有起来(有些 cpu 不支持 AVX，无法用 mongo5，需要换成 mongo4.x，可以dockerhub找个最新的4.x，修改镜像版本，重新运行）
 2. 环境变量（账号密码，注意host和port）
+
+## 本地开发问题
+
+### TypeError: Cannot read properties of null (reading 'useMemo' )
+
+用 Node18 试试，可能最新的 Node 有问题。 本地开发流程：
+
+1. 根目录: `pnpm i`
+2. 复制 `config.json` -> `config.local.json`
+3. 复制 `.env.template` -> `.env.local`
+4. `cd projects/app`
+5. `pnpm dev`
diff --git a/docSite/content/docs/development/upgrading/466.md b/docSite/content/docs/development/upgrading/466.md
@@ -13,8 +13,11 @@ weight: 830
 
 ## V4.6.6 更新说明
 
-1. 新增 - 搜索方式：分离向量语义检索，全文检索和重排，通过 RRF 进行排序合并。
-2. 优化 - 问题分类提示词，id引导。测试国产商用 api 模型（百度阿里智谱讯飞）使用 Prompt 模式均可分类。
-3. UI 优化，未来将逐步替换新的UI设计。
-4. 优化代码：Icon 抽离和自动化获取。
-5. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8)
+1. 查看 [FastGPT 2024 RoadMap](https://github.com/labring/FastGPT?tab=readme-ov-file#-%E5%9C%A8%E7%BA%BF%E4%BD%BF%E7%94%A8)
+2. 新增 - Http 模块请求头支持 Json 编辑器。
+3. 新增 - [ReRank模型部署](/docs/development/custom-models/reranker/)
+4. 新增 - 搜索方式：分离向量语义检索，全文检索和重排，通过 RRF 进行排序合并。
+5. 优化 - 问题分类提示词，id引导。测试国产商用 api 模型（百度阿里智谱讯飞）使用 Prompt 模式均可分类。
+6. UI 优化，未来将逐步替换新的UI设计。
+7. 优化代码：Icon 抽离和自动化获取。
+8. 修复 - 链接读取的数据集，未保存选择器，导致同步时不使用选择器。
diff --git a/packages/global/common/file/api.d.ts b/packages/global/common/file/api.d.ts
@@ -12,4 +12,5 @@ export type UrlFetchParams = {
 export type UrlFetchResponse = {
   url: string;
   content: string;
+  selector?: string;
 }[];
diff --git a/packages/global/common/string/tiktoken/index.ts b/packages/global/common/string/tiktoken/index.ts
@@ -35,7 +35,7 @@ export function countPromptTokens(
   const text = `${role}\n${prompt}`;
   try {
     const encodeText = enc.encode(text);
-    return encodeText.length + 3; // 补充 role 估算值
+    return encodeText.length + role.length; // 补充 role 估算值
   } catch (error) {
     return text.length;
   }

diff --git a/packages/global/common/system/types/index.d.ts b/packages/global/common/system/types/index.d.ts
@@ -44,7 +44,8 @@ export type FastGPTFeConfigsType = {
     google?: string;
   };
   limit?: {
-    exportLimitMinutes?: number;
+    exportDatasetLimitMinutes?: number;
+    websiteSyncLimitMinuted?: number;
   };
   scripts?: { [key: string]: string }[];
   favicon?: string;

diff --git a/packages/global/core/dataset/constant.ts b/packages/global/core/dataset/constant.ts
@@ -73,6 +73,19 @@ export const DatasetCollectionTrainingTypeMap = {
   }
 };
 
+export enum DatasetCollectionSyncResultEnum {
+  sameRaw = 'sameRaw',
+  success = 'success'
+}
+export const DatasetCollectionSyncResultMap = {
+  [DatasetCollectionSyncResultEnum.sameRaw]: {
+    label: 'core.dataset.collection.sync.result.sameRaw'
+  },
+  [DatasetCollectionSyncResultEnum.success]: {
+    label: 'core.dataset.collection.sync.result.success'
+  }
+};
+
 /* ------------ data -------------- */
 export enum DatasetDataIndexTypeEnum {
   chunk = 'chunk',
@@ -150,16 +163,24 @@ export enum SearchScoreTypeEnum {
 }
 export const SearchScoreTypeMap = {
   [SearchScoreTypeEnum.embedding]: {
-    label: 'core.dataset.search.score.embedding'
+    label: 'core.dataset.search.score.embedding',
+    desc: 'core.dataset.search.score.embedding desc',
+    showScore: true
   },
   [SearchScoreTypeEnum.fullText]: {
-    label: 'core.dataset.search.score.fullText'
+    label: 'core.dataset.search.score.fullText',
+    desc: 'core.dataset.search.score.fullText desc',
+    showScore: false
   },
   [SearchScoreTypeEnum.reRank]: {
-    label: 'core.dataset.search.score.reRank'
+    label: 'core.dataset.search.score.reRank',
+    desc: 'core.dataset.search.score.reRank desc',
+    showScore: true
   },
   [SearchScoreTypeEnum.rrf]: {
-    label: 'core.dataset.search.score.rrf'
+    label: 'core.dataset.search.score.rrf',
+    desc: 'core.dataset.search.score.rrf desc',
+    showScore: false
   }
 };
 

diff --git a/packages/global/core/dataset/type.d.ts b/packages/global/core/dataset/type.d.ts
@@ -49,7 +49,10 @@ export type DatasetCollectionSchemaType = {
   qaPrompt?: string;
   rawTextLength?: number;
   hashRawText?: string;
-  metadata?: Record<string, any>;
+  metadata?: {
+    webPageSelector?: string;
+    [key: string]: any;
+  };
 };
 
 export type DatasetDataIndexItemType = {

diff --git a/packages/global/core/module/node/constant.ts b/packages/global/core/module/node/constant.ts
@@ -7,7 +7,10 @@ export enum FlowNodeInputTypeEnum {
   slider = 'slider',
   target = 'target', // data input
   switch = 'switch',
+
+  // editor
   textarea = 'textarea',
+  JSONEditor = 'JSONEditor',
 
   addInputParam = 'addInputParam', // params input
 

diff --git a/packages/global/core/module/template/system/http.ts b/packages/global/core/module/template/system/http.ts
@@ -55,7 +55,7 @@ export const HttpModule: FlowModuleTemplateType = {
     },
     {
       key: ModuleInputKeyEnum.httpHeader,
-      type: FlowNodeInputTypeEnum.textarea,
+      type: FlowNodeInputTypeEnum.JSONEditor,
       valueType: ModuleIOValueTypeEnum.string,
       label: 'core.module.input.label.Http Request Header',
       description: 'core.module.input.description.Http Request Header',

diff --git a/packages/global/support/user/team/type.d.ts b/packages/global/support/user/team/type.d.ts
@@ -10,6 +10,10 @@ export type TeamSchema = {
   balance: number;
   maxSize: number;
   lastDatasetBillTime: Date;
+  limit: {
+    lastExportDatasetTime: Date;
+    lastWebsiteSyncTime: Date;
+  };
 };
 
 export type TeamMemberSchema = {

diff --git a/packages/global/support/user/type.d.ts b/packages/global/support/user/type.d.ts
@@ -17,10 +17,6 @@ export type UserModelSchema = {
     key: string;
     baseUrl: string;
   };
-  limit: {
-    exportKbTime?: Date;
-    datasetMaxCount?: number;
-  };
 };
 
 export type UserType = {

diff --git a/packages/service/common/string/cheerio.ts b/packages/service/common/string/cheerio.ts
@@ -15,7 +15,8 @@ export const cheerioToHtml = ({
   // get origin url
   const originUrl = new URL(fetchUrl).origin;
 
-  const selectDom = $(selector || 'body');
+  const usedSelector = selector || 'body';
+  const selectDom = $(usedSelector);
 
   // remove i element
   selectDom.find('i,script').remove();
@@ -49,7 +50,10 @@ export const cheerioToHtml = ({
     .get()
     .join('\n');
 
-  return html;
+  return {
+    html,
+    usedSelector
+  };
 };
 export const urlsFetch = async ({
   urlList,
@@ -66,25 +70,25 @@ export const urlsFetch = async ({
           });
 
           const $ = cheerio.load(fetchRes.data);
-
-          const md = await htmlToMarkdown(
-            cheerioToHtml({
-              fetchUrl: url,
-              $,
-              selector
-            })
-          );
+          const { html, usedSelector } = cheerioToHtml({
+            fetchUrl: url,
+            $,
+            selector
+          });
+          const md = await htmlToMarkdown(html);
 
           return {
             url,
-            content: md
+            content: md,
+            selector: usedSelector
           };
         } catch (error) {
           console.log(error, 'fetch error');
 
           return {
             url,
-            content: ''
+            content: '',
+            selector: ''
           };
         }
       })

diff --git a/packages/service/common/string/markdown.ts b/packages/service/common/string/markdown.ts
@@ -21,6 +21,9 @@ export const htmlToMarkdown = (html?: string | null) =>
       worker.terminate();
       reject(err);
     });
+    worker.on('exit', (code) => {
+      console.log('html 2 md finish', code);
+    });
 
     worker.postMessage(html);
   });
diff --git a/packages/service/core/dataset/collection/controller.ts b/packages/service/core/dataset/collection/controller.ts
@@ -19,14 +19,16 @@ export async function createOneCollection({
   qaPrompt,
   hashRawText,
   rawTextLength,
-  metadata = {}
-}: CreateDatasetCollectionParams & { teamId: string; tmbId: string }) {
+  metadata = {},
+  ...props
+}: CreateDatasetCollectionParams & { teamId: string; tmbId: string; [key: string]: any }) {
   const { _id } = await MongoDatasetCollection.create({
-    name,
+    ...props,
     teamId,
     tmbId,
-    datasetId,
     parentId: parentId || null,
+    datasetId,
+    name,
     type,
     trainingType,
     chunkSize,

diff --git a/packages/service/core/dataset/collection/schema.ts b/packages/service/core/dataset/collection/schema.ts
@@ -75,6 +75,7 @@ const DatasetCollectionSchema = new Schema({
   qaPrompt: {
     type: String
   },
+
   rawTextLength: {
     type: Number
   },