Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

索引构建异常 #6962

Open
4 tasks done
Deepsea-CN opened this issue Aug 7, 2024 · 11 comments
Open
4 tasks done

索引构建异常 #6962

Deepsea-CN opened this issue Aug 7, 2024 · 11 comments
Labels
bug Something isn't working stale No activity for more than 30 days

Comments

@Deepsea-CN
Copy link

Please make sure of the following things

  • I have read the documentation.
    我已经阅读了文档。

  • I'm sure there are no duplicate issues or discussions.
    我确定没有重复的issue或讨论。

  • I'm sure it's due to AList and not something else(such as Network ,Dependencies or Operational).
    我确定是AList的问题，而不是其他原因（例如网络，依赖或操作）。

  • I'm sure this issue is not fixed in the latest version.
    我确定这个问题在最新版本中没有被修复。

AList Version / AList 版本

3.36.0

Driver used / 使用的存储驱动

多种

Describe the bug / 问题描述

第一次构建索引有10000多条，还算正常，也能搜索，后面因为更改挂载了所以就点了重建索引，但是不论删除再建还是直接重建都是扫到三四千条之后就停止了，实际搜索也确实很多搜不到

Reproduction / 复现链接

略

Config / 配置

非全文搜索

Logs / 日志

No response
@Deepsea-CN Deepsea-CN added the bug Something isn't working label Aug 7, 2024
@Muione
Copy link
Contributor

Muione commented Aug 10, 2024

有检查是不是因为构建索引的请求太多被网盘限制的原因吗？

@Deepsea-CN
Copy link
Author

有检查是不是因为构建索引的请求太多被网盘限制的原因吗？

应该不是的，可能出问题的比如阿里我直接加入排除名单了，参与索引的只有115，OneDrive，本地和R2储存桶，第一次索引是正常的没有撞限制墙，后面重建以及重试、更新都是有隔一两天的，应该碰不到限制，目前能建立的索引几千条就是这几个储存里各有一点

@Muione
Copy link
Contributor

Muione commented Aug 10, 2024

有检查是不是因为构建索引的请求太多被网盘限制的原因吗？

应该不是的，可能出问题的比如阿里我直接加入排除名单了，参与索引的只有115，OneDrive，本地和R2储存桶，第一次索引是正常的没有撞限制墙，后面重建以及重试、更新都是有隔一两天的，应该碰不到限制，目前能建立的索引几千条就是这几个储存里各有一点

你有打开alist的日志记录吗？有的话可以检查一下日志里的内容，构建索引会有下面这样的记录：

INFO[2024-07-27 13:56:25] index obj count: 194                         
INFO[2024-07-27 13:56:25] index obj count: 1                           
INFO[2024-07-27 13:56:25] index obj count: 2196                        
INFO[2024-07-27 13:56:25] index obj count: 1                           
INFO[2024-07-27 13:56:25] index obj count: 0                           
INFO[2024-07-27 13:56:25] index obj count: 357                         
INFO[2024-07-27 13:56:25] index obj count: 47                          
INFO[2024-07-27 13:56:25] index obj count: 4                           
INFO[2024-07-27 13:56:25] index obj count: 86

其中可能会记录构建索引的错误信息

@Deepsea-CN
Copy link
Author

Deepsea-CN commented Aug 10, 2024 •
edited
Loading

^[[36mINFO^[[0m[2024-08-07 19:38:24] build index for: [/]
^[[36mINFO^[[0m[2024-08-07 19:38:24] ignore paths: [/cloudflare_R2 /访客路å¾<84> /ali]
[GIN] 2024/08/07 - 19:38:24 | 200 | 925.318µs | 113..188 | GET "/api/admin/index/progress"
[GIN] 2024/08/07 - 19:38:28 | 200 | 253.684µs | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:29] index obj count: 0
[GIN] 2024/08/07 - 19:38:33 | 200 | 1.954914ms | 113...188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:34] index obj count: 46
[GIN] 2024/08/07 - 19:38:38 | 200 | 1.131198ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:39] index obj count: 169
[GIN] 2024/08/07 - 19:38:44 | 200 | 5.322924ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:44] index obj count: 273
[GIN] 2024/08/07 - 19:38:49 | 200 | 1.566337ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:49] index obj count: 274
[GIN] 2024/08/07 - 19:38:49 | 200 | 841.625512ms | 113..188 | POST "/api/fs/list"
[GIN] 2024/08/07 - 19:38:52 | 200 | 256.139749ms | 113..188 | POST "/api/fs/list"
[GIN] 2024/08/07 - 19:38:54 | 200 | 1.344336ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:54] index obj count: 509
[GIN] 2024/08/07 - 19:38:59 | 200 | 2.59974ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:38:59] index obj count: 736
[GIN] 2024/08/07 - 19:39:04 | 200 | 2.371982ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:04] index obj count: 890
[GIN] 2024/08/07 - 19:39:09 | 200 | 2.708838ms | 113.132.10.188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:09] index obj count: 1155
[GIN] 2024/08/07 - 19:39:14 | 200 | 1.537599ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:14] index obj count: 1638
[GIN] 2024/08/07 - 19:39:16 | 200 | 490.89674ms | 113..188 | POST "/api/fs/list"
[GIN] 2024/08/07 - 19:39:19 | 200 | 2.993995ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:19] index obj count: 1781
[GIN] 2024/08/07 - 19:39:21 | 200 | 1.133373475s | 113..188 | POST "/api/fs/get"
[GIN] 2024/08/07 - 19:39:24 | 200 | 2.369365ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:24] index obj count: 1938
[GIN] 2024/08/07 - 19:39:29 | 200 | 2.320558ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:29] index obj count: 2325
[GIN] 2024/08/07 - 19:39:33 | 200 | 2.617906ms | 113.1.188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:34] index obj count: 2619
[GIN] 2024/08/07 - 19:39:39 | 200 | 2.349522ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:39] index obj count: 3212
^[[36mINFO^[[0m[2024-08-07 19:39:44] index obj count: 3418
[GIN] 2024/08/07 - 19:39:44 | 200 | 57.476934ms | 113..188 | GET "/api/admin/index/progress"
[GIN] 2024/08/07 - 19:39:49 | 200 | 1.507867ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:49] index obj count: 3576
[GIN] 2024/08/07 - 19:39:54 | 200 | 1.910858ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:54] index obj count: 3819
[GIN] 2024/08/07 - 19:39:59 | 200 | 2.644726ms | 113..188 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-07 19:39:59] index obj count: 4052
^[[36mINFO^[[0m[2024-08-07 19:40:01] success build index, count: 4385

@Muione
Copy link
Contributor

Muione commented Aug 10, 2024

我刚刚试了下我这里构建索引没有问题，你这个日志里也没有错误信息。
你试下清除数据库索引，然后重启alist，然后再构建索引试试。

@Deepsea-CN
Copy link
Author

我刚刚试了下我这里构建索引没有问题，你这个日志里也没有错误信息。 你试下清除数据库索引，然后重启alist，然后再构建索引试试。

不行啊，点击重建索引现在更少了，这次只有3000多了，这是什么原因导致的。
有没有办法直接清掉这个的数据库文件，恢复到最开始那个创建索引
[GIN] 2024/08/10 - 23:07:50 | 200 | 719.721µs | 219.144.137.214 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-10 23:07:51] index obj count: 2494
[GIN] 2024/08/10 - 23:07:55 | 200 | 718.172µs | 219.144.137.214 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-10 23:07:56] index obj count: 2678
[GIN] 2024/08/10 - 23:08:00 | 200 | 803.585µs | 219.144.137.214 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-10 23:08:01] index obj count: 2826
[GIN] 2024/08/10 - 23:08:05 | 200 | 885.569µs | 219.144.137.214 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-10 23:08:06] index obj count: 3024
[GIN] 2024/08/10 - 23:08:10 | 200 | 1.280073ms | 219.144.137.214 | GET "/api/admin/index/progress"
^[[36mINFO^[[0m[2024-08-10 23:08:11] index obj count: 3347
^[[36mINFO^[[0m[2024-08-10 23:08:11] success build index, count: 3666
[GIN] 2024/08/10 - 23:08:15 | 200 | 563.34µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:20 | 200 | 149.528µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:25 | 200 | 180.202µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:30 | 200 | 176.923µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:35 | 200 | 263.18µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:40 | 200 | 328.144µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:45 | 200 | 271.248µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:51 | 200 | 221.951µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:08:56 | 200 | 344.26µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:09:01 | 200 | 188.843µs | 219.144.137.214 | GET "/api/admin/index/progress"
[GIN] 2024/08/10 - 23:09:06 | 200 | 224.867µs | 219.144.137.214 | GET "/api/admin/index/progress"

@Muione
Copy link
Contributor

Muione commented Aug 10, 2024

恢复之前的索引应该不行。
你尝试下单独使用更新索引对某个网盘进行索引更新试试
image

@Deepsea-CN
Copy link
Author

不行·，单加的话OneDrive1129 115有2532，没有变化，我看他显示完成之后都还会有一溜GET "/api/admin/index/progress"
是不是实际上就没有全部扫完他就结束了

@Muione
Copy link
Contributor

Muione commented Aug 10, 2024

不行·，单加的话OneDrive1129 115有2532，没有变化，我看他显示完成之后都还会有一溜GET "/api/admin/index/progress" 是不是实际上就没有全部扫完他就结束了

GET "/api/admin/index/progress" 这个请求是网页前端向服务器发起的索引状态的请求。
^[[36mINFO^[[0m[2024-08-10 23:08:11] success build index, count: 3666 这一行日志显示已经完成索引构建了。
应该是网盘那边限制了你的请求，过段时间再重新构建吧。

@rammiah
Copy link
Contributor

rammiah commented Aug 18, 2024

索引构建部分有并发问题，有可能导致索引不全 #7035

@stale Stale
Copy link

stale bot commented Oct 1, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale No activity for more than 30 days label Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working stale No activity for more than 30 days
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rammiah @Muione @Deepsea-CN