[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

anolting · 2024-10-11T12:12:28Z

⚠️ This issue respects the following points: ⚠️

This is a bug, not a question or a configuration/webserver/proxy issue.
This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
I agree to follow Nextcloud's Code of Conduct.

Bug description

Copying a large amount of files and folders using the web interface leads to a web view presentation where it seems as not all of the files and folders where copied. Only a fraction of the ae presented.

Steps to reproduce

Steps I used to reproduce this issue (at least in my installation). All steps are done via the web interface.

Mount external NC storage using External Storage feature in AIO NC 30
Navigate to folder containing large amounts of files and folders.
Select all and use Copy/Move to copy all to a folder residing on the local NC. The amount is about 35000 files in approximitely 30 folders and 20 GB in size.
Waiting about 24 hours to let the copying process finishing the job.
After the copying process only a fraction of files and folders are shown.
A WebDAV connection to the local AIO NC 30 shows all files and folder and a random check proves them all as accessible. See screen shot below.

The bug exists for a longer time period and I noticed it the first time after I started again using NC in version 24.

Expected behavior

All files and folder are indexed and presented in the web interface.

Nextcloud Server version

30

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Default user-backend (database)
LDAP/ Active Directory
SSO - SAML
Other

Configuration report

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "appsallowlist": false,
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.your.domain.tld",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "nextcloud.your.domain.tld"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "29.0.7.1",
        "overwrite.cli.url": "https:\/\/nextcloud.your.domain.tld\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "updatedirectory": "\/nc-updater",
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "allow_local_remote_servers": true,
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "DE",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 100,
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - admin_audit: 1.19.0
  - analytics: 5.0.1
  - analytics_sourcepack: 0.0.3
  - app_api: 3.2.1
  - appointments: 2.1.12
  - assistant: 1.1.0
  - calendar: 4.7.16
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - collectives: 2.14.4
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - context_chat: 2.2.1
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.2
  - drawio: 3.0.3
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.1
  - files_antivirus: 5.5.7
  - files_automatedtagging: 1.19.0
  - files_bpm: 1.5.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_fulltextsearch: 29.0.1
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - forms: 4.3.1
  - fulltextsearch: 29.0.1
  - fulltextsearch_elasticsearch: 29.0.1
  - integration_collaboard: 1.0.10
  - integration_openstreetmap: 1.0.12
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.8
  - maps: 1.4.0
  - multiboards: 1.0.4
  - nextcloud-aio: 0.6.0
  - nextcloud_announcements: 1.18.0
  - notes: 4.11.0
  - notifications: 2.17.0
  - notify_push: 0.7.0
  - oauth2: 1.17.1
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.7
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - spreed: 19.0.9
  - support: 1.12.0
  - survey_client: 1.17.0
  - systemtags: 1.19.0
  - tables: 0.8.0
  - tasks: 0.16.1
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - whiteboard: 1.0.3
  - workflowengine: 2.11.0
  - xwiki: 0.1.2
Disabled:
  - bruteforcesettings: 2.9.0
  - encryption: 2.17.0
  - suspicious_login: 7.0.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

This log is flooded with this two different messages:

{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}

Additional info

The text was updated successfully, but these errors were encountered:

joshtrichards · 2024-10-11T13:32:51Z

The logs remind of this: nextcloud/all-in-one#4806

joshtrichards · 2024-10-11T14:04:03Z

Are there any errors in your browser inspector Console tab or Network tab?

anolting · 2024-10-11T14:53:48Z

The logs remind of this: nextcloud/all-in-one#4806

As of my investigation this is another issue and I will place another issue as soon the reason for this one has been identified.

anolting · 2024-10-11T15:01:23Z

Are there any errors in your browser inspector Console tab or Network tab?

I hadn't open the browser inspector while copying the files. As I understand the architecture this happen on server and not by the browser.

joshtrichards · 2024-10-11T15:10:32Z

I hadn't open the browser inspector while copying the files. As I understand the architecture this happen on server and not by the browser.

The copying, yes. If I understood your report correctly though, the issue isn't the copying. It's that the files aren't displayed in the Web client, even though they're actually there based on your WebDAV checks (suggesting the copy was successful).

So I was referring to checking the browser inspect when you open that folder in the Web UI.

anolting · 2024-10-11T15:41:05Z

successful).

So I was referring to checking the browser inspect when you open that folder in the Web UI

A single digit number of error like this:

A setting with the same name is already registered [Settings.js:41:11](webpack:///nextcloud/apps/files/src/services/Settings.js)
    register Settings.js:41
    render main.js:58
    mounted FilesListHeader.vue:41
    VueJS 21
    updateNodes files.ts:77
    E pinia.mjs:1380
    fetchContent FilesList.vue:321
    dir FilesList.vue:273
    VueJS 11
    init vue-router.esm.js:3005
    init vue-router.esm.js:3004
    updateRoute vue-router.esm.js:2414
    transitionTo vue-router.esm.js:2263
    confirmTransition vue-router.esm.js:2402
    a vue-router.esm.js:2084
    Ee vue-router.esm.js:2095
    confirmTransition vue-router.esm.js:2396
    a vue-router.esm.js:2084
    a vue-router.esm.js:2088
    _ vue-router.esm.js:2384
    g vue-router.esm.js:2162
    _ vue-router.esm.js:2362
    a vue-router.esm.js:2087
    a vue-router.esm.js:2091
    Ee vue-router.esm.js:2095
    confirmTransition vue-router.esm.js:2392
    transitionTo vue-router.esm.js:2260
    push vue-router.esm.js:2606
    push vue-router.esm.js:3036
    push vue-router.esm.js:3035
    push router.ts:11
    goToRoute RouterService.ts:38

and some like this:


Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
    onmessage index.js:94
    o index.js:86
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
[index.js:94:28](webpack:///notifications/node_modules/@nextcloud/notify_push/dist/index.js)
    onmessage index.js:94
    (Async: EventHandlerNonNull)
    o index.js:86
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108

joshtrichards · 2024-10-12T23:52:41Z

Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So it may have something to do with the notify_push app (either that or you have second issue there).

anolting · 2024-10-13T07:42:20Z

Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So it may have something to do with the notify_push app (either that or you have second issue there).

As far I understand the usage of vueJS the errors have no connection to the bug this issue is based on.

IMHO, the vueJS messages, for example, so far only show a presumably missing error handling or indicate incorrect assumptions regarding the presence of certain information.

anolting · 2024-10-13T07:51:38Z

To make troubleshooting a little more targeted and not to randomly search for problems, I have carried out further tests.

To do this, I copied a section of one of the missing folders (missing in the web interface) again using NC's file management functions and made sure that there were only a few hundred files. These folders and the file can now be seen in the web interface.

So you could now simply copy the files in small steps. This doesn't seem to make much sense to me, it's not a workaround, and the problem seems to be possibly related to the lack of indexing of the copied data.

After reapeating this I noticed (which I have forgotten) the unfortunate chosen behavior to not overwrite an existing but instead add the string (maybe incorrect translated into english) "*_copy" to the end of the file name.

anolting added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Oct 11, 2024

szaimen changed the title ~~[Bug]: AIO NC 30 file manage does not show all copied files and folders after copying large amount of files and folders~~ [Bug]: NC 30 file manage does not show all copied files and folders after copying large amount of files and folders Oct 11, 2024

szaimen added the 30-feedback label Oct 11, 2024

joshtrichards added the feature: files label Oct 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

anolting commented Oct 11, 2024 •

edited by joshtrichards

Loading

joshtrichards commented Oct 11, 2024

joshtrichards commented Oct 11, 2024

anolting commented Oct 11, 2024

anolting commented Oct 11, 2024 •

edited

Loading

joshtrichards commented Oct 11, 2024 •

edited

Loading

anolting commented Oct 11, 2024

joshtrichards commented Oct 12, 2024

anolting commented Oct 13, 2024 •

edited

Loading

anolting commented Oct 13, 2024 •

edited

Loading

[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

Comments

anolting commented Oct 11, 2024 • edited by joshtrichards Loading

⚠️ This issue respects the following points: ⚠️

Bug description

Steps to reproduce

Expected behavior

Nextcloud Server version

Operating system

PHP engine version

Web server

Database engine version

Is this bug present after an update or on a fresh install?

Are you using the Nextcloud Server Encryption module?

What user-backends are you using?

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

Additional info

joshtrichards commented Oct 11, 2024

joshtrichards commented Oct 11, 2024

anolting commented Oct 11, 2024

anolting commented Oct 11, 2024 • edited Loading

joshtrichards commented Oct 11, 2024 • edited Loading

anolting commented Oct 11, 2024

joshtrichards commented Oct 12, 2024

anolting commented Oct 13, 2024 • edited Loading

anolting commented Oct 13, 2024 • edited Loading

anolting commented Oct 11, 2024 •

edited by joshtrichards

Loading

anolting commented Oct 11, 2024 •

edited

Loading

joshtrichards commented Oct 11, 2024 •

edited

Loading

anolting commented Oct 13, 2024 •

edited

Loading

anolting commented Oct 13, 2024 •

edited

Loading