Issues with some webdav clients

[kty0mka]

It seemed to me a good idea to connect the DMSF to the Nextcloud as an external webdav storage, but unfortunately it did not work. Here is log:

INFO -- : Started PROPFIND "/dmsf/webdav/3d/" for 192.168.4.12 at ...
INFO -- : Processing WebDAV request: /dmsf/webdav/3d/ (for 192.168.4.12 at ...) [PROPFIND]
INFO -- : Completed in: 8 ms | 207 [http://project.lan/dmsf/webdav/3d/]

Then I tried connecting via Cyberduck and again failed:

INFO -- : Started PROPFIND "/dmsf/webdav/3d" for 192.168.4.43 at ...
INFO -- : Processing WebDAV request: /dmsf/webdav/3d (for 192.168.4.43 at ...) [PROPFIND]
WARN -- : Scoped order is ignored, it's forced to be batch order.
INFO -- : Completed in: 12 ms | 207 [http://project.lan/dmsf/webdav/3d]

At the same time, I can access the DMSF webdav via Safari, Firefox, native webdav clients in osx Finder or Gnome Nautilus.

I tested on fresh Redmine 4.1 with the only DMSF 2.4.2 plugin installed, but Redmine 4.0.7 with DMSF 2.4.1 behaves the same.

Environment:
  Redmine version                4.1.1.stable
  Ruby version                   2.6.6-p146 (2020-03-31) [x86_64-linux]
  Rails version                  5.2.4.2
  Environment                    production
  Database adapter               SQLite
  Mailer queue                   ActiveJob::QueueAdapters::AsyncAdapter
  Mailer delivery                smtp
SCM:
  Subversion                     1.10.4
  Mercurial                      4.8.2
  Bazaar                         2.8.0
  Git                            2.20.1
  Filesystem                     
Redmine plugins:
  redmine_dmsf                   2.4.2

[picman]

The snippets of the log don't say to much. The return code 207 is a correct answer on PROPFIND request. However, it's not visible what is inside the response.
I guess that the problem might be in usage of HTTP instead of HTTPS. Some WebDAV clients refuse authentication via HTTP.

[kty0mka]

HTTP was used only for testing. I specifically removed everything unnecessary for test conditions. At first I thought that maybe the problem is somewhere in the reverse proxy or passenger. Therefore, the logs show the HTTP protocol.
In real, I have nginx as a reverse-proxy and a valid SSL certificate. And the result the same...

I also first thought about the specific implementation of the webdav client in Nextcloud, which is why I tried Cyberduck.

In CLI mode Cyberduck gives the following:

Authenticating as user***…
> HEAD /dmsf/webdav/3d/ HTTP/1.1
> Host: pm.midl.com.ua
> Connection: Keep-Alive
> User-Agent: Cyberduck/7.4.0.32960 (Mac OS X/10.15.4) (x86_64)
> Accept-Encoding: gzip,deflate
> Authorization: ********
< HTTP/1.1 200 OK
< Server: nginx/1.17.5
< Date: Fri, 12 Jun 2020 13:38:13 GMT
< Content-Type: inode/directory
< Content-Length: 4096
< Connection: keep-alive
< Status: 200 OK
< Last-Modified: Tue, 05 May 2020 22:06:11 GMT
< X-Request-Id: 6854de49-0749-4b25-9eed-882b1723b51c
< Etag: 0-1000-5eb1e353
< X-Runtime: 0.009093
< X-Powered-By: Phusion Passenger 6.0.5
Login successful…
> PROPFIND /dmsf/webdav/3d/ HTTP/1.1
> Depth: 1
> Content-Type: text/xml; charset=utf-8
> Content-Length: 297
> Host: pm.midl.com.ua
> Connection: Keep-Alive
> User-Agent: Cyberduck/7.4.0.32960 (Mac OS X/10.15.4) (x86_64)
> Accept-Encoding: gzip,deflate
> Authorization: ********
< HTTP/1.1 207 Multi-Status
< Server: nginx/1.17.5
< Date: Fri, 12 Jun 2020 13:38:17 GMT
< Content-Type: application/xml; charset=utf-8
< Content-Length: 4518
< Connection: keep-alive
< Status: 207 Multi-Status
< X-Request-Id: 824c3ee4-059d-4ff3-975e-ef6a765baeff
< X-Runtime: 0.048199
< X-Powered-By: Phusion Passenger 6.0.5
< Strict-Transport-Security: max-age=31536000
Listing directory 3d failed. Status code: 207, reason phrase: Not a valid DAV response (207 Multi-Status).

I'm not sure whether it will give something to understand.
Perhaps there is a way to get a more detailed log from server side?

[kty0mka]

I sent a bug report to Mountainduck/Cyberduck and this is what they answered me:

• PROPFIND returns 400 Bad Request chrisroberts/dav4rack#46 (comment)
• Look for the DAV: namespace (it may not be the first), and allow resolving other namespaces. chrisroberts/dav4rack#33
• https://github.com/schmurfy/dav4rack

Further information:
Cyberduck creates a PROPFIND request to WebDAV with following content:

<propfind xmlns="DAV:">
   <prop>
       <creationdate/>
       <displayname/>
       <getcontentlength/>
       <getcontenttype/>
       <getetag/>
       <getlastmodified/>
       <resourcetype/>
       <s:lastmodified xmlns:s="SAR:"/>
       <s:lastmodified_server xmlns:s="SAR:"/>
   </prop>
</propfind>

There is a custom namespace included for features not supported by the WebDAV-protocol (application configurable lastmodified-dates).
The underlying WebDAV implementation of Redmine DMSF (dav4rack) does not support custom namespaces and returns with an invalid unknown:-prefixed property - thus violing the WebDAV protocol standard which in turn results in Mountain Duck and Cyberduck not being able to connect to this server. For reference, this is the invalid answer from your DMSF setup:

<d:response>
 <d:href>https://pm.midl.com.ua:443/dmsf/webdav/</d:href>
 <d:propstat>
   <d:prop>
     <d:creationdate>2020-06-15T12:00:41+03:00</d:creationdate>
     <d:displayname>/</d:displayname>
     <d:getcontentlength>4096</d:getcontentlength>
     <d:getcontenttype>inode/directory</d:getcontenttype>
     <d:getetag>1-1000-5ee738b9</d:getetag>
     <d:getlastmodified>Mon, 15 Jun 2020 09:00:41 GMT</d:getlastmodified>
     <d:resourcetype>
       <d:collection/>
     </d:resourcetype>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
 </d:propstat>
 <d:propstat>
   <d:prop>
     <unknown58131:lastmodified/>
     <unknown58131:lastmodified_server/>
   </d:prop>
   <d:status>HTTP/1.1 404 Not Found</d:status>
 </d:propstat>
</d:response>

This would be a valid response:

<d:response>
 <d:href>/remote.php/webdav/</d:href>
 <d:propstat>
   <d:prop>
     <d:getetag>&quot;b6e653d7f0186d18c77993b30c756a22&quot;</d:getetag>
     <d:getlastmodified>Wed, 10 Jun 2020 07:54:58 GMT</d:getlastmodified>
     <d:resourcetype>
       <d:collection/>
     </d:resourcetype>
   </d:prop>
   <d:status>HTTP/1.1 200 OK</d:status>
 </d:propstat>
 <d:propstat>
   <d:prop>
     <d:creationdate/>
     <d:displayname/>
     <d:getcontentlength/>
     <d:getcontenttype/>
     <x1:lastmodified xmlns:x1="SAR:"/>
     <x1:lastmodified_server xmlns:x1="SAR:"/>
   </d:prop>
   <d:status>HTTP/1.1 404 Not Found</d:status>
 </d:propstat>
</d:response>

[picman]

How about adding the custom namespace here:

lib/redmine_dmsf/webdav/custom_middleware.rb

39 namespaces: {
40   'http://apache.org/dav/props/' => 'd',
41   'http://ucb.openoffice.org/dav/props/' => 'd'
-> add your namespace here
42 }

[kty0mka]

Hi, @picman

Sory, I'm not a software developer, so cannot advise on how to do it better.
I think any option that fixes the problem will be good.

[picman]

After adding Cyberduck's namespace 94e41f1 it seems to be working...

$ duck -l http://localhost:3000/dmsf/webdav/c1
Username (anonymous): ***
Login as ***
Password: 
Login successful…
[c1-sub]
.Issues
f3
folder[1]
folder[2]

[kty0mka]

Hi, @picman

Thank you for your time! Adding the desired client’s namespace really looks like a simple step. I just didn’t quite understand what it was about.

Cyberduck now works as a client, but I just used it for testing purposes. What started it all when I ran into the bug was connecting WebDav (by DMSF) as external storage to Nextcloud.

I have already tried adding to lib/redmine_dmsf/webdav/custom_middleware.rb all possible namespaces related to Nextcloud / OwnCloud / Sabre Dav (which is used in the first two as a webdav backend):

              namespaces: {
                'http://apache.org/dav/props/' => 'd',
                'http://ucb.openoffice.org/dav/props/' => 'd',
                'SAR:' => 'd',
                'http://owncloud.org/ns/' => 'd',
                'http://open-collaboration-services.org/ns/' => 'd',
                'http://sabredav.org/ns/' => 'd'
              }

But all is unsuccessful...

I also ran tests for compliant with the WebDAV protocol using litmus WebDav test suite and got the following results:

-> running `basic':
 0. init.................. pass
 1. begin................. pass
 2. options............... pass
 3. put_get............... pass
 4. put_get_utf8_segment.. pass
 5. put_no_parent......... pass
 6. mkcol_over_plain...... pass
 7. delete................ pass
 8. delete_null........... pass
 9. delete_fragment....... pass
10. mkcol................. pass
11. mkcol_again........... pass
12. delete_coll........... pass
13. mkcol_no_parent....... pass
14. mkcol_with_body....... pass
15. finish................ pass
<- summary for `basic': of 16 tests run: 16 passed, 0 failed. 100.0%

-> running `copymove':
 0. init.................. pass
 1. begin................. pass
 2. copy_init............. pass
 3. copy_simple........... FAIL (simple resource COPY:
500 Internal Server Error)
 4. copy_overwrite........ FAIL (COPY-on-existing with 'Overwrite: F' MUST fail with 412 (RFC4918:10.6):
500 Internal Server Error)
 5. copy_nodestcoll....... pass
 6. copy_cleanup.......... pass
 7. copy_coll............. FAIL (COPY-on-existing-coll with overwrite: 501 Not Implemented)
 8. copy_shallow.......... FAIL (MKCOL on `/dmsf/webdav/brands_database/litmus/ccsrc/': 405 Method Not Allowed)
 9. move.................. FAIL (MOVE onto existing resource with Overwrite: T `/dmsf/webdav/brands_database/litmus/move2' to `/dmsf/webdav/brands_database/litmus/movedest': 501 Not Implemented)
10. move_coll............. FAIL (MOVE-on-existing-coll with overwrite)
11. move_cleanup.......... pass
12. finish................ pass
<- summary for `copymove': of 13 tests run: 7 passed, 6 failed. 53.8%

-> running `props':
 0. init.................. pass
 1. begin................. pass
 2. propfind_invalid...... pass
 3. propfind_invalid2..... pass
 4. propfind_d0........... FAIL (No responses returned)
 5. propinit.............. pass
 6. propset............... FAIL (PROPPATCH on `/dmsf/webdav/brands_database/litmus/prop': 207 Multi-Status)
 7. propget............... SKIPPED
 8. propextended.......... pass
 9. propmove.............. SKIPPED
10. propget............... SKIPPED
11. propdeletes........... SKIPPED
12. propget............... SKIPPED
13. propreplace........... SKIPPED
14. propget............... SKIPPED
15. propnullns............ SKIPPED
16. propget............... SKIPPED
17. prophighunicode....... SKIPPED
18. propget............... SKIPPED
19. propremoveset......... SKIPPED
20. propget............... SKIPPED
21. propsetremove......... SKIPPED
22. propget............... SKIPPED
23. propvalnspace......... SKIPPED
24. propwformed........... pass
25. propinit.............. pass
26. propmanyns............ FAIL (PROPPATCH on `/dmsf/webdav/brands_database/litmus/prop': 207 Multi-Status)
27. propget............... FAIL (PROPFIND on `/dmsf/webdav/brands_database/litmus/prop': XML parse error at line 1: undeclared namespace prefix)
28. propcleanup........... pass
29. finish................ pass
-> 16 tests were skipped.
<- summary for `props': of 14 tests run: 10 passed, 4 failed. 71.4%

-> running `locks':
 0. init.................. pass
 1. begin................. pass
 2. options............... pass
 3. precond............... pass
 4. init_locks............ pass
 5. put................... pass
 6. lock_excl............. pass
 7. discover.............. pass
 8. refresh............... pass
 9. notowner_modify....... FAIL (DELETE of locked resource should fail)
10. notowner_lock......... WARNING: LOCK failed with 207 not 423
    ...................... pass (with 1 warning)
11. owner_modify.......... FAIL (PROPPATCH on locked resouce on `/dmsf/webdav/brands_database/litmus/lockme': 207 Multi-Status)
12. notowner_modify....... FAIL (DELETE of locked resource should fail)
13. notowner_lock......... WARNING: LOCK failed with 207 not 423
    ...................... pass (with 1 warning)
14. copy.................. FAIL (could not COPY locked resource:
404 Not Found)
15. cond_put.............. SKIPPED
16. fail_cond_put......... SKIPPED
17. cond_put_with_not..... pass
18. cond_put_corrupt_token FAIL (conditional PUT with invalid lock-token should fail: 201 Created)
19. complex_cond_put...... pass
20. fail_complex_cond_put. FAIL (PUT with complex bogus conditional should fail with 412: 201 Created)
21. unlock................ pass
22. fail_cond_put_unlocked FAIL (conditional PUT with invalid lock-token should fail: 201 Created)
23. lock_shared........... FAIL (requested lockscope not satisfied!  got shared, wanted exclusive)
24. notowner_modify....... SKIPPED
25. notowner_lock......... SKIPPED
26. owner_modify.......... SKIPPED
27. double_sharedlock..... SKIPPED
28. notowner_modify....... SKIPPED
29. notowner_lock......... SKIPPED
30. unlock................ SKIPPED
31. prep_collection....... pass
32. lock_collection....... pass
33. owner_modify.......... FAIL (PUT on locked resource failed: 502 Bad Gateway)
34. notowner_modify....... FAIL (DELETE of locked resource should fail)
35. refresh............... pass
36. indirect_refresh...... FAIL (indirect refresh LOCK on /dmsf/webdav/brands_database/litmus/lockcoll/ via /dmsf/webdav/brands_database/litmus/lockcoll/lockme.txt: No activelock for <4bd7a1e9-b7cd-47fc-8007-460b38c2d972> returned in LOCK refresh response)
37. unlock................ pass
38. unmapped_lock......... FAIL (LOCK on /dmsf/webdav/brands_database/litmus/lockcoll/ via /dmsf/webdav/brands_database/litmus/unmapped_url: 207 Multi-Status)
39. unlock................ SKIPPED
40. finish................ pass
-> 10 tests were skipped.
<- summary for `locks': of 31 tests run: 19 passed, 12 failed. 61.3%
-> 2 warnings were issued.

-> running `http':
 0. init.................. pass
 1. begin................. pass
 2. expect100............. SKIPPED (skipping for SSL server)
 3. finish................ pass
-> 1 test was skipped.
<- summary for `http': of 3 tests run: 3 passed, 0 failed. 100.0%

In the "props" test block 4 of 14 tests fail. Perhaps there is a problem here. Also, unfortunately, there are a lot of failed tests in the "copymove" and "locks" blocks.

Here is detailed litmus debug log:
litmus_debug.log

[picman]

How do you work with DMSF <==> Nextcloud? How can I test it?

[kty0mka]

You can test on these test hosts:

Redmine: https://pm.midl.com.ua
Nextcloud: https://nc2.midl.com.ua

Credentials (for both hosts):
admin:20Dmsf21

I used the latest stable version of Redmine and the DMSF plugin from the devel-2.4.6 branch.

In Nextcloud, I set up two WebDav connections:

1. Loopback to Nextcloud itself host (working)
2. Connecting to the above Redmine host (not working)

You can view and change the settings for connecting to WebDav resources in Nextcloud here: Settings (under user icon in the right top corner), External storages (in the Personal block).

Nextcloud logs can be viewed also in Settings -> Logging (in the Administration block).

[picman]

I've improved WebDAV of DMSF in order to pass Litmus tests. Could you re-test it with Nextcloud? (devel-2.4.6) The URLs are not available from my computer.

[kty0mka]

I haven't seen any activity on the test host for the past few weeks, so I thought you had done some testing already and decided to shut down the hosts. I brought them up again.

Anyway, I ran litmus tests with the latest development branch. All tests pass. Other clients still work with DMSF, as well as Cyberduck, even though custom namespaces are no longer used. Therefore, I already doubt that I can consider this a bug on the DMSF side.

UPD: Nextcloud still cannot connect to the DMSF as a client.

[smoebody]

Hi, I am watching this issue because i experienced problems connecting with gnome's nautilus filemanager as well. With the provided credentials above i was able to connect to successfully.
I can give a thumbsup on the changes happened here.

Thanks for the good work. 👍

[picman]

@kty0mka I haven't figured out how to configure the connection from Nextcloud to Redmine. I see there an existing connection but can't edit it or add a new one. I suppose you have configured it with a right url and user with right permissions. All in all what I'd need is production.log. So try to connect to Redmine from Nextcloud and post what is in the production.log.

[kty0mka]

Here's everything that is shown in the Redmine log when trying to connect from Nextcloud:

App 132 output: I, [2021-02-08T14:36:40.885625 #132]  INFO -- : Started PROPFIND "/dmsf/webdav/test-project/" for 35.210.35.195 at 2021-02-08 14:36:40 +0000
App 132 output: I, [2021-02-08T14:36:40.886968 #132]  INFO -- : Processing WebDAV request: /dmsf/webdav/test-project/ (for 35.210.35.195 at 2021-02-08 14:36:40 UTC) [PROPFIND]
App 132 output: I, [2021-02-08T14:36:40.891099 #132]  INFO -- : Completed in: 5 ms | 401 [https://pm.midl.com.ua/dmsf/webdav/test-project/]
App 132 output: I, [2021-02-08T14:36:40.894637 #132]  INFO -- : Started PROPFIND "/dmsf/webdav/test-project/" for 35.210.35.195 at 2021-02-08 14:36:40 +0000
App 132 output: I, [2021-02-08T14:36:40.895084 #132]  INFO -- : Processing WebDAV request: /dmsf/webdav/test-project/ (for 35.210.35.195 at 2021-02-08 14:36:40 UTC) [PROPFIND]
App 132 output: I, [2021-02-08T14:36:40.924526 #132]  INFO -- : Completed in: 29 ms | 207 [https://pm.midl.com.ua/dmsf/webdav/test-project/]

The user whose credentials I gave (admin:20Dmsf21) has all administrator rights (actually, this is the only user on the test host). Direct link to the external storages settings: https://nc2.midl.com.ua/settings/user/externalstorages
Here's how to add a new WebDav connection:

[picman]

I discovered another problem related to Cadaver and fixed it. It might be related to Nextcloud too. Could you update your Redmine and re-test it, please?

[kty0mka]

Wow! It finally works with Nextcloud!
@picman thank you so much!

[picman]

Great!