Spatial file manager

[probonopd]

The current https://github.com/helloSystem/Filer/ implementation is not spatial.

Spatial Interfaces are explained by John Siracusa at
https://arstechnica.com/gadgets/2003/04/finder/2/

It would be nice if we could have a spacial file manager.

[probonopd]

Also see https://www.asktog.com/columns/034OSX-FirstLook.html

[probonopd]

Possibly store the metadata for this using extended attributes?

[ghost]

Were you thinking that Filer should be extended to have a spatial mode or having an alternative to Filer that is spatial? I don't think there's any existing project in Qt (Gnome 2 had spatial mode at one time - we don't want that)

[probonopd]

Either way. One thing I don't like about Filer is that it still draws in some glib based components, and I was considering alternatives including DFilemanager but I could not get it to compile yet: helloSystem/Filer#23

[ghost]

I have started extending Filer for spatial mode support - helloSystem/Filer#35

I agree it would be nice not to have the glib components at all, but I think the current Filer is probably the best option still and for pragmatism it might be worth sticking with it and hope that eventually it can be refactored (well, libfm-qt)? I got DFilemanager to compile but there are more problems there.

[ghost]

Extended attributes looks viable - this is the next step in extending Filer. As a proof-of-concept from the command line:

HelloChris% setextattr user window.height 100 Documents
HelloChris% getextattr user window.height Documents 
Documents       100

See https://www.freebsd.org/cgi/man.cgi?setextattr

[ghost]

I had some spare time (I'm on 'holiday' this week :) ) so I went ahead with the extended attribute implementation for window position and size (modified the same PR).

This implements spatial folder browsing, so obviously there's an outstanding task for storing icon position as well but I think this is good to start with?

We might want to hide the sidebar and navigation controls when in spatial mode to provide a similar experience to Haiku's tracker or classic Mac finder. OK, I had a bit more time and sorted this out too :)

It would also be nice if the 'maximise' window button behaviour would do the same as tracker in this case (resize to the minimum size to fit all items without scrollbars). That's a different topic though, I guess!

[probonopd]

Wow @moochris this is awesome! Thank you very much.

Let's think about a few more aspects of spatial navigation Maybe we should think about creating a spec for the metainformation needed to make a truly spatial file manager first, ideally in a way so that any file manager could implement the spec in a compatible way, e.g.,

• Window size in pixels at normal (no-HiDPI) resolution
• Window position on screen in a coordinate system
• Display mode (icons view, list view, Miller columns,...) and a definition of fallbacks (e.g., if Miller columns were set but the file manager does not have them, then use list view)
• Icon size for the window
• How icons are positioned (automatic/by type/sorted/manual...) and sort order
• Icon positions in a coordinate system for each icon
• Window background wallpaper (only in icon view)
• Custom icons for files and folders pasted by the user
• Tags/colors of files
• ...
  As for storing the information, we should think about consider
• Extended Attributes (advantage: no additional files cluttered around for storing this information)
• One hidden file per directory (advantage: survives archiving and copy operations even over the network, etc.)
• Possibly a combination of the two or something else.

We should investigate how other spatial file managers are doing this.

[probonopd]

More food for thought from reviewer John Siracusa and Classic Mac UX auhtority Bruce "Tog" Tognazzini:

• https://arstechnica.com/gadgets/2003/04/finder/2/
• https://www.asktog.com/columns/034OSX-FirstLook.html

For me, the possibility to open new windows for each folder is just one (and probably the most controversial - hence Apple made it optional in Mac OS X) aspect. Other aspects should also be considered, such as remembering for each location in which mode (list/icons,...) it is displayed, where each icon is (in icon view mode), how things are sorted (in list view mode).

[ghost]

Yep I think a spec would be a good idea.

I started on a kind of structure for the names of the extended attributes in the initial implementation, but it probably won't scale to include the other fields.

I had thought of a few similar to what you have listed, but also another nice thing might be to reuse Haiku's HVIF format to store icons in extended attributes? (I'm sure you're aware of this, but https://www.haiku-os.org/articles/2006-11-13_why_haiku_vector_icons_are_so_small/ link anyway 😄 )

And yes storage is definitely a consideration - e.g. it would be nice if Nextcloud (or Dropbox or whatever) could synchronise the metadata. I quite like the extended attributes (seem to work well in this first attempt) but I guess if they are transparent to the user, then some other file based mechanism like dot files would be preferable.

[probonopd]

I quite like the extended attributes (seem to work well in this first attempt) but I guess if they are transparent to the user, then some other file based mechanism like dot files would be preferable.

Let's have a look at where other systems are storing which kind of information; Haiku and the Mac are coming to mind immediately.

Speaking of Haiku:

Haiku Vector Icons

Yes, I know about them but since many applications don't have them I think we needd to support png and svg in any case.

[ghost]

An interesting post (albeit from a while ago now) on extended attributes:

https://www.lesbonscomptes.com/pages/extattrs.html

Hopefully some of those pitfalls have been solved in the meantime. I note that it mentions ROX using them, so worth adding that to the list of projects to investigate.

[probonopd]

Possibly we need to consider a combination of extended attributes (for performance) and files (for robustness) (I think this is what Haiku is doing). In any case, the spatial information should travel with the folder, so that e.g., if someone accesses the same folder over the network the spatial information should be available to that user, too.

[probonopd]

Need to keep in mind use cases like this one:

Here, whoever created that disk image

• Set the sidebar to disabled for that window
• Set a custom window wallpaper
• Set the size and location of the icons

...and all of it traveled along inside the disk image.

[ghost]

Possibly we need to consider a combination of extended attributes (for performance) and files (for robustness) (I think this is what Haiku is doing). In any case, the spatial information should travel with the folder, so that e.g., if someone accesses the same folder over the network the spatial information should be available to that user, too.

I think Haiku only uses attributes for Tracker - it's these here:
https://git.haiku-os.org/haiku/tree/src/kits/tracker/Attributes.h

e.g. _trk/windframe

But yes I agree these things should be portable. For Haiku, for example, they modified zip and unzip to preserve attributes. We should look into whether other formats can preseve extended attributes already - e.g. tar already does.

[probonopd]

Uh. I am not a huge fan of patching half the system. Maybe better use dot files in each directory. E.g, .extattr? When Filer sees such a file, it could read it and apply the extended attributes in the filesystem. Something along those lines (not fully thought through yet).

[ghost]

Yeah, OK, I think you're right.

We'll need a spec - doesn't have to be fully fleshed out, but if we can get a standard for naming to start with that would be good for modifying the spatial Filer PR (as a prototype).

All files/folders will need a dot file I think, rather than a just one for the directory or you have all sorts of management of what then becomes like a flat database.

e.g.:

/home/chris/projects
/home/chris/.extattr_projects
/home/chris/projects/document.odt
/home/chris/projects/.extattr_document.odt

The dot files can contain key value pairs for the attributes, although I'm not sure about the best format - possibly just key=value one per line?

It doesn't need to apply the extended attributes at all really, as the dot files will need to be kept up to date - may as well just use the dot files?

[ghost]

OK, I see the problem with that - the directory needs to contain its own dotfile or you'll lose it on archiving.

So:

/home/chris/projects
/home/chris/projects/.extattr
/home/chris/projects/document.odt
/home/chris/projects/.extattr_document.odt

[probonopd]

Instead of having

/home/chris/projects/.extattr_document.odt

we could have that information in

/home/chris/projects/.extattr

as well. Only directories would need those files, the files could also describe the documents inside. When Filer sees the directory, it would apply its extattrs to the files inside.

[ghost]

It won't scale for very large numbers of files in a directory I think

[probonopd]

People are gonna hate this if we have such files cluttered around in big numbers.

Actually, let's have a look at how .DS_Store files work. They are created by the Finder in some circumstances and seem to contain the metadata we are discussing here. The format is proprietary, but maybe we can deduce some of the "theory of operation".

[probonopd]

• https://en.wikipedia.org/wiki/.DS_Store
• https://0day.work/parsing-the-ds_store-file-format/
• https://wiki.mozilla.org/DS_Store_File_Format
• http://search.cpan.org/~wiml/Mac-Finder-DSStore/DSStoreFormat.pod
• https://digi.ninja/projects/fdb.php

[ghost]

Looks a bit complicated!

We might want to consider something like sqlite?

[probonopd]

Our motto: Aim for making everything 10x as simple! But it cannot hurt to understand the "prior art" concepts first.

We might want to consider something like sqlite?

Sure. But we need to think about what to put where, so that it can travel along, e.g., in zip files and such. And we need to think about performance. The Wikipedia article has some good hints. (To interpret them liberally: By default do not store such files on network shares, and only save those files after a window was open for a certin number of seconds and the user changed its appearance - something like that)

[ghost]

Sure. But we need to think about what to put where, so that it can travel along, e.g., in zip files and such. And we need to think about performance. The Wikipedia article has some good hints.

I'm suggesting that the single .extattr file in the directory is an sqlite database, so it would persist with the zipped directory. But yes, not sure about performance.

[probonopd]

Yes, that would be worth a try.

[probonopd]

Check this out!

https://ds-store.readthedocs.io/en/latest/

Manipulate Finder .DS_Store files from Python.
(...)
This module allows programmatic access to and construction of .DS_Store files directly from Python, with no Mac OS X specific code involved.

Wouldn't it rock if we could use the same format even, given that someone seems to already have written the implementation?

Reading through the comments in the code of https://github.com/al45tair/ds_store/blob/master/ds_store/store.py alone is versy interesting, it gives some hints such as:

Works by manipulating the file
on the disk---so this code will work with .DS_Store files for very
large directories.

I'd say let's play around with https://github.com/al45tair/ds_store a bit and in case we decide against using it we know at least why ;-)

[antranigv]

Please let’s not use DS_Store, as a former macOS user it was a PITA for me, specially when doing development. This means it will be very useful for the daily users, but very bad experience for the power users and developers.

…

On Sat, Feb 20, 2021, at 3:03 PM, probonopd wrote: Check this out! https://ds-store.readthedocs.io/en/latest/ Manipulate Finder `.DS_Store` files from Python. Wouldn't it rock if we could use the same format even, given that someone seems to already have written the implementation? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKXL3P5TT6XEZ5X37FO4LTS76JIVANCNFSM4VQPWWMQ>.

[probonopd]

One more thought, since those files will be mainly written automatically (rather than by hand), would using a binary format optimized for speed be an advantage? I suspect there is a reason why binary plists exist.

[ghost]

Yes, I would expect it would be faster. If it's fast enough, we could just write to the dot file at the same time as extended attributes

[probonopd]

binary format optimized for speed

Looks like MessagePack is such a protocol.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

https://github.com/romixlab/qmsgpack

There is packers and unpackers for QColor, QTime, QDate, QDateTime, QPoint, QSize, QRect. Also you can create your own packer/unpacker methods for Qt or your own types.

Not sure whether using something like this would be premature optimization for our use case. Maybe we shoud start with something like JSON and check what the performance is.

[ghost]

I think in parallel we should start a new filer implementation from scratch, with only Qt dependencies (perhaps leveraging FreeBSD or ZFS features along the way).

I think adapting existing projects will get us so far, but never to exactly what we want.

The spatial/metadata spec is of course useful for doing this.

[probonopd]

I think in parallel we should start a new filer implementation from scratch, with only Qt dependencies (perhaps leveraging FreeBSD or ZFS features along the way).

All the power to you, but I assume it'd be a massive undertaking.

Maybe we could leverage one of the existing ones that have been mentioned, like DFileManager or https://github.com/cyberos/cyber-fm which in turn is based on https://invent.kde.org/maui/index-fm. cyber-fm and index-fm are written in Qml though, I am not sure yet whether this is a great match for a file manager but I guess it is worth some experimentation.

[ghost]

Yes, you might be right 🙂

Ok, certainly some worth investigating there.

[probonopd]

Let's take on manageable changes. Like helloSystem/Filer#9 - things like that woud already make a huge difference and can be accomplished in a small fraction of the time compared to a whole new file manager.

[ghost]

Not sure whether using something like this would be premature optimization for our use case. Maybe we shoud start with something like JSON and check what the performance is.

Interestingly, I was just looking at an example for JSON support in Qt here:
https://doc.qt.io/qt-5/qtcore-serialization-savegame-example.html

"With QJsonDocument, you also have the ability to serialize a document in a CBOR format, which is great if you don't want the save file to be readable, or if you need to keep the file size down"

I wonder if saving to CBOR is also quicker?

[probonopd]

Interesting, I had never heard of it. Maybe we should try JSON and then see whether additional optimization is needed.

[probonopd]

Yes, or an option in the preferences to turn off the .DirInfo files?

There needs to be a way to disable the creation of .DirInfo files for a certain volume (mount point). E.g., think of a network share where the owner wants no .DirInfo files, regardless of what various users may have set up in Filer.

[ghost]

There needs to be a way to disable the creation of .DirInfo files for a certain volume (mount point). E.g., think of a network share where the owner wants no .DirInfo files, regardless of what various users may have set up in Filer.

OK - do you think we need to walk up each level in the path and check for a .DisableDirInfo file? Just thinking they might not necessarily have them at the mount point.

[probonopd]

I'd say who owns the volume root directory decides ;-) so let's not make it overly complex and slow.

[grahamperrin]

On the origins of DS_store

… I was the technical lead for the Mac OS X Finder …

[probonopd]

When using Spatial mode, I urgently feel the need to have a "Go to... Command+Shift+G" dialog

[grahamperrin]

The wish to enter a location is not limited to spatial mode.

For a non-swapped keyboard layout, Control-L will be ideal.

[probonopd]

The wish to enter a location is not limited to spatial mode.

Indeed.

Control-L

L? Why L? Command+Shift+G is it... in line with #58

https://osxdaily.com/2011/08/31/go-to-folder-useful-mac-os-x-keyboard-shortcut/

[probonopd]

@moochris do you think you can hook this in?

Go -> Go To Folder... Ctrl+Shift+G

    QWidget *centralWidget = new QWidget;

    QVBoxLayout *layout = new QVBoxLayout;

    label = new QLabel;
    label->setText("Go to the folder:");
    layout->addWidget(label);

    lineEdit = new QLineEdit;
    lineEdit->setMinimumWidth(480);
    layout->addWidget(lineEdit);

    buttonBox = new QDialogButtonBox;
    buttonBox->addButton(tr("Go"), QDialogButtonBox::AcceptRole);
    buttonBox->addButton(tr("Cancel"),QDialogButtonBox::RejectRole);

    // TODO: Implement functionality

    layout->addWidget(buttonBox);

    centralWidget->setLayout(layout);
    setCentralWidget(centralWidget);

    setWindowTitle(tr("Go To Folder"));
    lineEdit->setFocus();

    completer = new QCompleter(this);
    completer->setCompletionMode(QCompleter::InlineCompletion);
    QFileSystemModel *fsModel = new QFileSystemModel(completer);
    fsModel->setFilter(QDir::Dirs|QDir::Drives|QDir::NoDotAndDotDot|QDir::AllDirs); // Only directories, no files
    completer->setModel(fsModel);
    fsModel->setRootPath(QString());
    lineEdit->setCompleter(completer);

[ghost]

Nice - yep, leave it with me and I'll hook it up.

I'm moving home this weekend so can't guarantee I'll get to it quickly.

[ghost]

@probonopd PR for it here: helloSystem/Filer#56

[KDGNOR]

some text about adding colors tags...

here : helloSystem/Filer#32

[probonopd]

Spatial file manager including storing icon position is being implemented in https://github.com/probonopd/Filer/.