Merge branch 'add-vfs'

Reloaded-Project · Apr 28, 2024 · 60c8e56 · 60c8e56
2 parents ff49f20 + ce83e59
commit 60c8e56
Show file tree

Hide file tree

Showing 11 changed files with 1,153 additions and 180 deletions.
diff --git a/docs/Mods/Essentials/Virtual-FileSystem/About.md b/docs/Mods/Essentials/Virtual-FileSystem/About.md
@@ -0,0 +1,56 @@
+# About the Reloaded Virtual FileSystem
+
+The Reloaded Virtual File System (VFS) is an invisible helper that sits between your games and the
+files they use. It allows your games to 'see' and open files that aren't really 'there', keeping
+your game folder unmodified.
+
+```mermaid
+flowchart LR
+
+    p[Game] -- Open File --> vfs[Reloaded VFS]
+    vfs -- Open Different File --> of[Operating System]
+```
+
+The VFS sits in the middle and does some magic 😇.
+
+```mermaid
+classDiagram
+
+    class `Mod Folder`
+    `Mod Folder` : data3.pak
+
+    class `Mod 2 Folder`
+    `Mod 2 Folder` : data4.pak
+
+    class `Real Game Folder`
+    `Real Game Folder` : data1.pak
+    `Real Game Folder` : data2.pak
+    `Real Game Folder` : game.exe
+
+    class `Virtual Game Folder [What Game Sees]`
+    `Virtual Game Folder [What Game Sees]` : data1.pak
+    `Virtual Game Folder [What Game Sees]` : data2.pak
+    `Virtual Game Folder [What Game Sees]` : data3.pak
+    `Virtual Game Folder [What Game Sees]` : data4.pak
+    `Virtual Game Folder [What Game Sees]` : game.exe
+
+    `Mod Folder` --|> `Virtual Game Folder [What Game Sees]`
+    `Mod 2 Folder` --|> `Virtual Game Folder [What Game Sees]`
+    `Real Game Folder` --|> `Virtual Game Folder [What Game Sees]`
+```
+
+## Characteristics
+
+Compared to Windows symlinks/hardlinks:
+
+- Links are only visible to the current application.
+- Write access to game folder is not needed. Can even link new content into read-only folders.
+- Administrator rights are not needed.
+- Can overlay multiple directories on top of the destination.
+
+And with the following benefits:
+
+- Easy to use API for programmers.
+- Practically zero overhead.
+- Can add/remove and remap files on the fly (without making changes on disk).
+- Supports Wine on Linux.
diff --git a/docs/Mods/Essentials/Virtual-FileSystem/Behaviours-Limitations.md b/docs/Mods/Essentials/Virtual-FileSystem/Behaviours-Limitations.md
@@ -0,0 +1,47 @@
+# Behaviours & Limitations
+
+!!! info "The Reloaded VFS is only intended to be used for '*well defined*' functionality."
+
+For now, this means the VFS is 'read only'. Write operations, such as creating a file are unaffected.
+
+If a game wants to write a new file (such as a savefile), no action will be taken and the file will
+be written to the game folder. If a native DLL plugin wants to write a config file, it will write it
+to the game folder, as normal.
+
+## Warnings
+
+!!! warning "Proceed with care if any of the following applies"
+
+- If your game's modding tools operate on a modified game directory (e.g. Skyrim xEdit),
+  using VFS is not recommended as new files might be written to the game folder.
+
+- Do not use VFS to redirect files deleted and then recreated by games; ***you will lose the files from inside your mod***.
+
+## Error Cases
+
+!!! error "Using this VFS is not appropriate for your game if any of the following is true"
+
+- This VFS does not handle child processes. Do not use VFS for games that can run
+  external tools with virtualized files.
+    - Will be implemented in the future if mods ever will end up opening external tools.
+    - However that workflow is not recommended... (e.g. might be problematic for Linux users)
+
+## Additional Limitations
+
+!!! note "The following limitations also exist but should not cause concern."
+
+- Reloaded VFS does not support Reparse Point Tags.
+    - However, this shouldn't cause issues with mods stored on cloud/OneDrive/etc.
+- Reloaded VFS does not return 8.3 DOS file names for virtualized files.
+
+## File Write Behaviours
+
+!!! warning "What happens when you try editing files in a 'read-only' VFS?"
+
+| Description                        | Action Performed                                                                                              |
+| ---------------------------------- | ------------------------------------------------------------------------------------------------------------- |
+| File Deletion                      | Delete the mod file instead of the original file                                                              |
+| New File Creation                  | Create new files in the original game folder                                                                  |
+| File Editing (in Place)            | Edits the redirected file                                                                                     |
+| File Delete & Recreate (New)       | Delete the overwritten file and place the new file in game folder                                             |
+| Renaming Folders to Other Location | Either move the original folder or files in original folder and overlaid folders (depends on how API is used) |
diff --git a/docs/Mods/Essentials/Virtual-FileSystem/Implementation-Details/Hooks.md b/docs/Mods/Essentials/Virtual-FileSystem/Implementation-Details/Hooks.md
@@ -0,0 +1,137 @@
+!!! info "This currently only contains information for Windows."
+
+    Native support for other OSes will be added in the future.
+
+## Used Hooks
+
+The VFS hooks several Win32 and NT File APIs to intercept file operations. The goal is to handle every API which:
+
+- Accepts a File Path: In this case we set a new path to our redirected file.
+
+- Returns Files at Given Path: In this case we inject new files into the result.
+
+Here is a flowchart of the hooked APIs:
+
+```mermaid
+flowchart LR
+    subgraph Win32
+
+    %% Definitions
+    FindFirstFileA
+    FindFirstFileExA
+    FindFirstFileW
+    FindFirstFileExW
+    FindFirstFileExFromAppW
+    FindNextFileA
+    FindNextFileW
+
+    CreateDirectoryA
+    CreateDirectoryW
+    CreateFileA
+    CreateFileW
+    CreateFile2
+    CreateFile2FromAppW
+    CreateFileFromAppW
+    CreateDirectoryExW
+    CreateDirectoryFromAppW
+    DeleteFileA
+    DeleteFileW
+    DeleteFileFromAppW
+    GetCompressedFileSizeA
+    GetCompressedFileSizeW
+    CloseHandle
+
+    GetFileAttributesA
+    GetFileAttributesExA
+    GetFileAttributesExFromAppW
+    GetFileAttributesExW
+    GetFileAttributesW
+    SetFileAttributesA
+    SetFileAttributesFromAppW
+    SetFileAttributesW
+
+    RemoveDirectoryA
+    RemoveDirectoryFromAppW
+    RemoveDirectoryW
+
+    %%% Win32 Internal Redirects
+    FindFirstFileA --> FindFirstFileExW
+    FindFirstFileExA --> FindFirstFileExW
+    FindFirstFileExFromAppW --> FindFirstFileExW
+    FindNextFileA --> FindNextFileW
+    CreateDirectoryA --> CreateDirectoryW
+    CreateFile2FromAppW --> CreateFile2
+    CreateDirectoryFromAppW --> CreateDirectoryExW
+    CreateFileFromAppW --> CreateFile2FromAppW
+    DeleteFileFromAppW --> DeleteFileW
+    DeleteFileA --> DeleteFileW
+    GetCompressedFileSizeA --> GetCompressedFileSizeW
+    GetFileAttributesA --> GetFileAttributesW
+    GetFileAttributesExA --> GetFileAttributesExW
+    GetFileAttributesExFromAppW --> GetFileAttributesExW
+    RemoveDirectoryA --> RemoveDirectoryW
+    RemoveDirectoryFromAppW --> RemoveDirectoryW
+    SetFileAttributesFromAppW --> SetFileAttributesW
+    SetFileAttributesA --> SetFileAttributesW
+    end
+
+    subgraph NT API
+    %% Definitions
+    NtCreateFile
+    NtOpenFile
+    NtQueryDirectoryFile
+    NtQueryDirectoryFileEx
+    NtDeleteFile
+    NtQueryAttributesFile
+    NtQueryFullAttributesFile
+    NtClose
+
+    %%% Win32 -> NT API
+    FindFirstFileExW --> NtOpenFile
+    FindFirstFileExW --> NtQueryDirectoryFileEx
+    FindFirstFileW --> NtOpenFile
+    FindFirstFileW --> NtQueryDirectoryFileEx
+    FindNextFileW --> NtQueryDirectoryFileEx
+    CreateFileA --> NtCreateFile
+    CreateFileW --> NtCreateFile
+    CreateFile2 --> NtCreateFile
+    CreateDirectoryW --> NtCreateFile
+    CreateDirectoryExW --> NtOpenFile
+    CreateDirectoryExW --> NtCreateFile
+    DeleteFileW --> NtOpenFile
+    RemoveDirectoryW --> NtOpenFile
+    GetCompressedFileSizeW --> NtOpenFile
+    CloseHandle --> NtClose
+    GetFileAttributesExW --> NtQueryFullAttributesFile
+    GetFileAttributesW --> NtQueryAttributesFile
+    SetFileAttributesW --> NtOpenFile
+    end
+
+    %%% Hooks
+    subgraph Hooks
+    NtCreateFile_Hook
+    NtOpenFile_Hook
+    NtQueryDirectoryFileEx_Hook
+    NtDeleteFile_Hook
+    NtQueryAttributesFile_Hook
+    NtQueryFullAttributesFile_Hook
+    NtClose_Hook
+
+    %% NT API -> Hooks
+    NtCreateFile --> NtCreateFile_Hook
+    NtOpenFile --> NtOpenFile_Hook
+    NtQueryDirectoryFileEx --> NtQueryDirectoryFileEx_Hook
+    NtQueryDirectoryFile --> NtQueryDirectoryFile_Hook
+
+    NtDeleteFile --> NtDeleteFile_Hook
+    NtQueryAttributesFile --> NtQueryAttributesFile_Hook
+    NtQueryFullAttributesFile --> NtQueryFullAttributesFile_Hook
+    NtClose --> NtClose_Hook
+    end
+```
+
+On Windows 10+, `NtQueryDirectoryFileEx` API becomes available and `NtQueryDirectoryFile` acts as
+a wrapper around it. On Wine and earlier Windows, only `NtQueryDirectoryFile` exists.
+
+In this VFS we hook both, and detect if one recurses to the other using a semaphore. If we're
+recursing from `NtQueryDirectoryFile` to `NtQueryDirectoryFileEx`, we skip the hook code.