Skip to content

Commit

Permalink
Merge branch 'tools-configs'
Browse files Browse the repository at this point in the history
  • Loading branch information
Sewer56 committed Aug 25, 2024
2 parents 9af7439 + 292a1b5 commit f85bf7f
Show file tree
Hide file tree
Showing 13 changed files with 1,158 additions and 402 deletions.
165 changes: 160 additions & 5 deletions docs/Server/Packaging/Package-Metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,10 @@ These fields are usually only found when [PackageType](#packagetype) == `Mod`:

These fields are usually only found when [PackageType](#packagetype) == `Tool`:

| Type | Name | Description |
| ------ | -------------- | ---------------------------------------------------- |
| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. |
| Type | Name | Description |
| ------------ | ---------------------------- | ---------------------------------------------------- |
| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. |
| ConfigFile[] | [ConfigFiles](#config-files) | List of configuration files synced with Reloaded-II. |

## Implicit Fields

Expand Down Expand Up @@ -216,7 +217,6 @@ automatically generated package. Specifically by:
- Appending the `architecture` part to the [Id](#id) field.
- Updating the [Targets](#targets) field to include the correct architecture (if needed).


!!! warning "***DO NOT*** use architectures be used in [dependency references](#dependency-info)."

-`reloaded3.api.windows.vfs.s56`
Expand Down Expand Up @@ -632,6 +632,157 @@ By default this value is `false` for code mods and `true` for asset mods. Howeve
the mods which read the contents of asset mods may choose to ignore the unload request
if they themselves don't support it. (These mods should log a warning to console if they do so.)

## Config Files

!!! info "For `Tools`, this allows the [ingesting of configuration files][ingest-config] from an external location."

Example section:

```toml
# A file inside the package directory
# If the tool writes config files to its own folders
[[ConfigFiles]]
Id = 0
Type = "File"
Description = "Main configuration file"
[[ConfigFiles.Paths]]
OS = "any"
Path = "{PackageDir}/bin/config/config.json"

# Example of using 'by extension' with a base directory
[[ConfigFiles]]
Id = 1
Type = "ByExtension"
Description = "All JSON files in the config directory"
[[ConfigFiles.Paths]]
OS = "any"
Path = "{PackageDir}/bin/config"
Extension = "json"
IncludeSubfolders = false

# Example of using AppData across different OS
[[ConfigFiles]]
Id = 2
Type = "File"
Description = "User settings in AppData folder matching across OSes"
[[ConfigFiles.Paths]]
OS = "any"
Path = "{AppData}/ToolName/settings.json"
# Windows: C:\Users\{Username}\AppData\Roaming\ToolName\settings.json
# Linux: /home/{Username}/.config/ToolName/settings.json
# macOS: /Users/{Username}/Library/Application Support/ToolName/settings.json

# Example of a folder with OS-specific locations
[[ConfigFiles]]
Id = 3
Type = "Folder"
Description = "OS-specific data folder"
[[ConfigFiles.Paths]]
OS = "win"
Path = "{LocalAppData}/ToolName/Data"
[[ConfigFiles.Paths]]
OS = "linux"
Path = "{XDG_DATA_HOME}/ToolName"
[[ConfigFiles.Paths]]
OS = "macos"
Path = "{Library}/Application Support/ToolName"
```

When a loadout is loaded or the relevant `Tool` started by Reloaded3 stops running, updated
configs are `ingested` (integrated) into the loadout.

For more details, see [Tools as Packages][tools-as-packages].

### ConfigFile

| Type | Name | Description |
| --------------- | ----------- | --------------------------------------------------------------- |
| u8 | Id | [Required] Unique identifier. Between 0-255. |
| string | Type | Type of the config entry: `File`, `Folder`, or `ByExtension` |
| [Path](#path)[] | Paths | Array of OS-specific paths for the config file or folder |
| string | Description | A brief description of the configuration file or group of files |

!!! info "The `Unique Identifier` (ID) is used to reference paths from external places."

In a way where configuration files can be transferred between platforms, for example Linux <=> Windows.
An example of this is [Unpacked Loadouts][external-config-paths].

#### Path

| Type | Name | Description |
| ------ | ----------------- | -------------------------------------------------------- |
| string | OS | Target OS: `any`, `win`, `linux` or `macos`. |
| string | Path | The path or to the file/folder, can include placeholders |
| string | Extension | Name of extension when `ByExtension` is used. |
| bool | IncludeSubfolders | Whether to recurse folders when `ByExtension` is used. |

The OS field names are based on the [Backend Names][native-backend].

## Path Placeholders

!!! info "The `Path` field allows for some placeholders."

These are based on the `Windows` folder names, with equivalent `Linux` and `macOS` folders.

| Placeholder | Windows | Linux | macOS |
| ---------------- | ------------------------- | ----------------------------------------- | ----------------------------------- |
| `{PackageDir}` | Package directory | Package directory | Package directory |
| `{AppData}` | `%APPDATA%` | `$XDG_CONFIG_HOME` or `$HOME/.config` | `$HOME/Library/Application Support` |
| `{LocalAppData}` | `%LOCALAPPDATA%` | `$XDG_DATA_HOME` or `$HOME/.local/share` | `$HOME/Library/Application Support` |
| `{ProgramData}` | `%PROGRAMDATA%` | `/usr/local/share` | `/Library/Application Support` |
| `{Temp}` | `%TEMP%` | `$XDG_RUNTIME_DIR` or `/tmp` | `$TMPDIR` |
| `{Documents}` | `%USERPROFILE%\Documents` | `$XDG_DOCUMENTS_DIR` or `$HOME/Documents` | `$HOME/Documents` |
| `{Home}` | `%USERPROFILE%` | `$HOME` | `$HOME` |

These are typically used for the following:

- `{PackageDir}`: The directory containing the current package.
- `{AppData}`: User-specific (roaming) application data folder
- `{LocalAppData}`: User-specific local application data folder
- `{ProgramData}`: Application data for all users
- `{Temp}`: Temporary folder
- `{Documents}`: User's Documents folder

### Linux Specific Placeholders

On `Linux`, the [XDG Path Spec][xdg-path-spec] is also supported:

| Placeholder | Linux Location |
| --------------------- | ----------------------------------------- |
| `{XDG_DATA_HOME}` | `$XDG_DATA_HOME` or `$HOME/.local/share` |
| `{XDG_CONFIG_HOME}` | `$XDG_CONFIG_HOME` or `$HOME/.config` |
| `{XDG_CACHE_HOME}` | `$XDG_CACHE_HOME` or `$HOME/.cache` |
| `{XDG_DOCUMENTS_DIR}` | `$XDG_DOCUMENTS_DIR` or `$HOME/Documents` |

Explanation of Linux Folders:

- `{XDG_DATA_HOME}`: Directory for user-specific data files, typically used for non-configuration files like runtime data. Defaults to $HOME/.local/share.
- `{XDG_CONFIG_HOME}`: Directory for user-specific configuration files. Defaults to $HOME/.config.
- `{XDG_CACHE_HOME}`: Directory for user-specific non-essential data files, such as cache. Defaults to $HOME/.cache.
- `{XDG_DOCUMENTS_DIR}`: Directory for user-specific documents, usually mapped to $HOME/Documents.

### macOS Specific Placeholders

On `macOS`, the following placeholders map to directories commonly used by macOS-specific applications and are not part of the common cross-platform placeholders:

| Placeholder | macOS Location |
| ---------------------- | --------------------------- |
| `{Library}` | `$HOME/Library` |
| `{LibraryCaches}` | `$HOME/Library/Caches` |
| `{LibraryPreferences}` | `$HOME/Library/Preferences` |
| `{LibraryLogs}` | `$HOME/Library/Logs` |
| `{LibraryContainers}` | `$HOME/Library/Containers` |
| `{SystemLibrary}` | `/Library` |

Explanation of macOS Folders:

- **`{Library}`**: The root directory for user-specific application support files, caches, preferences, etc. Located at `$HOME/Library`.
- **`{LibraryCaches}`**: Directory for user-specific cache files. Located at `$HOME/Library/Caches`.
- **`{LibraryPreferences}`**: Directory for user-specific application preference files. Located at `$HOME/Library/Preferences`.
- **`{LibraryLogs}`**: Directory for user-specific application log files. Located at `$HOME/Library/Logs`.
- **`{LibraryContainers}`**: Directory for containerized app data, used primarily for sandboxed apps. Located at `$HOME/Library/Containers`.
- **`{SystemLibrary}`**: The system-wide equivalent of the user’s Library folder, containing system-level application support files. Located at `/Library`.

<!-- Links -->
[game-metadata-id]: ../Storage/Games/About.md#id
[backend]: ../../Loader/Backends/About.md
Expand Down Expand Up @@ -660,4 +811,8 @@ if they themselves don't support it. (These mods should log a warning to console
[platforms]: ../../Loader/Backends/About.md#platforms
[architecture]: ../../Loader/Backends/About.md#architectures
[tasks]: ./Tasks.md
[package-structure]: ./About.md#package-structure
[package-structure]: ./About.md#package-structure
[ingest-config]: ../Storage/Locations.md#package-config-handling
[tools-as-packages]: ./Tools-As-Packages.md#chosen-approach
[xdg-path-spec]: https://specifications.freedesktop.org/basedir-spec/latest/index.html
[external-config-paths]: ../Storage/Loadouts/File-Format/Unpacked.md#external-config-pathsbin
14 changes: 7 additions & 7 deletions docs/Server/Packaging/Tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,9 @@ Name = "Launch Game"
GroupNames = ["GOG"]
Description = "Launches the game."
Path = {
Default = "", # Cross Platform Binary
Windows = "Game.exe",
Linux = "Game.sh",
MacOS = "Game.app"
win+x64-any = "Game.exe",
linux+x64-any = "Game.elf",
macos+x64-any = "Game.app"
}
IsPrimary = true
IsHidden = false
Expand Down Expand Up @@ -124,6 +123,10 @@ The `Path` field specifies the relative path to the executable file or the URL t
- If the task is declared for a game, the `Path` is relative to the folder that contains the main executable of the game.
- The one marked [IsPrimary](#isprimary).

// TODO:
- [Platform][platforms] only: `win`, `linux`. When package has all architecture binaries.
- [Platform][platforms]+[Arch][architecture]: `win+x64-v3`, `linux+x64-v3`. When package has only one specific architecture.

**Examples:**

1. Cross-platform executable:
Expand Down Expand Up @@ -218,9 +221,6 @@ The available placeholders are:
If the task is sourced from a package, the following variables are also available:

- `{PackageDir}`: The absolute path to the base folder of the package.
- `{PackageConfigDir}`: The absolute path to the config folder for the package.
- [This is the `Package Configs` (`PackageConfigs/{packageId}`) folder.][items-to-store]
- This folder allows for
- `{PackageUserCacheDir}`: The absolute path to the config folder for the package.
- [This is the `Package Cache Files (User)` (`Cache/{packageId}`) folder.][items-to-store]
- `{PackageMachineCacheDir}`: The absolute path to the config folder for the package.
Expand Down
90 changes: 90 additions & 0 deletions docs/Server/Packaging/Tools-As-Packages.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
!!! info "This page details the caveats involved in shipping 3rd party tools are packages."

Think of it as an ADR.

## Configurations

!!! info "External tools don't know about Reloaded's Loadout system."

We must be able to observe changes to configurations and *integrate* them into the loadout data.

Doing so is relatively tricky:

- The user may want to run the tool outside of Reloaded3.
- The tool may write its configuration files to arbitrary locations.
- The user may want to have global configurations for tools.
- And also possibly per-version configurations.
- Some configurations, e.g. file paths are not portable across PCs.

### Configuration Location

!!! info "Where are the software configurations stored?"

There are multiple ways to work here:

- **Reloaded 3 Manages the Tool's Configuration**: Reloaded3 monitors where the tool writes its configurations and automatically syncs them.
- 🛈 Requires regex filters & extra fields to monitor config locations.

- **Request Specific Config Location**: Reloaded3 requests a specific location to write config via commandline parameter.
- ❌ Cannot ingest changes made by running tool outside of Reloaded3.

These are somewhat opposites of each other.

For example, R3 managing the tool's configuration would mean that any tool can be shipped as a package
without any source changes. However as a consequence, that means it's more likely to sync unnecessary
(machine specific) data, and thus increase my cloud storage costs. etc.

There's also the question of whether you want the configurations to 'leak' outside of Reloaded3
as a user. I think for most users, that may be 'yes' as most users would only install a tool once,
without separate per-loadout configurations.

### Synchronization Times

!!! info "When do we check for configuration changes?"

- **Process Sync**: The tool is run within Reloaded3 and its configurations are automatically synced.
- ✅ Fairly efficient.
- ❌ This means users running the tool outside of Reloaded3 will have suboptimal experience.
- ❌ Fails if tool launches sub-process.

- **File System Watcher Sync**: The FileSystem is periodically checked for changes
- ✅ Always works.
- ❌ May produce multiple config change event over a single run of application.

- **Sync on Run**: Let the user choose between local and loadout config if they don't match at startup.
- ✅ Works in and out of process.
- ✅ Does not unnecessarily ingest.
- ❌ Increases startup latency.

### Handling Multiple Configuration Files

!!! info "How do we handle multiple configuration files?"

The [Loadout][loadout] format is written primarily under the assumption that 1 mod == 1 config,
because that is the expected standard for mods.

However for `Tools`, it's necessary to support multiple configuration files, as we're not in control
of how they are built.

That proposes 2 approaches:

- **Merge Configurations**: Merge all configurations into a single file.
- ✅ Simple to implement.
- ❌ Inefficient on disk space (uncompressed).

- **Ingest Individual Files**: Store each file separately.
- ✅ Efficient on disk space (uncompressed).
- ⚠ Requires extensions to event format.

### Chosen Approach

The following approaches have been chosen:

- **Configuration Location**: Reloaded3 manages the tool's configuration.
- **Synchronization Times**: Process Sync ***AND*** Sync on Run.
- The user is prompted to choose between local and loadout config if they mismatch.
- **Handling Multiple Configuration Files**: Ingest Individual Files.

The `Ingest` operation will happen upon loading of a tool.

[loadout]: ../Storage/Loadouts/About.md
5 changes: 3 additions & 2 deletions docs/Server/Storage/Loadouts/About.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,13 @@ be written to each file as we replay the events, then truncate everything after
In the event of a crash, corrupt loadout or power loss, we have 3 sources of truth:

- [Packed Loadout][packed-loadout]
- [Unpacked Loadout][unpacked-loadout] (if Exists)
- [Unpacked Loadout][unpacked-loadout]
- [Snapshot][snapshot]

We restore the loadout using the newest of these sources.<br/>
This is determined by the [event count][event-count-header] field.

If the newest is a [packed][packed-loadout] or loadout,
If the newest is a [packed][packed-loadout] or loadout,
we simply replace the other 2 sources and we're done.

If the latest source however is a [snapshot][snapshot], we use that as a fallback and diff it against
Expand All @@ -161,6 +161,7 @@ Loadouts are stored in the `Loadouts` folder.
```
.
├── 7f2cc8b7d9f1e3a5
│ └── ... unpacked files
└── database.mdb
├── 7f2cc8b7d9f1e3a5.nx
└── 7f2cc8b7d9f1e3a5.snapshot.bin
Expand Down
Loading

0 comments on commit f85bf7f

Please sign in to comment.