From f0855b93416664ca3d4c10ac9dd415773e93d0ff Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Mon, 19 Aug 2024 02:31:23 +0100 Subject: [PATCH 01/17] WIP: Config Support for Tools and LMDB Database --- docs/Server/Packaging/Package-Metadata.md | 1 + docs/Server/Packaging/Tasks.md | 2 +- docs/Server/Packaging/Tools-As-Packages.md | 90 +++++++++++++++++++ docs/Server/Storage/Loadouts/About.md | 5 +- .../Storage/Loadouts/File-Format/Snapshot.md | 3 + docs/Server/Storage/Locations.md | 36 ++++++-- mkdocs.yml | 1 + 7 files changed, 128 insertions(+), 10 deletions(-) create mode 100644 docs/Server/Packaging/Tools-As-Packages.md diff --git a/docs/Server/Packaging/Package-Metadata.md b/docs/Server/Packaging/Package-Metadata.md index 57423ca..81d6ffb 100644 --- a/docs/Server/Packaging/Package-Metadata.md +++ b/docs/Server/Packaging/Package-Metadata.md @@ -39,6 +39,7 @@ These fields are usually only found when [PackageType](#packagetype) == `Tool`: | Type | Name | Description | | ------ | -------------- | ---------------------------------------------------- | | Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. | +| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. | ## Implicit Fields diff --git a/docs/Server/Packaging/Tasks.md b/docs/Server/Packaging/Tasks.md index 2e05f28..5579f42 100644 --- a/docs/Server/Packaging/Tasks.md +++ b/docs/Server/Packaging/Tasks.md @@ -219,7 +219,7 @@ If the task is sourced from a package, the following variables are also availabl - `{PackageDir}`: The absolute path to the base folder of the package. - `{PackageConfigDir}`: The absolute path to the config folder for the package. - - [This is the `Package Configs` (`PackageConfigs/{packageId}`) folder.][items-to-store] + - [This is the `Package Configs` (`PackageConfigs/{loadoutId}/{packageId}`) folder.][items-to-store] - This folder allows for - `{PackageUserCacheDir}`: The absolute path to the config folder for the package. - [This is the `Package Cache Files (User)` (`Cache/{packageId}`) folder.][items-to-store] diff --git a/docs/Server/Packaging/Tools-As-Packages.md b/docs/Server/Packaging/Tools-As-Packages.md new file mode 100644 index 0000000..de9d4ae --- /dev/null +++ b/docs/Server/Packaging/Tools-As-Packages.md @@ -0,0 +1,90 @@ +!!! info "This page details the caveats involved in shipping 3rd party tools are packages." + +Think of it as an ADR. + +## Configurations + +!!! info "External tools don't know about Reloaded's Loadout system." + +We must be able to observe changes to configurations and *integrate* them into the loadout data. + +Doing so is relatively tricky: + +- The user may want to run the tool outside of Reloaded3. +- The tool may write its configuration files to arbitrary locations. +- The user may want to have global configurations for tools. + - And also possibly per-version configurations. +- Some configurations, e.g. file paths are not portable across PCs. + +### Configuration Location + +!!! info "Where are the software configurations stored?" + +There are multiple ways to work here: + +- **Reloaded 3 Manages the Tool's Configuration**: Reloaded3 monitors where the tool writes its configurations and automatically syncs them. + - πŸ›ˆ Requires regex filters & extra fields to monitor config locations. + +- **Request Specific Config Location**: Reloaded3 requests a specific location to write config via commandline parameter. + - ❌ Cannot ingest changes made by running tool outside of Reloaded3. + +These are somewhat opposites of each other. + +For example, R3 managing the tool's configuration would mean that any tool can be shipped as a package +without any source changes. However as a consequence, that means it's more likely to sync unnecessary +(machine specific) data, and thus increase my cloud storage costs. etc. + +There's also the question of whether you want the configurations to 'leak' outside of Reloaded3 +as a user. I think for most users, that may be 'yes' as most users would only install a tool once, +without separate per-loadout configurations. + +### Synchronization Times + +!!! info "When do we check for configuration changes?" + +- **Process Sync**: The tool is run within Reloaded3 and its configurations are automatically synced. + - βœ… Fairly efficient. + - ❌ This means users running the tool outside of Reloaded3 will have suboptimal experience. + - ❌ Fails if tool launches sub-process. + +- **File System Watcher Sync**: The FileSystem is periodically checked for changes + - βœ… Always works. + - ❌ May produce multiple config change event over a single run of application. + +- **Sync on Run**: Let the user choose between local and loadout config if they don't match at startup. + - βœ… Works in and out of process. + - βœ… Does not unnecessarily ingest. + - ❌ Increases startup latency. + +### Handling Multiple Configuration Files + +!!! info "How do we handle multiple configuration files?" + +The [Loadout][loadout] format is written primarily under the assumption that 1 mod == 1 config, +because that is the expected standard for mods. + +However for `Tools`, it's necessary to support multiple configuration files, as we're not in control +of how they are built. + +That proposes 2 approaches: + +- **Merge Configurations**: Merge all configurations into a single file. + - βœ… Simple to implement. + - ❌ Inefficient on disk space (uncompressed). + +- **Ingest Individual Files**: Store each file separately. + - βœ… Efficient on disk space (uncompressed). + - ⚠ Requires extensions to event format. + +### Chosen Approach + +The following approaches have been chosen: + +- **Configuration Location**: Reloaded3 manages the tool's configuration. +- **Synchronization Times**: Process Sync ***AND*** Sync on Run. + - The user is prompted to choose between local and loadout config if they mismatch. +- **Handling Multiple Configuration Files**: Ingest Individual Files. + +The `Ingest` operation will happen upon loading of a tool. + +[loadout]: ../Storage/Loadouts/About.md \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/About.md b/docs/Server/Storage/Loadouts/About.md index 40dada0..6391c04 100644 --- a/docs/Server/Storage/Loadouts/About.md +++ b/docs/Server/Storage/Loadouts/About.md @@ -131,13 +131,13 @@ be written to each file as we replay the events, then truncate everything after In the event of a crash, corrupt loadout or power loss, we have 3 sources of truth: - [Packed Loadout][packed-loadout] -- [Unpacked Loadout][unpacked-loadout] (if Exists) +- [Unpacked Loadout][unpacked-loadout] - [Snapshot][snapshot] We restore the loadout using the newest of these sources.
This is determined by the [event count][event-count-header] field. -If the newest is a [packed][packed-loadout] or loadout, +If the newest is a [packed][packed-loadout] or loadout, we simply replace the other 2 sources and we're done. If the latest source however is a [snapshot][snapshot], we use that as a fallback and diff it against @@ -161,6 +161,7 @@ Loadouts are stored in the `Loadouts` folder. ``` . β”œβ”€β”€ 7f2cc8b7d9f1e3a5 + β”‚ └── ... unpacked files └── database.mdb β”œβ”€β”€ 7f2cc8b7d9f1e3a5.nx └── 7f2cc8b7d9f1e3a5.snapshot.bin diff --git a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md index 91e4ecf..f8a381a 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md +++ b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md @@ -12,6 +12,9 @@ struct Snapshot { // Metadata num_events: u32, + // Loadout ID (This is the same as the loadout {UID}.) + loadout_id: String, + // Packages packages: Vec, diff --git a/docs/Server/Storage/Locations.md b/docs/Server/Storage/Locations.md index 8cd16fa..1ab5e95 100644 --- a/docs/Server/Storage/Locations.md +++ b/docs/Server/Storage/Locations.md @@ -71,12 +71,12 @@ in persistent storage. We will auto clean them ourselves. ### User -| Item | Subfolder | Description | -| ------------------------------------------------- | ---------------------------- | -------------------------------------------------------------- | -| [Loadouts](#loadouts) | `Loadouts` | Loadouts that are private to the current user. | -| [Added Games][game-metadata] | `Games/{gameId}` | And all of user's global preferences for that game. | -| Package Configs | `PackageConfigs/{packageId}` | Config/Save files for packages. ***This contains user data***. | -| [Package Cache Files](#cache-files-user-specific) | `Cache/Package/{packageId}` | Cache files that have inputs ***with user data***. | +| Item | Subfolder | Description | +| ------------------------------------------------- | ---------------------------- | --------------------------------------------------------------------- | +| [Loadouts](#loadouts) | `Loadouts` | Loadouts that are private to the current user. | +| [Added Games][game-metadata] | `Games/{gameId}` | And all of user's global preferences for that game. | +| Package Configs | Inside [Loadouts](#loadouts) | Extra details in [Package Config Handling](#package-config-handling). | +| [Package Cache Files](#cache-files-user-specific) | `Cache/Package/{packageId}` | Cache files that have inputs ***with user data***. | ## Extra Details on Stored Items @@ -144,6 +144,7 @@ Loadouts are stored in the `Loadouts` folder. ``` . β”œβ”€β”€ 7f2cc8b7d9f1e3a5 + β”‚ └── ... unpacked files └── database.mdb β”œβ”€β”€ 7f2cc8b7d9f1e3a5.nx └── 7f2cc8b7d9f1e3a5.snapshot.bin @@ -183,6 +184,26 @@ Details on how the hash cache files are is in the [Usage in Server][hash-cache-u The hash cache files are stored inside a [lmdb][fs-performance] database, named `hashes.mdb`. +### Package Config Handling + +!!! info "Package configs are stored inside [Loadouts](#loadouts)." + +However, that may raise some questions, so here they are. + +1. **Why are package configs stored in Loadouts?** + - Performance. Random file access is slow on Windows; so we want to avoid that. + - See [Loader Binary Format](https://github.com/Reloaded-Project/Reloaded-III/issues/34) for more info. + +2. **How can I modify configs in real time?** + - Configs can be adjusted in real time in two ways, **with saving** and **without saving**. + - Without saving can be done entirely in-process when inside a game. + - With saving will spawn a server process (if one does not already exist), and commit a save via server API. + +3. **How are tool configs stored?** + - Tools can specify locations of config files inside [Package Metadata][package-metadata]. + - When a loadout is loaded or tool stops running, updated configs are `ingested` (integrated) into the loadout. + - If local files are newer than remote files, the user is prompted if they wish to `integrate`. + ## Extra ### Multi-User Networked Systems @@ -247,4 +268,5 @@ Loadouts and package configs may be shared in the future on a machine level. [hashcache-docs]: ../../Common/Hash-Cache/About.md [hash-cache-usage]: ../../Common/Hash-Cache/Usage-In-Server.md [fs-performance]: ../../Research/FileSystem-Performance.md -[esoteric platforms]: ../../Code-Guidelines/Code-Guidelines.md#esoteric-and-embedded-platforms \ No newline at end of file +[esoteric platforms]: ../../Code-Guidelines/Code-Guidelines.md#esoteric-and-embedded-platforms +[tools-as-packages-approach]: ../Packaging/Tools-As-Packages.md#chosen-approach \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index ab62c78..d5e1cba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -119,6 +119,7 @@ nav: - Packaging: - About: Server/Packaging/About.md - Package Metadata: Server/Packaging/Package-Metadata.md + - Tools as Packages: Server/Packaging/Tools-As-Packages.md - Publishing Packages: Server/Packaging/Publishing-Packages.md - Tasks: Server/Packaging/Tasks.md - Workflows: From 00aa5e6d2782166cefa5078437d3d4f004c7c323 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Mon, 19 Aug 2024 02:59:28 +0100 Subject: [PATCH 02/17] Backup --- docs/Server/Packaging/Package-Metadata.md | 74 +++++++++++++++++++++-- docs/Server/Packaging/Tasks.md | 14 ++--- 2 files changed, 76 insertions(+), 12 deletions(-) diff --git a/docs/Server/Packaging/Package-Metadata.md b/docs/Server/Packaging/Package-Metadata.md index 81d6ffb..9421e0e 100644 --- a/docs/Server/Packaging/Package-Metadata.md +++ b/docs/Server/Packaging/Package-Metadata.md @@ -36,10 +36,10 @@ These fields are usually only found when [PackageType](#packagetype) == `Mod`: These fields are usually only found when [PackageType](#packagetype) == `Tool`: -| Type | Name | Description | -| ------ | -------------- | ---------------------------------------------------- | -| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. | -| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. | +| Type | Name | Description | +| ------------ | ---------------------------- | ---------------------------------------------------- | +| Task[] | [Tasks][tasks] | List of binaries that the `tool` package ships with. | +| ConfigFile[] | [ConfigFiles](#config-files) | List of configuration files synced with Reloaded-II. | ## Implicit Fields @@ -633,6 +633,68 @@ By default this value is `false` for code mods and `true` for asset mods. Howeve the mods which read the contents of asset mods may choose to ignore the unload request if they themselves don't support it. (These mods should log a warning to console if they do so.) +## Config Files + +!!! info "For `Tools`, this allows the [ingesting of configuration files][ingest-config] from an external location." + +Example section: + +```toml +[[ConfigFiles]] +Type = "File" +Path = "config.toml" +Description = "Main configuration file" + +[[ConfigFiles]] +Type = "File" +Path = "{AppData}/ToolName/settings.json" +Description = "User-specific settings" + +[[ConfigFiles]] +Type = "Folder" +Path = "presets" +Description = "Preset configurations" + +[[ConfigFiles]] +Type = "Glob" +Pattern = "*.cfg" +Description = "All CFG files in the root directory" + +[[ConfigFiles]] +Type = "Folder" +Path = "{LocalAppData}/ToolName/Logs" +Description = "Log files" +``` + +When a loadout is loaded or the relevant `Tool` started by Reloaded3 stops running, updated +configs are `ingested` (integrated) into the loadout. + +For more details, see [Tools as Packages][tools-as-packages]. + +### ConfigFile + +| Type | Name | Description | +| --------------- | ----------- | --------------------------------------------------------------- | +| string | Type | Type of the config entry: "File", "Folder", or "Glob" | +| [Path](#path)[] | Paths | Array of OS-specific paths for the config file or folder | +| string | Description | A brief description of the configuration file or group of files | + +#### Path + +| Type | Name | Description | +| ------ | ---- | -------------------------------------------------------- | +| string | OS | Target OS: "Windows", "Linux", "macOS", or "Default" | +| string | Path | The path or to the file/folder, can include placeholders | + +The `Path` field allows for some placeholders: + +- `{PackageDir}`: The directory containing the package. +- `{AppData}`: User-specific application data folder +- `{LocalAppData}`: User-specific local application data folder +- `{ProgramData}`: Application data for all users +- `{Temp}`: Temporary folder +- `{Documents}`: User's Documents folder + [game-metadata-id]: ../Storage/Games/About.md#id [backend]: ../../Loader/Backends/About.md @@ -661,4 +723,6 @@ if they themselves don't support it. (These mods should log a warning to console [platforms]: ../../Loader/Backends/About.md#platforms [architecture]: ../../Loader/Backends/About.md#architectures [tasks]: ./Tasks.md -[package-structure]: ./About.md#package-structure \ No newline at end of file +[package-structure]: ./About.md#package-structure +[ingest-config]: ../Storage/Locations.md#package-config-handling +[tools-as-packages]: ./Tools-As-Packages.md#chosen-approach \ No newline at end of file diff --git a/docs/Server/Packaging/Tasks.md b/docs/Server/Packaging/Tasks.md index 5579f42..0eb3701 100644 --- a/docs/Server/Packaging/Tasks.md +++ b/docs/Server/Packaging/Tasks.md @@ -27,10 +27,9 @@ Name = "Launch Game" GroupNames = ["GOG"] Description = "Launches the game." Path = { - Default = "", # Cross Platform Binary - Windows = "Game.exe", - Linux = "Game.sh", - MacOS = "Game.app" + win+x64-any = "Game.exe", + linux+x64-any = "Game.elf", + macos+x64-any = "Game.app" } IsPrimary = true IsHidden = false @@ -124,6 +123,10 @@ The `Path` field specifies the relative path to the executable file or the URL t - If the task is declared for a game, the `Path` is relative to the folder that contains the main executable of the game. - The one marked [IsPrimary](#isprimary). +// TODO: + - [Platform][platforms] only: `win`, `linux`. When package has all architecture binaries. + - [Platform][platforms]+[Arch][architecture]: `win+x64-v3`, `linux+x64-v3`. When package has only one specific architecture. + **Examples:** 1. Cross-platform executable: @@ -218,9 +221,6 @@ The available placeholders are: If the task is sourced from a package, the following variables are also available: - `{PackageDir}`: The absolute path to the base folder of the package. -- `{PackageConfigDir}`: The absolute path to the config folder for the package. - - [This is the `Package Configs` (`PackageConfigs/{loadoutId}/{packageId}`) folder.][items-to-store] - - This folder allows for - `{PackageUserCacheDir}`: The absolute path to the config folder for the package. - [This is the `Package Cache Files (User)` (`Cache/{packageId}`) folder.][items-to-store] - `{PackageMachineCacheDir}`: The absolute path to the config folder for the package. From d4aa375eb595e11ea4d1505ea14aa81c7a338abd Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Thu, 22 Aug 2024 03:09:40 +0100 Subject: [PATCH 03/17] Improved: Config Files Examples and Placeholders for Different Platforms Still needs Event Definition in Loadout and Message --- docs/Server/Packaging/Package-Metadata.md | 127 ++++++++++++++++++---- 1 file changed, 103 insertions(+), 24 deletions(-) diff --git a/docs/Server/Packaging/Package-Metadata.md b/docs/Server/Packaging/Package-Metadata.md index 9421e0e..18edbcb 100644 --- a/docs/Server/Packaging/Package-Metadata.md +++ b/docs/Server/Packaging/Package-Metadata.md @@ -217,7 +217,6 @@ automatically generated package. Specifically by: - Appending the `architecture` part to the [Id](#id) field. - Updating the [Targets](#targets) field to include the correct architecture (if needed). - !!! warning "***DO NOT*** use architectures be used in [dependency references](#dependency-info)." - βœ… `reloaded3.api.windows.vfs.s56` @@ -640,30 +639,49 @@ if they themselves don't support it. (These mods should log a warning to console Example section: ```toml +# A file inside the package directory +# If the tool writes config files to its own folders [[ConfigFiles]] Type = "File" -Path = "config.toml" Description = "Main configuration file" +[[ConfigFiles.Paths]] +OS = "any" +Path = "{PackageDir}/bin/config/config.json" +# Example of using globs with a base directory +[[ConfigFiles]] +Type = "Glob" +Description = "All XML files in the config directory" +[[ConfigFiles.Paths]] +OS = "any" +Path = "{PackageDir}/bin/config" +Extension = "json" +IncludeSubfolders = false + +# Example of using AppData across different OS [[ConfigFiles]] Type = "File" +Description = "User settings in AppData folder matching across OSes" +[[ConfigFiles.Paths]] +OS = "any" Path = "{AppData}/ToolName/settings.json" -Description = "User-specific settings" - -[[ConfigFiles]] -Type = "Folder" -Path = "presets" -Description = "Preset configurations" - -[[ConfigFiles]] -Type = "Glob" -Pattern = "*.cfg" -Description = "All CFG files in the root directory" +# Windows: C:\Users\{Username}\AppData\Roaming\ToolName\settings.json +# Linux: /home/{Username}/.config/ToolName/settings.json +# macOS: /Users/{Username}/Library/Application Support/ToolName/settings.json +# Example of a folder with OS-specific locations [[ConfigFiles]] Type = "Folder" -Path = "{LocalAppData}/ToolName/Logs" -Description = "Log files" +Description = "OS-specific data folder" +[[ConfigFiles.Paths]] +OS = "win" +Path = "{LocalAppData}/ToolName/Data" +[[ConfigFiles.Paths]] +OS = "linux" +Path = "{XDG_DATA_HOME}/ToolName" +[[ConfigFiles.Paths]] +OS = "macos" +Path = "{Library}/Application Support/ToolName" ``` When a loadout is loaded or the relevant `Tool` started by Reloaded3 stops running, updated @@ -675,26 +693,86 @@ For more details, see [Tools as Packages][tools-as-packages]. | Type | Name | Description | | --------------- | ----------- | --------------------------------------------------------------- | -| string | Type | Type of the config entry: "File", "Folder", or "Glob" | +| string | Type | Type of the config entry: `File`, `Folder`, or `Glob` | | [Path](#path)[] | Paths | Array of OS-specific paths for the config file or folder | | string | Description | A brief description of the configuration file or group of files | #### Path -| Type | Name | Description | -| ------ | ---- | -------------------------------------------------------- | -| string | OS | Target OS: "Windows", "Linux", "macOS", or "Default" | -| string | Path | The path or to the file/folder, can include placeholders | +| Type | Name | Description | +| ------ | ----------------- | -------------------------------------------------------- | +| string | OS | Target OS: `any`, `win`, `linux` or `macos`. | +| string | Path | The path or to the file/folder, can include placeholders | +| string | Extension | Name of extension when `Glob` is used. | +| bool | IncludeSubfolders | Whether to recurse folders when `Glob` is used. | + +The OS field names are based on the [Backend Names][native-backend]. + +## Path Placeholders + +!!! info "The `Path` field allows for some placeholders." -The `Path` field allows for some placeholders: +These are based on the `Windows` folder names, with equivalent `Linux` and `macOS` folders. -- `{PackageDir}`: The directory containing the package. -- `{AppData}`: User-specific application data folder +| Placeholder | Windows | Linux | macOS | +| ---------------- | ------------------------- | ----------------------------------------- | ----------------------------------- | +| `{PackageDir}` | Package directory | Package directory | Package directory | +| `{AppData}` | `%APPDATA%` | `$XDG_CONFIG_HOME` or `$HOME/.config` | `$HOME/Library/Application Support` | +| `{LocalAppData}` | `%LOCALAPPDATA%` | `$XDG_DATA_HOME` or `$HOME/.local/share` | `$HOME/Library/Application Support` | +| `{ProgramData}` | `%PROGRAMDATA%` | `/usr/local/share` | `/Library/Application Support` | +| `{Temp}` | `%TEMP%` | `$XDG_RUNTIME_DIR` or `/tmp` | `$TMPDIR` | +| `{Documents}` | `%USERPROFILE%\Documents` | `$XDG_DOCUMENTS_DIR` or `$HOME/Documents` | `$HOME/Documents` | +| `{Home}` | `%USERPROFILE%` | `$HOME` | `$HOME` | + +These are typically used for the following: + +- `{PackageDir}`: The directory containing the current package. +- `{AppData}`: User-specific (roaming) application data folder - `{LocalAppData}`: User-specific local application data folder - `{ProgramData}`: Application data for all users - `{Temp}`: Temporary folder - `{Documents}`: User's Documents folder +### Linux Specific Placeholders + +On `Linux`, the [XDG Path Spec][xdg-path-spec] is also supported: + +| Placeholder | Linux Location | +| --------------------- | ----------------------------------------- | +| `{XDG_DATA_HOME}` | `$XDG_DATA_HOME` or `$HOME/.local/share` | +| `{XDG_CONFIG_HOME}` | `$XDG_CONFIG_HOME` or `$HOME/.config` | +| `{XDG_CACHE_HOME}` | `$XDG_CACHE_HOME` or `$HOME/.cache` | +| `{XDG_DOCUMENTS_DIR}` | `$XDG_DOCUMENTS_DIR` or `$HOME/Documents` | + +Explanation of Linux Folders: + +- `{XDG_DATA_HOME}`: Directory for user-specific data files, typically used for non-configuration files like runtime data. Defaults to $HOME/.local/share. +- `{XDG_CONFIG_HOME}`: Directory for user-specific configuration files. Defaults to $HOME/.config. +- `{XDG_CACHE_HOME}`: Directory for user-specific non-essential data files, such as cache. Defaults to $HOME/.cache. +- `{XDG_DOCUMENTS_DIR}`: Directory for user-specific documents, usually mapped to $HOME/Documents. + +### macOS Specific Placeholders + +On `macOS`, the following placeholders map to directories commonly used by macOS-specific applications and are not part of the common cross-platform placeholders: + +| Placeholder | macOS Location | +| ---------------------- | --------------------------- | +| `{Library}` | `$HOME/Library` | +| `{LibraryCaches}` | `$HOME/Library/Caches` | +| `{LibraryPreferences}` | `$HOME/Library/Preferences` | +| `{LibraryLogs}` | `$HOME/Library/Logs` | +| `{LibraryContainers}` | `$HOME/Library/Containers` | +| `{SystemLibrary}` | `/Library` | + +Explanation of macOS Folders: + +- **`{Library}`**: The root directory for user-specific application support files, caches, preferences, etc. Located at `$HOME/Library`. +- **`{LibraryCaches}`**: Directory for user-specific cache files. Located at `$HOME/Library/Caches`. +- **`{LibraryPreferences}`**: Directory for user-specific application preference files. Located at `$HOME/Library/Preferences`. +- **`{LibraryLogs}`**: Directory for user-specific application log files. Located at `$HOME/Library/Logs`. +- **`{LibraryContainers}`**: Directory for containerized app data, used primarily for sandboxed apps. Located at `$HOME/Library/Containers`. +- **`{SystemLibrary}`**: The system-wide equivalent of the user’s Library folder, containing system-level application support files. Located at `/Library`. + [game-metadata-id]: ../Storage/Games/About.md#id [backend]: ../../Loader/Backends/About.md @@ -725,4 +803,5 @@ The `Path` field allows for some placeholders: [tasks]: ./Tasks.md [package-structure]: ./About.md#package-structure [ingest-config]: ../Storage/Locations.md#package-config-handling -[tools-as-packages]: ./Tools-As-Packages.md#chosen-approach \ No newline at end of file +[tools-as-packages]: ./Tools-As-Packages.md#chosen-approach +[xdg-path-spec]: https://specifications.freedesktop.org/basedir-spec/latest/index.html \ No newline at end of file From e748e78616e8bc4f898eeb980cb254824229ed0c Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Fri, 23 Aug 2024 04:30:12 +0100 Subject: [PATCH 04/17] Added: Encoding for External Configuration Files Still need to add event messages for this. --- docs/Server/Packaging/Package-Metadata.md | 25 ++++-- .../Storage/Loadouts/File-Format/DataTypes.md | 3 +- .../Storage/Loadouts/File-Format/Events.md | 54 ++++++++++--- .../Storage/Loadouts/File-Format/Unpacked.md | 80 ++++++++++++++++++- 4 files changed, 142 insertions(+), 20 deletions(-) diff --git a/docs/Server/Packaging/Package-Metadata.md b/docs/Server/Packaging/Package-Metadata.md index 18edbcb..8d3fb63 100644 --- a/docs/Server/Packaging/Package-Metadata.md +++ b/docs/Server/Packaging/Package-Metadata.md @@ -642,16 +642,18 @@ Example section: # A file inside the package directory # If the tool writes config files to its own folders [[ConfigFiles]] +Id = 0 Type = "File" Description = "Main configuration file" [[ConfigFiles.Paths]] OS = "any" Path = "{PackageDir}/bin/config/config.json" -# Example of using globs with a base directory +# Example of using 'by extension' with a base directory [[ConfigFiles]] -Type = "Glob" -Description = "All XML files in the config directory" +Id = 1 +Type = "ByExtension" +Description = "All JSON files in the config directory" [[ConfigFiles.Paths]] OS = "any" Path = "{PackageDir}/bin/config" @@ -660,6 +662,7 @@ IncludeSubfolders = false # Example of using AppData across different OS [[ConfigFiles]] +Id = 2 Type = "File" Description = "User settings in AppData folder matching across OSes" [[ConfigFiles.Paths]] @@ -671,6 +674,7 @@ Path = "{AppData}/ToolName/settings.json" # Example of a folder with OS-specific locations [[ConfigFiles]] +Id = 3 Type = "Folder" Description = "OS-specific data folder" [[ConfigFiles.Paths]] @@ -693,18 +697,24 @@ For more details, see [Tools as Packages][tools-as-packages]. | Type | Name | Description | | --------------- | ----------- | --------------------------------------------------------------- | -| string | Type | Type of the config entry: `File`, `Folder`, or `Glob` | +| u8 | Id | [Required] Unique identifier. Between 0-255. | +| string | Type | Type of the config entry: `File`, `Folder`, or `ByExtension` | | [Path](#path)[] | Paths | Array of OS-specific paths for the config file or folder | | string | Description | A brief description of the configuration file or group of files | +!!! info "The `Unique Identifier` (ID) is used to reference paths from external places." + + In a way where configuration files can be transferred between platforms, for example Linux <=> Windows. + An example of this is [Unpacked Loadouts][external-config-paths]. + #### Path | Type | Name | Description | | ------ | ----------------- | -------------------------------------------------------- | | string | OS | Target OS: `any`, `win`, `linux` or `macos`. | | string | Path | The path or to the file/folder, can include placeholders | -| string | Extension | Name of extension when `Glob` is used. | -| bool | IncludeSubfolders | Whether to recurse folders when `Glob` is used. | +| string | Extension | Name of extension when `ByExtension` is used. | +| bool | IncludeSubfolders | Whether to recurse folders when `ByExtension` is used. | The OS field names are based on the [Backend Names][native-backend]. @@ -804,4 +814,5 @@ Explanation of macOS Folders: [package-structure]: ./About.md#package-structure [ingest-config]: ../Storage/Locations.md#package-config-handling [tools-as-packages]: ./Tools-As-Packages.md#chosen-approach -[xdg-path-spec]: https://specifications.freedesktop.org/basedir-spec/latest/index.html \ No newline at end of file +[xdg-path-spec]: https://specifications.freedesktop.org/basedir-spec/latest/index.html +[external-config-paths]: ../Storage/Loadouts/File-Format/Unpacked.md#external-config-pathsbin \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md index 6f5bc88..eb7089d 100644 --- a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md +++ b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md @@ -2,9 +2,10 @@ - Max number of Package Download Data/Metadata (MetadataIdx): `268,435,456` (28 bits) - Max number of Configs (ConfigIdx): `134,217,727` (27 bits) +- Max number of External Configs (ExtConfigIdx): `134,217,727` (27 bits) - Max number of Events: `4,294,967,295` (32 bits) - Max number of Game Versions/Revisions (GameVerIdx): `65,536` (16 bits) -- Max timestamp. R3TimeStamp: `4,294,967,295` (32 bits). +- Max timestamp. (R3TimeStamp): `4,294,967,295` (32 bits). - This is the number of seconds since `1st January 2024`. - Max year 2160. diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index bf65c46..bdda047 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -43,15 +43,13 @@ Suppose we want to write the following sequence of events: Here's how these events would be written to the file: -``` -| Byte 0-7 | Meaning | -| ----------- | ---------------------------------------------------------- | -| 01 | [GameLaunched](#gamelaunched) event | -| 40 ?? | [PackageStatusChanged8](#0100-packagestatuschanged8) event | -| 41 ?? | [ConfigUpdated8](#0101-configupdated8) event | -| 00 00 00 | NOP padding to align next event to 8-byte boundary | -| 85 85 ?? ?? | [PackageUpdated16](#1005-packageupdated16) event | -``` +| Byte 0-7 | Meaning | +| ------------- | ---------------------------------------------------------- | +| `01` | [GameLaunched](#gamelaunched) event | +| `40 ??` | [PackageStatusChanged8](#0100-packagestatuschanged8) event | +| `41 ??` | [ConfigUpdated8](#0101-configupdated8) event | +| `00 00 00` | NOP padding to align next event to 8-byte boundary | +| `85 85 ?? ??` | [PackageUpdated16](#1005-packageupdated16) event | Explanation: @@ -620,6 +618,39 @@ This event is emitted the files of the game match a known new store manifest/rev | --------- | ------ | ----- | ---------------------------------------------------------------------------------------------------------------- | | `u8` | Length | X | [0-255] Length of new commandline parameters in [commandline-parameter-data.bin][commandline-parameter-data.bin] | +## ExternalConfigUpdated + +!!! info "This event indicates that a tool's configuration file has been updated." + +### Messages + +- [EXTERNAL_CONFIG_UPDATED_V0][external-config-updated-v0] + +TODO: ExternalConfigUpdated24 + +### {11}+07: ExternalConfigUpdated56 + +| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | MetadataIdx (44-63) | +| ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | +| C7 (`{11} + 07`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | + +| Data Type | Name | Label | Description | +| ------------------- | ----------- | ----- | ------------------------------------------------------------------------------------- | +| `u18` | PathIndex | X | [0-256K] Index of file in [external-config.bin][external-config-bin] | +| `u18` | FileIndex | Y | [0-256K] Index of file path in [external-config-paths.bin][external-config-paths-bin] | +| `u20` (MetadataIdx) | MetadataIdx | Z | [0-1M] Index of metadata in [Package References][packagemetadatabin] | + +### {11}+08: ExternalConfigUpdatedFull + +| EventType (0-7) | ExternalConfigIdx (8-35) | MetadataIdx (36-63) | +| ---------------- | ----------------------------------------- | ----------------------------------------- | +| C8 (`{11} + 08`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | + +| Data Type | Name | Label | Description | +| -------------------- | ----------------- | ----- | ----------------------------------------------------------------------------------------- | +| `u28` (ExtConfigIdx) | ExternalConfigIdx | X | [0-268M] Index of extended reference in [external-config-refs.bin][external-config-refs]. | +| `u28` (MetadataIdx) | MetadataIdx | Y | [0-268M] Index of metadata in [Package References][packagemetadatabin]. | + [configbin]: Unpacked.md#configbin [events-bin]: Unpacked.md#eventsbin [packagemetadatabin]: Unpacked.md#package-references @@ -677,4 +708,7 @@ This event is emitted the files of the game match a known new store manifest/rev [package-installed-as-dependency-v0]: ./Commit-Messages.md#package_installed_as_dependency_v0 [mod-installed-as-dependency-v0]: ./Commit-Messages.md#mod_installed_as_dependency_v0 [translation-installed-as-dependency-v0]: ./Commit-Messages.md#translation_installed_as_dependency_v0 -[tool-installed-as-dependency-v0]: ./Commit-Messages.md#tool_installed_as_dependency_v0 \ No newline at end of file +[tool-installed-as-dependency-v0]: ./Commit-Messages.md#tool_installed_as_dependency_v0 +[external-config-bin]: ./Unpacked.md#external-configbin +[external-config-paths-bin]: ./Unpacked.md#external-config-pathsbin +[external-config-refs]: ./Unpacked.md#external-config-refsbin \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index 1692166..43f406a 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -9,6 +9,7 @@ | [Timestamps](#timestampsbin) | `timestamps.bin` | Timestamps for each commit. | | [Commit Parameters](#commit-parameters) | `commit-parameter-types.bin`
+`commit-parameter-lengths-{x}.bin`
+ `commit-parameters-{x}.bin` | List of commit message parameters for each event. | | [Configs](#configbin) | `config.bin`
+ `config-data.bin` | Package Configurations. | +| [External Configs](#external-configbin) | `external-config.bin`
+ `external-config-data.bin`
+ `external-config-paths.bin` | Package Configurations. | | [Package Reference (IDs)](#package-references) | `package-reference-ids.bin` | Hashes of package IDs in this loadout. | | [Package Reference (Versions)](#package-references) | `package-reference-versions-len.bin`
+ `package-reference-versions.bin` | String versions of package IDs in this loadout. | | [Store Manifests](#storesbin) | `stores.bin`
+ `store-data.bin` | Game store specific info to restore game to last version if possible. | @@ -91,7 +92,7 @@ This is an array of 32-bit timestamps ([`R3TimeStamp[]`][max-numbers]). The numb ## config.bin -!!! info "This stores all historical mod configurations for any point in time." +!!! info "This stores all historical package configurations for any point in time." This is the array of file sizes, each being: @@ -121,10 +122,81 @@ every time a config is added. Emitted events refer to this index. ### config-data.bin -!!! info "This is a buffer of raw, unmodified configuration files." +!!! info "This is a buffer of raw, unmodified, ***unique*** configuration files." You can get the file size and offsets from the [config.bin](#configbin) file. +## external-config.bin + +!!! info "This stores all historical external configurations for any point in time." + + External configurations are these generated by tools, as opposed to the ones generated by R3. + +This is the array of file sizes, each being: + +| Data Type | Name | Description | +| --------- | -------- | ------------------------------- | +| `u32` | FileSize | Size of the configuration file. | + +Every new config is appended to [external-config-data.bin](#external-config-databin) as it is added. + +The path of every config is appended to [external-config-paths.bin](#external-config-pathsbin). + +Each unique config has an index, a.k.a. [ConfigIdx][max-numbers], which is an incrementing value from 0 +every time a config is added. Emitted events refer to this index. + +!!! question "How do you use this data?" + + See: [config.bin](#configbin)'s `How do you use this data?` section. + +### external-config-data.bin + +!!! info "This is a buffer of raw, unmodified, ***unique*** configuration files." + +You can get the file size and offsets from the [config.bin](#configbin) file. + +### external-config-paths.bin + +!!! info "This is a buffer of paths to configuration files generated by external tools/packages/programs." + +This is a list of ***unique*** `String8`s (1 byte prefixed UTF-8 Strings). + +Each path is a path *relative* to the path specified in [Config File Paths][config-file-paths]. + +!!! example "This is best illustrated with an example." + + For a config with the path set to + + ```toml + [[ConfigFiles]] + Id = 3 + Type = "Folder" # or 'ByExtension + Description = "OS-specific data folder" + [[ConfigFiles.Paths]] + OS = "any" + Path = "{LocalAppData}/ToolName/Data" + ``` + + Storing a path of `config.json` in `external-config-paths.bin` would equal the final path to be: + + - `"{LocalAppData}/ToolName/Data/config.json"` + + If the `Type` is `File`. An empty path would be used instead. + +### external-config-refs.bin + +!!! info "This is a buffer of `references` to [external-config-paths.bin](#external-config-pathsbin) and [external-config.bin](#external-configbin)" + +This is an array of the following structure + +| Data Type | Name | Description | +| --------- | --------- | -------------------------------------------------------------------------------------------- | +| `u32` | PathIndex | [0-4M] Index of configuration file in [external-config.bin][external-config-bin] | +| `u32` | FileIndex | [0-4M] Index of configuration path in [external-config-paths.bin][external-config-paths-bin] | + +This file is used in the event that one of the components of [ExternalConfigUpdated56][event-externalconfigupdated] +is out of range and cannot fit into the encoding. + ## Package References !!! info "A 'package reference' consists of a [XXH3(PackageID)][hashing] and Version." @@ -757,5 +829,9 @@ as regular parameters in [Commit Parameters](#commit-parameters). [commit-param-backrefs]: ./Unpacked.md#back-references [commit-param-lists]: ./Unpacked.md#parameter-lists [event-packagestatuschanged]: ./Events.md#packagestatuschanged +[event-externalconfigupdated]: ./Events.md#1107-externalconfigupdated56 [package-added]: ./Commit-Messages.md#package_added_v0 [snapshots]: ./Snapshot.md +[config-files]: ../../../Packaging/Package-Metadata.md#config-files +[config-file-paths]: ../../../Packaging/Package-Metadata.md#path +[config-file]: ../../../Packaging/Package-Metadata.md#configfile From 5f18289a79c18849387040d3571600c0dc23818c Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Fri, 23 Aug 2024 15:18:44 +0100 Subject: [PATCH 05/17] Added: Commit Message & Short Version of ExternalConfigUpdated --- .../Loadouts/File-Format/Commit-Messages.md | 14 ++++++++++++++ .../Server/Storage/Loadouts/File-Format/Events.md | 15 +++++++++++++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md index 00ae46f..6b5ba4c 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md +++ b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md @@ -451,6 +451,20 @@ to - `{NewCommandLine}` ``` +## External Config Updated (V0) + +### Parameters + +- `Name` [0]: The name of the package (mod, tool, etc.). [String][commit-param-type] +- `ID` [1]: The package ID. [String][commit-param-type] +- `ConfigPath` [2]: The path of the updated configuration file. [String][commit-param-type] + +### EXTERNAL_CONFIG_UPDATED_V0 + +``` +Updated external configuration file '{ConfigPath}' for '{Name}' (ID: '{ID}'). +``` + [events-bin]: ./Unpacked.md#eventsbin [event-packagestatuschanged]: ./Events.md#packagestatuschanged [event-config-updated]: ./Events.md#configupdated diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index bdda047..b0841a0 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -626,7 +626,17 @@ This event is emitted the files of the game match a known new store manifest/rev - [EXTERNAL_CONFIG_UPDATED_V0][external-config-updated-v0] -TODO: ExternalConfigUpdated24 +### {10}+0C: ExternalConfigUpdated24 + +| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | MetadataIdx (44-63) | +| ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | +| 8C (`{10} + 0C`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | + +| Data Type | Name | Label | Description | +| ------------------- | ----------- | ----- | ------------------------------------------------------------------------------------ | +| `u7` | PathIndex | X | [0-127] Index of file in [external-config.bin][external-config-bin] | +| `u7` | FileIndex | Y | [0-127] Index of file path in [external-config-paths.bin][external-config-paths-bin] | +| `u10` (MetadataIdx) | MetadataIdx | Z | [0-1024] Index of metadata in [Package References][packagemetadatabin] | ### {11}+07: ExternalConfigUpdated56 @@ -711,4 +721,5 @@ TODO: ExternalConfigUpdated24 [tool-installed-as-dependency-v0]: ./Commit-Messages.md#tool_installed_as_dependency_v0 [external-config-bin]: ./Unpacked.md#external-configbin [external-config-paths-bin]: ./Unpacked.md#external-config-pathsbin -[external-config-refs]: ./Unpacked.md#external-config-refsbin \ No newline at end of file +[external-config-refs]: ./Unpacked.md#external-config-refsbin +[external-config-updated-v0]: ./Commit-Messages.md#external_config_updated_v0 \ No newline at end of file From bf46fac09d85c4b4e56f106699cf8521e7121798 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Fri, 23 Aug 2024 16:05:11 +0100 Subject: [PATCH 06/17] Added: Missing Links & Small Fixups in Loadout --- .../Storage/Loadouts/File-Format/Packed.md | 4 +++ .../Storage/Loadouts/File-Format/Unpacked.md | 26 ++++++++++--------- 2 files changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Packed.md b/docs/Server/Storage/Loadouts/File-Format/Packed.md index 1d3f4e6..c665f3a 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Packed.md +++ b/docs/Server/Storage/Loadouts/File-Format/Packed.md @@ -45,6 +45,10 @@ The following tree structure represents the files included in the packed .nx arc β”œβ”€β”€ commit-parameters-lists.bin β”œβ”€β”€ config.bin β”œβ”€β”€ config-data.bin +β”œβ”€β”€ external-config.bin +β”œβ”€β”€ external-config-data.bin +β”œβ”€β”€ external-config-paths.bin +β”œβ”€β”€ external-config-refs.bin β”œβ”€β”€ package-reference-ids.bin β”œβ”€β”€ package-reference-versions-len.bin β”œβ”€β”€ package-reference-versions.bin diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index 43f406a..fc475db 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -40,14 +40,15 @@ performing a cleanup of unused data (by truncating remaining files). Format: -| Data Type | Name | Description | -| --------- | --------------- | ------------------------------------------------------------ | -| `u16` | Version | Version of the loadout format. | -| `u16` | Reserved | | -| `u32` | NumEvents | Total number of events and timestamps in this loadout. | -| `u32` | NumMetadata | Total number of package metadata files in this loadout. | -| `u32` | NumConfigs | Total number of package configuration files in this loadout. | -| `u32` | NumGameVersions | Total number of game versions. | +| Data Type | Name | Description | +| --------- | ------------------ | ------------------------------------------------------------------------------------ | +| `u16` | Version | Version of the loadout format. | +| `u16` | Reserved | | +| `u32` | NumEvents | Total number of [events][events-bin] and timestamps in this loadout. | +| `u32` | NumMetadata | Total number of [package metadata](#package-reference-idsbin) files in this loadout. | +| `u32` | NumConfigs | Total number of [package configuration](#configbin) files in this loadout. | +| `u32` | NumGameVersions | Total number of [game versions](#storesbin) (store entries). | +| `u32` | NumExternalConfigs | Total number of [external configuration](#external-configbin) files in this loadout. | !!! warning "Backwards compatibility is supported but not forwards." @@ -189,10 +190,10 @@ Each path is a path *relative* to the path specified in [Config File Paths][conf This is an array of the following structure -| Data Type | Name | Description | -| --------- | --------- | -------------------------------------------------------------------------------------------- | -| `u32` | PathIndex | [0-4M] Index of configuration file in [external-config.bin][external-config-bin] | -| `u32` | FileIndex | [0-4M] Index of configuration path in [external-config-paths.bin][external-config-paths-bin] | +| Data Type | Name | Description | +| --------- | --------- | ----------------------------------------------------------------------------------- | +| `u32` | FileIndex | [0-4M] Index of file in [external-config.bin](#external-configbin) | +| `u32` | PathIndex | [0-4M] Index of file path in [external-config-paths.bin](#external-config-pathsbin) | This file is used in the event that one of the components of [ExternalConfigUpdated56][event-externalconfigupdated] is out of range and cannot fit into the encoding. @@ -835,3 +836,4 @@ as regular parameters in [Commit Parameters](#commit-parameters). [config-files]: ../../../Packaging/Package-Metadata.md#config-files [config-file-paths]: ../../../Packaging/Package-Metadata.md#path [config-file]: ../../../Packaging/Package-Metadata.md#configfile +[community-repository]: ../../../../Services/Community-Repository.md \ No newline at end of file From 7fcdffd70ca310d527c606dbd31a4a8f39a51414 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sat, 24 Aug 2024 03:57:14 +0100 Subject: [PATCH 07/17] Improved: Further optimization of 'Events' format, with separation of package IDs and versions. --- .../Loadouts/File-Format/Commit-Messages.md | 2 +- .../Storage/Loadouts/File-Format/DataTypes.md | 3 +- .../Storage/Loadouts/File-Format/Events.md | 216 +++++++++--------- .../Storage/Loadouts/File-Format/Packed.md | 6 +- .../Storage/Loadouts/File-Format/Snapshot.md | 6 +- .../Storage/Loadouts/File-Format/Unpacked.md | 73 +++--- docs/Services/Central-Server.md | 2 +- 7 files changed, 162 insertions(+), 146 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md index 6b5ba4c..38249fe 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md +++ b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md @@ -470,7 +470,7 @@ Updated external configuration file '{ConfigPath}' for '{Name}' (ID: '{ID}'). [event-config-updated]: ./Events.md#configupdated [event-loadout-display-setting-changed]: ./Events.md#1004-loadoutdisplaysettingchanged [event-game-launched]: ./Events.md#gamelaunched -[event-packageupdated]: ./Events.md#packageupdated +[event-packageupdated]: ./Events.md#packageversionchanged [event-configupdated]: ./Events.md#configupdated [event-packageloadorderchanged]: ./Events.md#packageloadorderchanged [event-updategamestoremanifest]: ./Events.md#updategamestoremanifest diff --git a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md index eb7089d..f7c649e 100644 --- a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md +++ b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md @@ -1,6 +1,7 @@ ## Max Numbers -- Max number of Package Download Data/Metadata (MetadataIdx): `268,435,456` (28 bits) +- Max number of Unique Package IDs (PackageIdIdx): `1,048,576` (20 bits) +- Max number of Unique Package Versions (PackageVerIdx): `1,048,576` (20 bits) - Max number of Configs (ConfigIdx): `134,217,727` (27 bits) - Max number of External Configs (ExtConfigIdx): `134,217,727` (27 bits) - Max number of Events: `4,294,967,295` (32 bits) diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index b0841a0..7d121dc 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -138,7 +138,7 @@ This way we can ensure alignment is maintained. ### Messages -`PackageType` is the type of package referred to `MetadataIdx`. +`PackageType` is the type of package referred to by package at [PackageIdIdx]. - [PACKAGE_ADDED_V0][package-added-v0] when `NewStatus == Added` when `PackageType` is not known. - [PACKAGE_REMOVED_V0][package-removed-v0] when `NewStatus == Removed` when `PackageType` is not known. @@ -165,53 +165,52 @@ This way we can ensure alignment is maintained. - [TRANSLATION_INSTALLED_AS_DEPENDENCY_V0][translation-installed-as-dependency-v0] when `NewStatus == InstalledAsDependency` and `PackageType == Translation`. - [TOOL_INSTALLED_AS_DEPENDENCY_V0][tool-installed-as-dependency-v0] when `NewStatus == InstalledAsDependency` and `PackageType == Tool`. - ### {01}+00: PackageStatusChanged8 -| EventType (0-7) | NewStatus (8-10) | MetadataIdx (11-15) | -| ---------------- | ---------------- | ------------------- | -| 40 (`{01} + 00`) | `{XXX}` | `{YYYYY}` | +| EventType (0-7) | NewStatus (8-10) | [PackageIdIdx] (11-15) | +| ---------------- | ---------------- | ---------------------- | +| 40 (`{01} + 00`) | `{XXX}` | `{YYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | -------------------------------------------------------------------- | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u5` | MetadataIdx | Y | [0-31] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | ---------------------------------------------------------------- | +| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | +| `u5` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-31] Index of package ID in [package-ids.bin][package-ids.bin] | ### {10}+00: PackageStatusChanged16 -| EventType (0-7) | Padding (8-15) | NewStatus (16-18) | MetadataIdx (19-31) | +| EventType (0-7) | Padding (8-15) | NewStatus (16-18) | PackageIdIdx (19-31) | | ---------------- | -------------- | ----------------- | -------------------- | | 80 (`{10} + 00`) | 80 | `{XXX}` | `{YYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | ---------------------------------------------------------------------- | -| `u8` | Padding | 80 | Constant `80`. Repeats previous byte. | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u13` | MetadataIdx | Y | [0-8192] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ------------------------------------------------------------------ | +| `u8` | Padding | 80 | Constant `80`. Repeats previous byte. | +| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | +| `u13` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-8192] Index of package ID in [package-ids.bin][package-ids.bin] | ### {10}+01: PackageStatusChanged24 -| EventType (0-7) | NewStatus (8-10) | MetadataIdx (11-31) | +| EventType (0-7) | NewStatus (8-10) | PackageIdIdx (11-31) | | ---------------- | ---------------- | ------------------------------- | | 81 (`{10} + 01`) | `{XXX}` | `{YYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | -------------------------------------------------------------------- | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u21` | MetadataIdx | Y | [0-2M] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ---------------------------------------------------------------- | +| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | +| `u21` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-2M] Index of package ID in [package-ids.bin][package-ids.bin] | ### {11}+00: PackageStatusChanged32 -| EventType (0-7) | Padding (8-31) | Unused (32-32) | NewStatus (33-35) | MetadataIdx (36-63) | +| EventType (0-7) | Padding (8-31) | Unused (32-32) | NewStatus (33-35) | PackageIdIdx (36-63) | | ---------------- | -------------- | -------------- | ----------------- | ----------------------------------------- | | C0 (`{11} + 00`) | C0 C0 C0 | 0 | `{XXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | ---------------------------------------------------------------------- | -| `u24` | Padding | C0 | Constant `C0`. Repeats previous byte. | -| `u1` | Unused | 0 | | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u28` | MetadataIdx | Y | [0-268M] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ------------------------------------------------------------------ | +| `u24` | Padding | C0 | Constant `C0`. Repeats previous byte. | +| `u1` | Unused | 0 | | +| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | +| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin] | ## GameLaunched @@ -250,60 +249,60 @@ When the exact changes are not known, the event is [written as V1][commit-messag ### {01}+01: ConfigUpdated8 -| EventType (0-7) | NewStatus (8-11) | MetadataIdx (12-15) | -| ---------------- | ---------------- | ------------------- | -| 41 (`{01} + 01`) | `{XXXX}` | `{YYYY}` | +| EventType (0-7) | NewStatus (8-11) | PackageIdIdx (12-15) | +| ---------------- | ---------------- | -------------------- | +| 41 (`{01} + 01`) | `{XXXX}` | `{YYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | -------------------------------------------------------------------- | -| `u4` (ConfigIdx) | ConfigIdx | X | [0-15] Index of associated configuration in [config.bin][configbin] | -| `u4` (MetadataIdx) | MetadataIdx | Y | [0-15] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | ------------------------------------------------------------------- | +| `u4` (ConfigIdx) | ConfigIdx | X | [0-15] Index of associated configuration in [config.bin][configbin] | +| `u4` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-15] Index of package ID in [package-ids.bin][package-ids.bin] | ### {10}+02: ConfigUpdated16 -| EventType (0-7) | Padding (8-15) | ConfigIdx (16-22) | MetadataIdx (23-31) | -| ---------------- | -------------- | ----------------- | ------------------- | -| 82 (`{10} + 02`) | 82 | `{XXXXXXX}` | `{Y} {YYYYYYYY}` | +| EventType (0-7) | Padding (8-15) | ConfigIdx (16-22) | PackageIdIdx (23-31) | +| ---------------- | -------------- | ----------------- | -------------------- | +| 82 (`{10} + 02`) | 82 | `{XXXXXXX}` | `{Y} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | ----------- | ----- | --------------------------------------------------------------------- | -| `u8` | Padding | 82 | Constant `82`. Repeats previous byte. | -| `u7` (ConfigIdx) | ConfigIdx | X | [0-127] Index of associated configuration in [config.bin][configbin] | -| `u9` (MetadataIdx) | MetadataIdx | Y | [0-511] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | -------------------------------------------------------------------- | +| `u8` | Padding | 82 | Constant `82`. Repeats previous byte. | +| `u7` (ConfigIdx) | ConfigIdx | X | [0-127] Index of associated configuration in [config.bin][configbin] | +| `u9` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-511] Index of package ID in [package-ids.bin][package-ids.bin] | ### {10}+03: ConfigUpdated24 -| EventType (0-7) | ConfigIdx (8-18) | MetadataIdx (19-31) | +| EventType (0-7) | ConfigIdx (8-18) | PackageIdIdx (19-31) | | ---------------- | ----------------------------- | ------------------------------- | | 83 (`{10} + 01`) | `{XXXXXXXX} {XXXXXXXX} {XXX}` | `{YYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | ----------- | ----- | ---------------------------------------------------------------------- | -| `u11` (ConfigIdx) | ConfigIdx | X | [0-2047] Index of associated configuration in [config.bin][configbin] | -| `u13` (MetadataIdx) | MetadataIdx | Y | [0-8191] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | --------------------------------------------------------------------- | +| `u11` (ConfigIdx) | ConfigIdx | X | [0-2047] Index of associated configuration in [config.bin][configbin] | +| `u13` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-8191] Index of package ID in [package-ids.bin][package-ids.bin] | ### {11}+01 ConfigUpdated32 -| EventType (0-7) | Padding (8-31) | ConfigIdx (32-46) | MetadataIdx (47-63) | +| EventType (0-7) | Padding (8-31) | ConfigIdx (32-46) | PackageIdIdx (47-63) | | ---------------- | -------------- | ---------------------- | --------------------------- | | C1 (`{11} + 01`) | C1 C1 C1 | `{XXXXXXX} {XXXXXXXX}` | `{Y} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | ----------- | ----- | ------------------------------------------------------------------------ | -| `u24` | Padding | C1 | Constant `C1`. Maximize compression. | -| `u15` (ConfigIdx) | ConfigIdx | X | [0-32767] Index of associated configuration in [config.bin][configbin] | -| `u17` (MetadataIdx) | MetadataIdx | Y | [0-131071] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ---------------------------------------------------------------------- | +| `u24` | Padding | C1 | Constant `C1`. Maximize compression. | +| `u15` (ConfigIdx) | ConfigIdx | X | [0-32767] Index of associated configuration in [config.bin][configbin] | +| `u17` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-131071] Index of package ID in [package-ids.bin][package-ids.bin] | ### {11}+02 ConfigUpdatedFull -| EventType (0-7) | ConfigIdx (8-35) | MetadataIdx (36-63) | +| EventType (0-7) | ConfigIdx (8-35) | PackageIdIdx (36-63) | | ---------------- | ---------------------- | --------------------------- | | C2 (`{11} + 02`) | `{XXXXXXX} {XXXXXXXX}` | `{Y} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | ----------- | ----- | ---------------------------------------------------------------------- | -| `u27` (ConfigIdx) | ConfigIdx | X | [0-134M] Index of associated configuration in [config.bin][configbin] | -| `u28` (MetadataIdx) | MetadataIdx | Y | [0-268M] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | --------------------------------------------------------------------- | +| `u27` (ConfigIdx) | ConfigIdx | X | [0-134M] Index of associated configuration in [config.bin][configbin] | +| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin] | ## LoadoutDisplaySettingChanged @@ -314,7 +313,6 @@ This is rarely changed so has a large 4-byte payload and can change multiple eve ### Messages - [LOADOUT_DISPLAY_SETTINGS_CHANGED_V0][loadout-display-settings-changed-v0] when multiple settings have changed. - - [LOADOUT_GRID_ENABLED_SORT_MODE_CHANGED_V0][loadout-grid-enabled-sort-mode-changed-v0] when only `LoadoutGridEnabledSortMode` has changed. - [LOADOUT_GRID_DISABLED_SORT_MODE_CHANGED_V0][loadout-grid-disabled-sort-mode-changed-v0] when only `LoadoutGridDisabledSortMode` has changed. - [MOD_LOAD_ORDER_SORT_CHANGED_V0][mod-load-order-sort-changed-v0] when only `ModLoadOrderSort` has changed. @@ -333,20 +331,16 @@ This is rarely changed so has a large 4-byte payload and can change multiple eve | `u2` [(SortOrder)][sortorder] | ModLoadOrderSort | Y | Sorting mode for load order reorderer. | | `u4` [(GridDisplayMode)][griddisplaymode] | LoadoutGridStyle | Z | Display mode for LoadoutGrid. | -## PackageUpdated +## PackageVersionChanged !!! info "This event indicates that a package has been updated to a new version." -This discards the previous manifest at `OldMetadataIdx` and replaces it with the new manifest at `NewMetadataIdx`. +This upgrades a package at [PackageIdIdx] from the old (previous) version to the new version at `NewPackageVerIdx`. -!!! note "`NewMetadataIdx` can point to either a newly written manifest or a previous one" +!!! note "`NewPackageVerIdx` can point to either a newly written manifest or a previous one" It's a previous one in case of a rollback/undo, otherwise it's a new one. -!!! note "Some mods can receive updates quite often" - - That's why `OldMetadatIdx` and `NewMetadataIdx` are evenly distributed in bits. - ### Messages - [PACKAGE_UPDATED_V0][package-updated-v0] when `PackageType` is not known. @@ -356,49 +350,49 @@ This discards the previous manifest at `OldMetadataIdx` and replaces it with the ### {10}+05: PackageUpdated16 -| EventType (0-7) | Padding (8-15) | OldMetadataIdx (16-23) | NewMetadataIdx (24-31) | -| ---------------- | -------------- | ---------------------- | ---------------------- | -| 85 (`{10} + 05`) | 85 | `{XXXXXXXX}` | `{YYYYYYYY}` | +| EventType (0-7) | Padding (8-15) | OldPackageVerIdx (16-23) | NewPackageVerIdx (24-31) | +| ---------------- | -------------- | ------------------------ | ------------------------ | +| 85 (`{10} + 05`) | 85 | `{XXXXXXXX}` | `{YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------ | -------------- | ----- | ------------------------------------------------------------------------ | -| `u8` | Padding | 85 | Constant `85`. Repeats previous byte. | -| `u8` (MetadataIdx) | OldMetadataIdx | X | [0-255] Index of old version in [Package References][packagemetadatabin] | -| `u8` (MetadataIdx) | NewMetadataIdx | Y | [0-255] Index of new version in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| --------------------- | ---------------- | ----- | ------------------------------------------------------------------------ | +| `u8` | Padding | 85 | Constant `85`. Repeats previous byte. | +| `u8` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-255] Index of old version in [Package References][packagemetadatabin] | +| `u8` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-255] Index of new version in [Package References][packagemetadatabin] | ### {10}+06: PackageUpdated24 -| EventType (0-7) | OldMetadataIdx (8-19) | NewMetadataIdx (20-31) | +| EventType (0-7) | OldPackageVerIdx (8-19) | NewPackageVerIdx (20-31) | | ---------------- | ------------------------------ | ------------------------------ | | 86 (`{10} + 06`) | `{XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | -------------- | ----- | ------------------------------------------------------------------------- | -| `u12` (MetadataIdx) | OldMetadataIdx | X | [0-4095] Index of old version in [Package References][packagemetadatabin] | -| `u12` (MetadataIdx) | NewMetadataIdx | Y | [0-4095] Index of new version in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | ---------------- | ----- | ------------------------------------------------------------------------- | +| `u12` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-4095] Index of old version in [Package References][packagemetadatabin] | +| `u12` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-4095] Index of new version in [Package References][packagemetadatabin] | ### {11}+03 PackageUpdated32 -| EventType (0-7) | Padding (8-31) | OldMetadataIdx (8-35) | NewMetadataIdx (36-63) | +| EventType (0-7) | Padding (8-31) | OldPackageVerIdx (8-35) | NewPackageVerIdx (36-63) | | ---------------- | -------------- | ----------------------------------------- | ----------------------------------------- | | C3 (`{11} + 03`) | C3 C3 C3 | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | -------------- | ----- | -------------------------------------------------------------------------- | -| `u24` | Padding | C3 | Constant `C3`. Repeats previous byte. | -| `u16` (MetadataIdx) | OldMetadataIdx | X | [0-65535] Index of old version in [Package References][packagemetadatabin] | -| `u16` (MetadataIdx) | NewMetadataIdx | Y | [0-65535] Index of new version in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | ---------------- | ----- | -------------------------------------------------------------------------- | +| `u24` | Padding | C3 | Constant `C3`. Repeats previous byte. | +| `u16` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-65535] Index of old version in [Package References][packagemetadatabin] | +| `u16` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-65535] Index of new version in [Package References][packagemetadatabin] | ### {11}+04 PackageUpdated56 -| EventType (0-7) | OldMetadataIdx (8-35) | NewMetadataIdx (36-63) | +| EventType (0-7) | OldPackageVerIdx (8-35) | NewPackageVerIdx (36-63) | | ---------------- | ----------------------------------------- | ----------------------------------------- | | C4 (`{11} + 04`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ------------------- | -------------- | ----- | ------------------------------------------------------------------------- | -| `u28` (MetadataIdx) | OldMetadataIdx | X | [0-268M] Index of old version in [Package References][packagemetadatabin] | -| `u28` (MetadataIdx) | NewMetadataIdx | Y | [0-268M] Index of new version in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | ---------------- | ----- | ------------------------------------------------------------------------- | +| `u28` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-268M] Index of old version in [Package References][packagemetadatabin] | +| `u28` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-268M] Index of new version in [Package References][packagemetadatabin] | ## PackageLoadOrderChanged @@ -620,7 +614,7 @@ This event is emitted the files of the game match a known new store manifest/rev ## ExternalConfigUpdated -!!! info "This event indicates that a tool's configuration file has been updated." +!!! info "This event indicates that an external config file belonging to a package has been updated." ### Messages @@ -628,42 +622,43 @@ This event is emitted the files of the game match a known new store manifest/rev ### {10}+0C: ExternalConfigUpdated24 -| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | MetadataIdx (44-63) | +| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | PackageIdIdx (44-63) | | ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | | 8C (`{10} + 0C`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | -| Data Type | Name | Label | Description | -| ------------------- | ----------- | ----- | ------------------------------------------------------------------------------------ | -| `u7` | PathIndex | X | [0-127] Index of file in [external-config.bin][external-config-bin] | -| `u7` | FileIndex | Y | [0-127] Index of file path in [external-config-paths.bin][external-config-paths-bin] | -| `u10` (MetadataIdx) | MetadataIdx | Z | [0-1024] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ------------------------------------------------------------------------------------ | +| `u7` | PathIndex | X | [0-127] Index of file in [external-config.bin][external-config-bin] | +| `u7` | FileIndex | Y | [0-127] Index of file path in [external-config-paths.bin][external-config-paths-bin] | +| `u10` ([PackageIdIdx]) | [PackageIdIdx] | Z | [0-1024] Index of package ID in [package-ids.bin][package-ids.bin] | ### {11}+07: ExternalConfigUpdated56 -| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | MetadataIdx (44-63) | +| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | [PackageIdIdx] (44-63) | | ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | | C7 (`{11} + 07`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | -| Data Type | Name | Label | Description | -| ------------------- | ----------- | ----- | ------------------------------------------------------------------------------------- | -| `u18` | PathIndex | X | [0-256K] Index of file in [external-config.bin][external-config-bin] | -| `u18` | FileIndex | Y | [0-256K] Index of file path in [external-config-paths.bin][external-config-paths-bin] | -| `u20` (MetadataIdx) | MetadataIdx | Z | [0-1M] Index of metadata in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| -------------------- | ------------ | ----- | ------------------------------------------------------------------------------------- | +| `u18` | PathIndex | X | [0-256K] Index of file in [external-config.bin][external-config-bin] | +| `u18` | FileIndex | Y | [0-256K] Index of file path in [external-config-paths.bin][external-config-paths-bin] | +| `u20` (PackageIdIdx) | PackageIdIdx | Z | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | ### {11}+08: ExternalConfigUpdatedFull -| EventType (0-7) | ExternalConfigIdx (8-35) | MetadataIdx (36-63) | +| EventType (0-7) | ExternalConfigIdx (8-35) | PackageIdIdx (36-63) | | ---------------- | ----------------------------------------- | ----------------------------------------- | | C8 (`{11} + 08`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| -------------------- | ----------------- | ----- | ----------------------------------------------------------------------------------------- | -| `u28` (ExtConfigIdx) | ExternalConfigIdx | X | [0-268M] Index of extended reference in [external-config-refs.bin][external-config-refs]. | -| `u28` (MetadataIdx) | MetadataIdx | Y | [0-268M] Index of metadata in [Package References][packagemetadatabin]. | +| Data Type | Name | Label | Description | +| ---------------------- | ----------------- | ----- | ----------------------------------------------------------------------------------------- | +| `u28` (ExtConfigIdx) | ExternalConfigIdx | X | [0-268M] Index of extended reference in [external-config-refs.bin][external-config-refs]. | +| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin]. | [configbin]: Unpacked.md#configbin [events-bin]: Unpacked.md#eventsbin -[packagemetadatabin]: Unpacked.md#package-references +[packagemetadatabin]: Unpacked.md#packages +[package-ids.bin]: Unpacked.md#package-idsbin [package-added-v0]: ./Commit-Messages.md#package_added_v0 [package-removed-v0]: ./Commit-Messages.md#package_removed_v0 [package-hidden-v0]: ./Commit-Messages.md#package_hidden_v0 @@ -722,4 +717,5 @@ This event is emitted the files of the game match a known new store manifest/rev [external-config-bin]: ./Unpacked.md#external-configbin [external-config-paths-bin]: ./Unpacked.md#external-config-pathsbin [external-config-refs]: ./Unpacked.md#external-config-refsbin -[external-config-updated-v0]: ./Commit-Messages.md#external_config_updated_v0 \ No newline at end of file +[external-config-updated-v0]: ./Commit-Messages.md#external_config_updated_v0 +[PackageIdIdx]: ./DataTypes.md \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/Packed.md b/docs/Server/Storage/Loadouts/File-Format/Packed.md index c665f3a..3e6deaa 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Packed.md +++ b/docs/Server/Storage/Loadouts/File-Format/Packed.md @@ -49,9 +49,9 @@ The following tree structure represents the files included in the packed .nx arc β”œβ”€β”€ external-config-data.bin β”œβ”€β”€ external-config-paths.bin β”œβ”€β”€ external-config-refs.bin -β”œβ”€β”€ package-reference-ids.bin -β”œβ”€β”€ package-reference-versions-len.bin -β”œβ”€β”€ package-reference-versions.bin +β”œβ”€β”€ package-ids.bin +β”œβ”€β”€ package-versions-len.bin +β”œβ”€β”€ package-versions.bin β”œβ”€β”€ stores.bin β”œβ”€β”€ store-data.bin └── commandline-parameter-data.bin diff --git a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md index f8a381a..38221bc 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md +++ b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md @@ -101,15 +101,17 @@ struct MicrosoftStoreData { 2. **Packages** - A list of all packages in the loadout, each containing: - Package ID: A unique identifier for the package ([XXH3 hash][hashing] of the package ID). - - Version: The semantic version of the package as a string (e.g., "1.2.3"). + - Version: The semantic version of the package as a string (e.g., `1.2.3`). - State: The current state of the package using the [`PackageState`][packagestate] enum. - Configuration Index: An optional index into the configurations Vec. - Corresponds to [Package References][packagereferenceidsbin]. + - Only one version of a package can be installed at a time. 3. **Configurations** - A list of raw configuration data for packages. - Indexed by the `configuration_index` in `PackageInfo`. - Corresponds to [config.bin & config-data.bin][configbin]. + - Configurations are not version specific. They persist across package versions. 4. **Mod Load Order** - An ordered list of indices representing the current load order of mods. @@ -191,4 +193,4 @@ there will be no migration code; as to avoid bloating the binary. [loadout-location]: ../About.md#location [bitcode]: ../../../../Research/Library-Sizes/Serializers.md#bitcode [configbin]: ./Unpacked.md#configbin -[packagereferenceidsbin]: ./Unpacked.md#package-reference-idsbin \ No newline at end of file +[packagereferenceidsbin]: ./Unpacked.md#package-idsbin \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index fc475db..575561f 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -10,8 +10,8 @@ | [Commit Parameters](#commit-parameters) | `commit-parameter-types.bin`
+`commit-parameter-lengths-{x}.bin`
+ `commit-parameters-{x}.bin` | List of commit message parameters for each event. | | [Configs](#configbin) | `config.bin`
+ `config-data.bin` | Package Configurations. | | [External Configs](#external-configbin) | `external-config.bin`
+ `external-config-data.bin`
+ `external-config-paths.bin` | Package Configurations. | -| [Package Reference (IDs)](#package-references) | `package-reference-ids.bin` | Hashes of package IDs in this loadout. | -| [Package Reference (Versions)](#package-references) | `package-reference-versions-len.bin`
+ `package-reference-versions.bin` | String versions of package IDs in this loadout. | +| [Package Reference (IDs)](#packages) | `package-ids.bin` | Hashes of package IDs in this loadout. | +| [Package Reference (Versions)](#packages) | `package-versions-len.bin`
+ `package-versions.bin` | String versions of package IDs in this loadout. | | [Store Manifests](#storesbin) | `stores.bin`
+ `store-data.bin` | Game store specific info to restore game to last version if possible. | | [Commandline Parameters](#commandline-parameter-databin) | `commandline-parameter-data.bin` | Raw data for commandline parameters. Length specified in event. | @@ -40,15 +40,16 @@ performing a cleanup of unused data (by truncating remaining files). Format: -| Data Type | Name | Description | -| --------- | ------------------ | ------------------------------------------------------------------------------------ | -| `u16` | Version | Version of the loadout format. | -| `u16` | Reserved | | -| `u32` | NumEvents | Total number of [events][events-bin] and timestamps in this loadout. | -| `u32` | NumMetadata | Total number of [package metadata](#package-reference-idsbin) files in this loadout. | -| `u32` | NumConfigs | Total number of [package configuration](#configbin) files in this loadout. | -| `u32` | NumGameVersions | Total number of [game versions](#storesbin) (store entries). | -| `u32` | NumExternalConfigs | Total number of [external configuration](#external-configbin) files in this loadout. | +| Data Type | Name | Description | +| --------- | ------------------ | -------------------------------------------------------------------------------------------- | +| `u16` | Version | Version of the loadout format. | +| `u16` | Reserved | | +| `u32` | NumEvents | Total number of [events][events-bin] and timestamps in this loadout. | +| `u32` | NumPackageIds | Total number of unique [package ID(s)](#package-idsbin) files in this loadout. | +| `u32` | NumPackageVersions | Total number of unique [package version(s)](#package-versions-lenbin) files in this loadout. | +| `u32` | NumConfigs | Total number of [package configuration](#configbin) files in this loadout. | +| `u32` | NumGameVersions | Total number of [game versions](#storesbin) (store entries). | +| `u32` | NumExternalConfigs | Total number of [external configuration](#external-configbin) files in this loadout. | !!! warning "Backwards compatibility is supported but not forwards." @@ -198,28 +199,32 @@ This is an array of the following structure This file is used in the event that one of the components of [ExternalConfigUpdated56][event-externalconfigupdated] is out of range and cannot fit into the encoding. -## Package References +## Packages -!!! info "A 'package reference' consists of a [XXH3(PackageID)][hashing] and Version." +!!! info "We refer to a 'unique package' by [XXH3(PackageID)][hashing]" - From [Package.toml][package-toml]. + The field `PackageID` being the `Id` field from [Package.toml][package-toml]. - This is the minimum amount of data required to uniquely identify a package. + There can only be one version of a package with given ID in a loadout + at a given time. -Packages are referred to by an index known as [MetadataIdx][max-numbers] in the events. +References to Package IDs (`XXH3(PackageID)`) are referred to by an index known as +[PackageIdIdx][max-numbers] in the events: -So a `MetadataIdx == 1` means `fetch the entry at index 1` of [package-reference-ids.bin](#package-reference-idsbin) -and [package-reference-versions.bin](#package-reference-idsbin). +- A `PackageIdIdx == 1` in an event means `fetch the entry at index 1` of [package-ids.bin](#package-idsbin). -As for how to use the data, it is similar to [config.bin](#configbin), essentially we deduplicate -entries by in-memory hash. So an event can always refer to a [MetadataIdx][max-numbers] created +Most events will only require a [PackageIdIdx][max-numbers]. However, in some cases the version +[PackageVerIdx](#package-versions) is also needed, for example, to upgrade packages. + +As for how to use the data, it is similar to [config.bin](#configbin). We deduplicate +entries by in-memory hash. So an event can always refer to a [PackageIdIdx][max-numbers] created in an earlier event to save space. !!! danger "Launcher MUST ensure each published mod has valid update/download data." - Otherwise this system could fail, as a hash of packageID is not useful. + Otherwise this system could fail, as a hash of packageID on its own is not useful. -### package-reference-ids.bin +### package-ids.bin !!! info "This is a buffer of [XXH3(PackageID)][hashing]" @@ -232,7 +237,7 @@ System can still always fail, we just pray it won't. !!! note "Some Numbers" - Nexus Mods alone hosts 815999 mods as of 30th of May 2024 (obtained via GraphQL API). + [Nexus Mods][nexus] alone hosts 815999 mods as of 30th of May 2024 (obtained via GraphQL API). The probability of a hash collision on whole mod set is roughly the following: @@ -257,9 +262,20 @@ and at that point a meteor is more likely to land on your house (no joke). !!! note "This ID is used to [restore the package](#restoring-actual-package-files)." -### package-reference-versions-len.bin +## Package Versions + +!!! info "In some contexts, it may also be useful to know the package version." + +Reference to unique *Package Version* are referred to by an index known as [PackageVerIdx][max-numbers] +in the events: + +- So a `PackageVerIdx == 1` in an event means `fetch the entry at index 1` of [package-versions.bin](#package-versionsbin). + +This information is sometimes used to `e.g.` upgrade packages. + +### package-versions-len.bin -Contains the lengths of entries in [package-reference-versions.bin](#package-reference-idsbin). +Contains the lengths of entries in [package-versions.bin](#package-idsbin). | Data Type | Name | Description | | --------- | ------------- | --------------------------- | @@ -269,9 +285,9 @@ Contains the lengths of entries in [package-reference-versions.bin](#package-ref Most versions are of form `X.Y.Z` so there is a lot of repetition of `05`. -### package-reference-versions.bin +### package-versions.bin -!!! info "This is a buffer consisting of package versions, whose length is defined in [package-reference.bin](#package-references)" +!!! info "This is a buffer consisting of package versions, whose length is defined in [package-reference.bin](#packages)" These versions are stored as UTF-8 strings. No null terminator. @@ -836,4 +852,5 @@ as regular parameters in [Commit Parameters](#commit-parameters). [config-files]: ../../../Packaging/Package-Metadata.md#config-files [config-file-paths]: ../../../Packaging/Package-Metadata.md#path [config-file]: ../../../Packaging/Package-Metadata.md#configfile -[community-repository]: ../../../../Services/Community-Repository.md \ No newline at end of file +[community-repository]: ../../../../Services/Community-Repository.md +[nexus]: https://www.nexusmods.com \ No newline at end of file diff --git a/docs/Services/Central-Server.md b/docs/Services/Central-Server.md index 3f397ee..02f54aa 100644 --- a/docs/Services/Central-Server.md +++ b/docs/Services/Central-Server.md @@ -930,6 +930,6 @@ By default, I recommend searching by substring. In Name and ModId. Summary can b [pages-limits]: https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages#usage-limits [package-id]: ../Server/Packaging/Package-Metadata.md#id [sync-loadout]: ../Server/Storage/Loadouts/File-Format/Unpacked.md#restoring-actual-package-files -[package-reference-ids]: ../Server/Storage/Loadouts/File-Format/Unpacked.md#package-reference-idsbin +[package-reference-ids]: ../Server/Storage/Loadouts/File-Format/Unpacked.md#package-idsbin [hashing]: ../Common/Hashing.md#stable-hashing-for-general-purpose-use [mod-metadata-search-image]: ../Server/Packaging/Package-Metadata.md#icon-search \ No newline at end of file From 099b3f58b91e5ac34dd3525573c15628fdc1bab8 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 03:43:57 +0100 Subject: [PATCH 08/17] Optimized: Opcode Allocation in Events.md to Reduce Size Usage --- .../Loadouts/File-Format/Commit-Messages.md | 128 +++- .../Storage/Loadouts/File-Format/DataTypes.md | 30 +- .../Storage/Loadouts/File-Format/Events.md | 632 +++++++++++------- .../Storage/Loadouts/File-Format/Packed.md | 1 - .../Storage/Loadouts/File-Format/Unpacked.md | 30 +- 5 files changed, 566 insertions(+), 255 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md index 38249fe..cc2610e 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md +++ b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md @@ -202,6 +202,99 @@ Installed translation '**{Name}**' with ID '**{ID}**' and version '**{Version}** Installed tool '**{Name}**' with ID '**{ID}**' and version '**{Version}**' as a dependency. ``` +## Package Added (with Config) (V0) + +### Parameters + +- `Name` [0]: The name of the package, mod, translation or tool. [String] +- `ID` [1]: The package ID. [String] +- `ChangeList` [2]: List of configuration changes. [Parameter List] +- `Version` [-1]: The package version. [String [Contextual]] + +### PACKAGE_ADDED_WITH_CONFIG_V0 + +``` +Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. + +You've set up the following initial settings: + +{ChangeList} + +(ID: '{ID}') +``` + +### MOD_ADDED_WITH_CONFIG_V0 + +``` +Added mod '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. + +You've configured these starter settings for the mod: + +{ChangeList} + +(ID: '{ID}') +``` + +### TRANSLATION_ADDED_WITH_CONFIG_V0 + +``` +Added language '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. + +You've set up these initial language settings: + +{ChangeList} + +(ID: '{ID}') +``` + +### TOOL_ADDED_WITH_CONFIG_V0 + +``` +Added tool '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. + +You've configured these initial settings: + +{ChangeList} + +(ID: '{ID}') +``` + +### PACKAGE_ADDED_WITH_CONFIG_V1 + +``` +Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. +You've set up some initial settings too. But we didn't understand the settings format. + +(ID: '{ID}') +``` + +### MOD_ADDED_WITH_CONFIG_V1 + +``` +Added mod '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. +You've set up some initial settings too. But we didn't understand the settings format. + +(ID: '{ID}') +``` + +### TRANSLATION_ADDED_WITH_CONFIG_V1 + +``` +Added language '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. +You've set up some initial settings too. But we didn't understand the settings format. + +(ID: '{ID}') +``` + +### TOOL_ADDED_WITH_CONFIG_V1 + +``` +Added tool '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. +You've set up some initial settings too. But we didn't understand the settings format. + +(ID: '{ID}') +``` + ## Package Updated (V0) !!! info "Source: Event [PackageUpdated][event-packageupdated]" @@ -254,10 +347,38 @@ e.g. `11th of May 2020`. - `EventTimestamp` [-1]: [TimeStamp [Contextual]](#contextual-parameters) -## Config Updated (V0) +## GAME_LAUNCHED_N_V0 + +!!! info "Invoked by [GameLaunchedN][event-game-launched-n]" ### Parameters +- `Timestamps` [-1]: List of launch timestamps. [TimeStamp List [Contextual]](#contextual-parameters) + +``` +Launched game multiple times: +- {Timestamps} +``` + +### GAME_LAUNCHED_N_PARAM_V0 + +!!! note "This is an individual item in the `Timestamps` list above" + +``` +- **{TimeStamp}** +``` + +Example: + +``` +- 11th of May 2020 +- 12th of May 2020 +- 13th of May 2020 +``` + +## Config Updated (V0) + +### Parameters - `Name` [0]: The name of the package, mod, translation or tool. [String][commit-param-type] - `ModID` [1]: The package ID for mod/tool. [String][commit-param-type] @@ -468,7 +589,7 @@ Updated external configuration file '{ConfigPath}' for '{Name}' (ID: '{ID}'). [events-bin]: ./Unpacked.md#eventsbin [event-packagestatuschanged]: ./Events.md#packagestatuschanged [event-config-updated]: ./Events.md#configupdated -[event-loadout-display-setting-changed]: ./Events.md#1004-loadoutdisplaysettingchanged +[event-loadout-display-setting-changed]: ./Events.md#16-loadoutdisplaysettingchanged [event-game-launched]: ./Events.md#gamelaunched [event-packageupdated]: ./Events.md#packageversionchanged [event-configupdated]: ./Events.md#configupdated @@ -479,4 +600,5 @@ Updated external configuration file '{ConfigPath}' for '{Name}' (ID: '{ID}'). [commit-param-type]: ./Unpacked.md#parametertype [commit-param-list]: ./Unpacked.md#parameter-lists [commit-param-version]: ./Unpacked.md#commit-parameters-versionsbin -[r3-locale-format]: ../../../../Common/Localisation/File-Format.md \ No newline at end of file +[r3-locale-format]: ../../../../Common/Localisation/File-Format.md +[event-game-launched-n]: ./Events.md#03-gamelaunchedn \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md index f7c649e..4eb8520 100644 --- a/docs/Server/Storage/Loadouts/File-Format/DataTypes.md +++ b/docs/Server/Storage/Loadouts/File-Format/DataTypes.md @@ -1,9 +1,13 @@ ## Max Numbers - Max number of Unique Package IDs (PackageIdIdx): `1,048,576` (20 bits) + - Realistically expect 10,000 in very, very large loadouts. + - And 100 in typical loadouts. - Max number of Unique Package Versions (PackageVerIdx): `1,048,576` (20 bits) -- Max number of Configs (ConfigIdx): `134,217,727` (27 bits) -- Max number of External Configs (ExtConfigIdx): `134,217,727` (27 bits) + - Realistically expect 200 in very, very large loadouts. +- Max number of Unique Configs (ConfigIdx): `1,048,576` (20 bits) +- Max number of Unique External Configs (ExtConfigIdx): `262,144` (18 bits) + - This is both max number of unique external paths, and unique external data. - Max number of Events: `4,294,967,295` (32 bits) - Max number of Game Versions/Revisions (GameVerIdx): `65,536` (16 bits) - Max timestamp. (R3TimeStamp): `4,294,967,295` (32 bits). @@ -26,6 +30,28 @@ - `4`: `Enabled`. The package was enabled in the loadout. - `5`: `InstalledAsDependency`. The package was installed as a dependency. +### PackageInstallStateChange + +!!! info "This is a sub-form of [PackageStateChange](#packagestatechange) that only includes install states." + + - Size: 2 bits + - Possible values: 0-3 + +- `0`: `Removed`. The package was removed from the loadout. +- `1`: `Hidden` The package was hidden from the loadout. +- `2`: `Added`. The package was added to the loadout. +- `3`: `InstalledAsDependency`. The package was installed as a dependency. + +### PackageEnabledStateChange + +!!! info "This is a sub-form of [PackageStateChange](#packagestatechange) that only includes enabled states." + + - Size: 1 bits + - Possible values: 0-1 + +- `0`: `Disabled` (Default State). The package was disabled in the loadout. +- `1`: `Enabled`. The package was enabled in the loadout. + ## PackageState !!! info "Represents the current state of a package in the loadout." diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index 7d121dc..d2f63a3 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -6,7 +6,7 @@ So an order like `u8`, and `u24` means 0:8 bits, then 8:32 bits. -Each event is represented by a 1 byte `EventType` (denoted in section title). +Each event is represented by a 1 byte `EventType` (denoted in section title ***in hex***). This is a power of 2, and can be followed by a 1, 3 or 7 byte payload.
This makes each event 1, 2, 4 or 8 bytes long. @@ -23,10 +23,6 @@ the event will be written. Any event needing data longer than 7 bytes, the data should be stored in another file and accessed by index. -!!! note "`EventType` 0xF0 - 0xFF are 2 byte codes." - - This allows extending the total number of opcodes to 4336. - ## An Example Let's consider a sequence of events to illustrate how padding and reading work in this system. @@ -34,29 +30,31 @@ We'll use a mix of different event types to show various scenarios. ### Writing Events +!!! info "The event types used here are be outdated. This is just an example only." + Suppose we want to write the following sequence of events: -1. [GameLaunched](#gamelaunched) (1 byte) -2. [PackageStatusChanged8](#0100-packagestatuschanged8) (2 bytes) +1. [GameLaunched][GameLaunched] (1 byte) +2. `PackageStatusChanged8` (2 bytes) 3. [ConfigUpdated8](#0101-configupdated8) (2 bytes) 4. [PackageUpdated16](#1005-packageupdated16) (4 bytes) Here's how these events would be written to the file: -| Byte 0-7 | Meaning | -| ------------- | ---------------------------------------------------------- | -| `01` | [GameLaunched](#gamelaunched) event | -| `40 ??` | [PackageStatusChanged8](#0100-packagestatuschanged8) event | -| `41 ??` | [ConfigUpdated8](#0101-configupdated8) event | -| `00 00 00` | NOP padding to align next event to 8-byte boundary | -| `85 85 ?? ??` | [PackageUpdated16](#1005-packageupdated16) event | +| Byte 0-7 | Meaning | +| ------------- | -------------------------------------------------- | +| `01` | [GameLaunched][GameLaunched] event | +| `40 ??` | `PackageStatusChanged8` event | +| `41 ??` | [ConfigUpdated8](#0101-configupdated8) event | +| `00 00 00` | NOP padding to align next event to 8-byte boundary | +| `85 85 ?? ??` | [PackageUpdated16](#1005-packageupdated16) event | Explanation: -- The [GameLaunched](#gamelaunched) event (`01`) takes 1 byte. -- The [PackageStatusChanged8](#0100-packagestatuschanged8) event (`40 XX`) takes 2 bytes. +- The [GameLaunched][GameLaunched] event (`01`) takes 1 byte. +- The `PackageStatusChanged8` event (`40 XX`) takes 2 bytes. - The [ConfigUpdated8](#0101-configupdated8) event (`41 XX`) takes 2 bytes. -- At this point, we've written 5 bytes. To ensure the next 4-byte event ([PackageUpdated16](#1005-packageupdated16)) starts on an 8-byte boundary, we add 3 bytes of [NOP](#0000-nop) padding (`00 00 00`). +- At this point, we've written 5 bytes. To ensure the next 4-byte event ([PackageUpdated16](#1005-packageupdated16)) starts on an 8-byte boundary, we add 3 bytes of [NOP](#00-nop) padding (`00 00 00`). - Finally, we write the [PackageUpdated16](#1005-packageupdated16) event (`85 85 XX XX`), which takes 4 bytes. ### Reading Events @@ -64,8 +62,8 @@ Explanation: When reading these events, the system would perform full 8-byte reads: 1. First read (8 bytes): `01 40 ?? 41 ?? 00 00 00` - - Processes [GameLaunched](#gamelaunched) (1 byte) - - Processes [PackageStatusChanged8](#0100-packagestatuschanged8) (2 bytes) + - Processes [GameLaunched][GameLaunched] (1 byte) + - Processes `PackageStatusChanged8` (2 bytes) - Processes [ConfigUpdated8](#0101-configupdated8) (2 bytes) - Skips NOP padding (3 bytes) @@ -80,11 +78,24 @@ When reading these events, the system would perform full 8-byte reads: To achieve this we do the following: - Padding bytes use same byte as EventType to increase repeated bytes. -- EventType(s) have forms with multiple lengths (to minimize unused bytes). + - For >=3 bytes. + - ZStandard does not compress duplicates shorter than 3 bytes well. +- EventType(s) have forms with multiple lengths. +- The same byte should not be repeated twice. + +- Use as many opcodes as possible. + - Any unused opcodes is wasted space, don't be afraid to add more. + - The format can always have a breaking change up until final release. -## Event Ranges +## Opcode Distribution -The payload size is determined by the 2 high bits of the event type. +!!! info "The opcode distribution is arbitrary." + + And may be subject to change between versions up until final release (although unlikely). + + However backwards compatibility in the library will always be maintained. + +A non-arbitrary optimization was previously considered in the form: | Sequence | Size | | -------- | ---- | @@ -93,35 +104,41 @@ The payload size is determined by the 2 high bits of the event type. | 10 | 3 | | 11 | 7 | -This leaves the remaining next 6 bits (0-63) for the event type. +This would reduce (x86) code size and improve decode speed by a small amount, but is not in use here +to favour instead reducing loadout size. + +!!! tip "We can instantly load last state from [Snapshots], therefore first load is not as huge a priority." -i.e. `{YY}{XXXXXX}`. + Instead, we mainly replay the events to revert a loadout, to an earlier state, a rare event. + It is therefore acceptable if this takes up a tiny bit longer (up to 300ms on a 100,000 event loadout). ## Event Representation Each event is represented with something like this: -| EventType (0-7) | Field X (8-11) | Field Y (12-15) | -| ---------------- | -------------- | --------------- | -| 01 (`{00} + 01`) | `{XXX}` | `{YYYY}` | +| EventType (0-7) | Field X (8-11) | Field Y (12-15) | +| --------------- | -------------- | --------------- | +| `01` | `{XXX}` | `{YYYY}` | Ranges are inclusive. Elements in curly brackets `{}` indicate binary packing. So the payload `{XXX}` means '`X` is stored in 3 bits'. These provide a visual representation of the ranges to prevent ambiguity. -In the EventType, the `{00}` denotes the size prefix, and the `+` shows the offset into -the given prefix (0-63, in hex). +In the EventType, the number is provided in hex. Sometimes it can be provided as a range, for example +`03` - `13`; when it is provided as a range, each entry slightly alters the behaviour of the event. -## {00}+00: NOP +For more details, see the event specific documentation. + +## [00] NOP !!! info "Dummy no-op event for restoring alignment." This event is used to pad the next event to the next 8 byte boundary* -| EventType (0-7) | -| ---------------- | -| 00 (`{00} + 00`) | +| EventType (0-7) | +| --------------- | +| `00` | This is used as padding if there is an event that needs to be written, but it will span over the 8 byte boundary. For example, if we've written 7 bytes and are about to write a 2 byte event. @@ -136,6 +153,10 @@ This way we can ensure alignment is maintained. !!! info "A new package has been added to `Package References` and can be seen from loadout." +!!! tip "When adding a new package for the first time, i.e. `NewStatus == Added` consider using [PackageAdded](#packageadded) instead." + + This also allows you to set the initial version of the package in a single operation. + ### Messages `PackageType` is the type of package referred to by package at [PackageIdIdx]. @@ -165,52 +186,19 @@ This way we can ensure alignment is maintained. - [TRANSLATION_INSTALLED_AS_DEPENDENCY_V0][translation-installed-as-dependency-v0] when `NewStatus == InstalledAsDependency` and `PackageType == Translation`. - [TOOL_INSTALLED_AS_DEPENDENCY_V0][tool-installed-as-dependency-v0] when `NewStatus == InstalledAsDependency` and `PackageType == Tool`. -### {01}+00: PackageStatusChanged8 - -| EventType (0-7) | NewStatus (8-10) | [PackageIdIdx] (11-15) | -| ---------------- | ---------------- | ---------------------- | -| 40 (`{01} + 00`) | `{XXX}` | `{YYYYY}` | - -| Data Type | Name | Label | Description | -| --------------------- | -------------- | ----- | ---------------------------------------------------------------- | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u5` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-31] Index of package ID in [package-ids.bin][package-ids.bin] | - -### {10}+00: PackageStatusChanged16 - -| EventType (0-7) | Padding (8-15) | NewStatus (16-18) | PackageIdIdx (19-31) | -| ---------------- | -------------- | ----------------- | -------------------- | -| 80 (`{10} + 00`) | 80 | `{XXX}` | `{YYYYY} {YYYYYYYY}` | +### [01] PackageStatusChanged24 -| Data Type | Name | Label | Description | -| ---------------------- | -------------- | ----- | ------------------------------------------------------------------ | -| `u8` | Padding | 80 | Constant `80`. Repeats previous byte. | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u13` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-8192] Index of package ID in [package-ids.bin][package-ids.bin] | - -### {10}+01: PackageStatusChanged24 - -| EventType (0-7) | NewStatus (8-10) | PackageIdIdx (11-31) | -| ---------------- | ---------------- | ------------------------------- | -| 81 (`{10} + 01`) | `{XXX}` | `{YYYYY} {YYYYYYYY} {YYYYYYYY}` | +| EventType (0-7) | NewStatus (8-10) | PackageIdIdx (11-30) | Reserved (31-31) | +| --------------- | ---------------- | ------------------------------- | ---------------- | +| `02` | `{XXX}` | `{YYYYY} {YYYYYYYY} {YYYYYYYY}` | `{Z}` | | Data Type | Name | Label | Description | | ---------------------- | -------------- | ----- | ---------------------------------------------------------------- | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u21` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-2M] Index of package ID in [package-ids.bin][package-ids.bin] | - -### {11}+00: PackageStatusChanged32 - -| EventType (0-7) | Padding (8-31) | Unused (32-32) | NewStatus (33-35) | PackageIdIdx (36-63) | -| ---------------- | -------------- | -------------- | ----------------- | ----------------------------------------- | -| C0 (`{11} + 00`) | C0 C0 C0 | 0 | `{XXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | +| [PackageStateChange] | NewStatus | X | See [PackageStateChange][pkgstatechange] | +| `u20` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | +| `u1` | Reserved | Z | Currently Unused | -| Data Type | Name | Label | Description | -| ---------------------- | -------------- | ----- | ------------------------------------------------------------------ | -| `u24` | Padding | C0 | Constant `C0`. Repeats previous byte. | -| `u1` | Unused | 0 | | -| PackageStateChange | NewStatus | X | See [PackageStateChange][pkgstatechange] | -| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin] | +!!! tip "For adding mods, please use [PackageAddedFull]." ## GameLaunched @@ -225,13 +213,31 @@ This event has no extra data. ### Messages -- [GAME_LAUNCHED_V0][game-launched-v0] +- [GAME_LAUNCHED_V0][game-launched-v0] when [GameLaunched] +- [GAME_LAUNCHED_N_V0][game-launched-n-v0] when [GameLaunchedN] + +### [02] GameLaunched + +| EventType (0-7) | +| --------------- | +| `02` | + +### [03] GameLaunchedN + +!!! info "This is equivalent to repeating [GameLaunched] event `N` times." -### {00}+01: GameLaunched +| EventType (0-7) | Length (8-15) | +| --------------- | ------------- | +| `03` | `{XXXXXXXX}` | -| EventType (0-7) | -| ---------------- | -| 01 (`{00} + 01`) | +| Data Type | Name | Label | Description | +| --------- | ------ | ----- | ----------------------------------------------------------- | +| `u8` | Length | N | [0-255] Number of times to repeat the [GameLaunched] event. | + +!!! danger "This event emits N timestamps in [timestamps.bin]." + + Make sure not to miss this in the decoder!! + This is a compression opcode over repeating the [GameLaunched] event `N` times. ## ConfigUpdated @@ -247,62 +253,51 @@ When the exact changes are not known, the event is [written as V1][commit-messag - [MOD_CONFIG_UPDATED_V1][mod-config-updated-v1] when `PackageType == Mod` and exact changes are not known. - [TOOL_CONFIG_UPDATED_V1][tool-config-updated-v1] when `PackageType == Tool` and exact changes are not known. -### {01}+01: ConfigUpdated8 - -| EventType (0-7) | NewStatus (8-11) | PackageIdIdx (12-15) | -| ---------------- | ---------------- | -------------------- | -| 41 (`{01} + 01`) | `{XXXX}` | `{YYYY}` | - -| Data Type | Name | Label | Description | -| --------------------- | -------------- | ----- | ------------------------------------------------------------------- | -| `u4` (ConfigIdx) | ConfigIdx | X | [0-15] Index of associated configuration in [config.bin][configbin] | -| `u4` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-15] Index of package ID in [package-ids.bin][package-ids.bin] | +### [04] - [13] ConfigUpdated24 -### {10}+02: ConfigUpdated16 +| EventType (0-7) | [ConfigIdx] (8-21) | [PackageIdIdx] (22-31) | +| --------------- | ------------------------------ | ---------------------- | +| `04` - `13` | `{XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY}` | -| EventType (0-7) | Padding (8-15) | ConfigIdx (16-22) | PackageIdIdx (23-31) | -| ---------------- | -------------- | ----------------- | -------------------- | -| 82 (`{10} + 02`) | 82 | `{XXXXXXX}` | `{Y} {YYYYYYYY}` | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ---------------------------------------------------------------------- | +| `u14` ([ConfigIdx]) | [ConfigIdx] | X | [0-16383] Index of associated configuration in [config.bin][configbin] | +| `u10` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1023] Index of package ID in [package-ids.bin][package-ids.bin] | -| Data Type | Name | Label | Description | -| --------------------- | -------------- | ----- | -------------------------------------------------------------------- | -| `u8` | Padding | 82 | Constant `82`. Repeats previous byte. | -| `u7` (ConfigIdx) | ConfigIdx | X | [0-127] Index of associated configuration in [config.bin][configbin] | -| `u9` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-511] Index of package ID in [package-ids.bin][package-ids.bin] | +The `EventType` has 16 reserved values, each for a different range of [PackageIdIdx]. -### {10}+03: ConfigUpdated24 +They function as follows: -| EventType (0-7) | ConfigIdx (8-18) | PackageIdIdx (19-31) | -| ---------------- | ----------------------------- | ------------------------------- | -| 83 (`{10} + 01`) | `{XXXXXXXX} {XXXXXXXX} {XXX}` | `{YYYYY} {YYYYYYYY} {YYYYYYYY}` | +- `04` [PackageIdIdx] has range [0-1023]. +- `05` [PackageIdIdx] has range [1024-2047]. +- ... +- `13` [PackageIdIdx] has range [15360-16383]. -| Data Type | Name | Label | Description | -| ---------------------- | -------------- | ----- | --------------------------------------------------------------------- | -| `u11` (ConfigIdx) | ConfigIdx | X | [0-2047] Index of associated configuration in [config.bin][configbin] | -| `u13` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-8191] Index of package ID in [package-ids.bin][package-ids.bin] | +i.e. Each `EventType` has a range of 256 package IDs. -### {11}+01 ConfigUpdated32 +### [14] ConfigUpdated32 -| EventType (0-7) | Padding (8-31) | ConfigIdx (32-46) | PackageIdIdx (47-63) | -| ---------------- | -------------- | ---------------------- | --------------------------- | -| C1 (`{11} + 01`) | C1 C1 C1 | `{XXXXXXX} {XXXXXXXX}` | `{Y} {YYYYYYYY} {YYYYYYYY}` | +| EventType (0-7) | Padding (8-31) | [ConfigIdx] (32-47) | PackageIdIdx (48-63) | +| --------------- | -------------- | ----------------------- | ----------------------- | +| `14` - `17` | CNST | `{XXXXXXXX} {XXXXXXXX}` | `{YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ---------------------- | -------------- | ----- | ---------------------------------------------------------------------- | -| `u24` | Padding | C1 | Constant `C1`. Maximize compression. | -| `u15` (ConfigIdx) | ConfigIdx | X | [0-32767] Index of associated configuration in [config.bin][configbin] | -| `u17` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-131071] Index of package ID in [package-ids.bin][package-ids.bin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ------------------------------------------------------------------------ | +| `u24` | Padding | CNST | Constant that repeats `EventType` field. Ignored. Maximizes compression. | +| `u16` ([ConfigIdx]) | [ConfigIdx] | X | [0-65535] Index of associated configuration in [config.bin][configbin] | +| `u16` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-65535] Index of package ID in [package-ids.bin][package-ids.bin] | -### {11}+02 ConfigUpdatedFull +### [15] ConfigUpdatedFull -| EventType (0-7) | ConfigIdx (8-35) | PackageIdIdx (36-63) | -| ---------------- | ---------------------- | --------------------------- | -| C2 (`{11} + 02`) | `{XXXXXXX} {XXXXXXXX}` | `{Y} {YYYYYYYY} {YYYYYYYY}` | +| EventType (0-7) | Padding (8-23) | ConfigIdx (24-43) | PackageIdIdx (44-63) | +| --------------- | -------------- | ----------------------------- | ------------------------------ | +| `15` | `15` `15` | `{XXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| ---------------------- | -------------- | ----- | --------------------------------------------------------------------- | -| `u27` (ConfigIdx) | ConfigIdx | X | [0-134M] Index of associated configuration in [config.bin][configbin] | -| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin] | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ------------------------------------------------------------------- | +| `u16` | Padding | `15` | Repeats `EventType` field. Ignored. Maximizes compression. | +| `u20` ([ConfigIdx]) | [ConfigIdx] | X | [0-1M] Index of associated configuration in [config.bin][configbin] | +| `u20` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | ## LoadoutDisplaySettingChanged @@ -318,11 +313,11 @@ This is rarely changed so has a large 4-byte payload and can change multiple eve - [MOD_LOAD_ORDER_SORT_CHANGED_V0][mod-load-order-sort-changed-v0] when only `ModLoadOrderSort` has changed. - [LOADOUT_GRID_STYLE_CHANGED_V0][loadout-grid-style-changed-v0] when only `LoadoutGridStyle` has changed. -### {10}+04 LoadoutDisplaySettingChanged +### [16] LoadoutDisplaySettingChanged -| EventType (0-7) | Unused (8-11) | LoadoutGridEnabledSortMode (12-18) | LoadoutGridDisabledSortMode (19-25) | ModLoadOrderSort (26-27) | LoadoutGridStyle (28-31) | -| ---------------- | ------------- | ---------------------------------- | ----------------------------------- | ------------------------ | ------------------------ | -| 84 (`{10} + 04`) | | `{WWWWWWW}` | `{XXXXXXX}` | `{YY}` | `{ZZZZ}` | +| EventType (0-7) | Unused (8-11) | LoadoutGridEnabledSortMode (12-18) | LoadoutGridDisabledSortMode (19-25) | ModLoadOrderSort (26-27) | LoadoutGridStyle (28-31) | +| --------------- | ------------- | ---------------------------------- | ----------------------------------- | ------------------------ | ------------------------ | +| `16` | | `{WWWWWWW}` | `{XXXXXXX}` | `{YY}` | `{ZZZZ}` | | Data Type | Name | Label | Description | | ----------------------------------------- | --------------------------- | ----- | ----------------------------------------------- | @@ -348,51 +343,30 @@ This upgrades a package at [PackageIdIdx] from the old (previous) version to the - [TRANSLATION_UPDATED_V0][translation-updated-v0] when `PackageType == Translation`. - [TOOL_UPDATED_V0][tool-updated-v0] when `PackageType == Tool`. -### {10}+05: PackageUpdated16 - -| EventType (0-7) | Padding (8-15) | OldPackageVerIdx (16-23) | NewPackageVerIdx (24-31) | -| ---------------- | -------------- | ------------------------ | ------------------------ | -| 85 (`{10} + 05`) | 85 | `{XXXXXXXX}` | `{YYYYYYYY}` | - -| Data Type | Name | Label | Description | -| --------------------- | ---------------- | ----- | ------------------------------------------------------------------------ | -| `u8` | Padding | 85 | Constant `85`. Repeats previous byte. | -| `u8` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-255] Index of old version in [Package References][packagemetadatabin] | -| `u8` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-255] Index of new version in [Package References][packagemetadatabin] | - -### {10}+06: PackageUpdated24 +### [17] PackageUpdated24 -| EventType (0-7) | OldPackageVerIdx (8-19) | NewPackageVerIdx (20-31) | -| ---------------- | ------------------------------ | ------------------------------ | -| 86 (`{10} + 06`) | `{XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY}` | +| EventType (0-7) | OldPackageVerIdx (8-19) | NewPackageVerIdx (20-31) | +| --------------- | ----------------------- | ------------------------ | +| `17` | `{XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY}` | | Data Type | Name | Label | Description | | ---------------------- | ---------------- | ----- | ------------------------------------------------------------------------- | | `u12` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-4095] Index of old version in [Package References][packagemetadatabin] | | `u12` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-4095] Index of new version in [Package References][packagemetadatabin] | -### {11}+03 PackageUpdated32 +The `EventType` has 16 reserved values, each for a different range of [PackageIdIdx]. -| EventType (0-7) | Padding (8-31) | OldPackageVerIdx (8-35) | NewPackageVerIdx (36-63) | -| ---------------- | -------------- | ----------------------------------------- | ----------------------------------------- | -| C3 (`{11} + 03`) | C3 C3 C3 | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | +### [18] PackageUpdatedFull -| Data Type | Name | Label | Description | -| ---------------------- | ---------------- | ----- | -------------------------------------------------------------------------- | -| `u24` | Padding | C3 | Constant `C3`. Repeats previous byte. | -| `u16` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-65535] Index of old version in [Package References][packagemetadatabin] | -| `u16` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-65535] Index of new version in [Package References][packagemetadatabin] | +| EventType (0-7) | Padding (8-23) | OldPackageVerIdx (24-43) | NewPackageVerIdx (44-63) | +| --------------- | -------------- | ----------------------------- | ------------------------------ | +| `18` | `18 18` | `{XXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY}` | -### {11}+04 PackageUpdated56 - -| EventType (0-7) | OldPackageVerIdx (8-35) | NewPackageVerIdx (36-63) | -| ---------------- | ----------------------------------------- | ----------------------------------------- | -| C4 (`{11} + 04`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | - -| Data Type | Name | Label | Description | -| ---------------------- | ---------------- | ----- | ------------------------------------------------------------------------- | -| `u28` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-268M] Index of old version in [Package References][packagemetadatabin] | -| `u28` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-268M] Index of new version in [Package References][packagemetadatabin] | +| Data Type | Name | Label | Description | +| ---------------------- | ---------------- | ----- | ----------------------------------------------------------------------- | +| `u16` | Padding | `18` | Repeats `EventType` field. Ignored. Maximizes compression. | +| `u20` ([PackageIdIdx]) | OldPackageVerIdx | X | [0-1M] Index of old version in [Package References][packagemetadatabin] | +| `u20` ([PackageIdIdx]) | NewPackageVerIdx | Y | [0-1M] Index of new version in [Package References][packagemetadatabin] | ## PackageLoadOrderChanged @@ -476,76 +450,65 @@ On something like a GameCube, `50ms`. - [MOD_LOAD_ORDER_CHANGED_V0][mod-load-order-changed-v0] when `PackageType == Mod`. - [TRANSLATION_LOAD_ORDER_CHANGED_V0][translation-load-order-changed-v0] when `PackageType == Translation`. -### {10}+07: PackageLoadOrderChanged16 +### [19] PackageLoadOrderChanged16 -| EventType (0-7) | OldPosition (8-15) | NewPosition (16-23) | -| ---------------- | ------------------ | ------------------- | -| 87 (`{10} + 07`) | `{XXXXXXXX}` | `{YYYYYYYY}` | +| EventType (0-7) | OldPosition (8-15) | NewPosition (16-23) | +| --------------- | ------------------ | ------------------- | +| `19` | `{XXXXXXXX}` | `{YYYYYYYY}` | | Data Type | Name | Label | Description | | --------- | ----------- | ----- | -------------------------------------------------- | | `u8` | OldPosition | X | [0-255] Old position of the mod in the load order. | | `u8` | NewPosition | Y | [0-255] New position of the mod in the load order. | -### {10}+08: PackageLoadOrderChanged24 +### [1A] PackageLoadOrderChanged24 -| EventType (0-7) | OldPosition (8-19) | NewPosition (20-31) | -| ---------------- | ------------------- | ------------------- | -| 88 (`{10} + 08`) | `{XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY}` | +| EventType (0-7) | OldPosition (8-19) | NewPosition (20-31) | +| --------------- | ------------------- | ------------------- | +| `1A` | `{XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY}` | | Data Type | Name | Label | Description | | --------- | ----------- | ----- | --------------------------------------------------- | | `u12` | OldPosition | X | [0-4095] Old position of the mod in the load order. | | `u12` | NewPosition | Y | [0-4095] New position of the mod in the load order. | -### {10}+09: PackageLoadOrderMovedToBottom24 +### [1B] PackageLoadOrderMovedToBottom24 Optimized form for common action of moving a mod to the bottom of the load order. -| EventType (0-7) | OldPosition (8-27) | OffsetFromBottom (28-31) | -| ---------------- | ------------------------------ | ------------------------ | -| 89 (`{10} + 09`) | `{XXXX} {XXXXXXXX} {XXXXXXXX}` | `{YYY}` | +| EventType (0-7) | OldPosition (8-27) | OffsetFromBottom (28-31) | +| --------------- | ------------------------------ | ------------------------ | +| `1B` | `{XXXX} {XXXXXXXX} {XXXXXXXX}` | `{YYY}` | | Data Type | Name | Label | Description | | --------- | ---------------- | ----- | ------------------------------------------------- | | `u20` | OldPosition | X | [0-1M] Old position of the mod in the load order. | | `u4` | OffsetFromBottom | Y | [0-15] Offset from bottom. | -### {10}+0A: PackageLoadOrderMovedToTop24 +### [1C] PackageLoadOrderMovedToTop24 Optimized form for common action of moving a mod to the top of the load order. -| EventType (0-7) | OldPosition (8-27) | OffsetFromTop (28-31) | -| ---------------- | ------------------------------ | --------------------- | -| 8A (`{10} + 0A`) | `{XXXX} {XXXXXXXX} {XXXXXXXX}` | `{YYY}` | +| EventType (0-7) | OldPosition (8-27) | OffsetFromTop (28-31) | +| --------------- | ------------------------------ | --------------------- | +| `1C` | `{XXXX} {XXXXXXXX} {XXXXXXXX}` | `{YYY}` | | Data Type | Name | Label | Description | | --------- | ------------- | ----- | ------------------------------------------------- | | `u20` | OldPosition | X | [0-1M] Old position of the mod in the load order. | | `u4` | OffsetFromTop | Y | [0-15] Offset from top. | -### {11}+05: PackageLoadOrderChanged32 +### [1D] PackageLoadOrderChanged32 -| EventType (0-7) | Padding (8-31) | OldPosition (32-47) | NewPosition (48-63) | -| ---------------- | -------------- | ----------------------- | ----------------------- | -| C5 (`{11} + 05`) | C5 C5 C5 | `{XXXXXXXX} {XXXXXXXX}` | `{YYYYYYYY} {YYYYYYYY}` | +| EventType (0-7) | Padding (8-23) | OldPosition (24-43) | NewPosition (44-63) | +| --------------- | -------------- | ----------------------------- | ------------------------------ | +| `1D` | `1D` `1D` | `{XXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY}` | -| Data Type | Name | Label | Description | -| --------- | ----------- | ----- | ---------------------------------------------------- | -| `u24` | Padding | C5 | Constant `C5`. Repeats previous byte. | -| `u16` | OldPosition | X | [0-65535] Old position of the mod in the load order. | -| `u16` | NewPosition | Y | [0-65535] New position of the mod in the load order. | - -### {11}+06: PackageLoadOrderChanged56 - -| EventType (0-7) | OldPosition (8-35) | NewPosition (36-63) | -| ---------------- | ----------------------------------------- | ----------------------------------------- | -| C6 (`{11} + 06`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | - -| Data Type | Name | Label | Description | -| --------- | ----------- | ----- | --------------------------------------------------- | -| `u28` | OldPosition | X | [0-268M] Old position of the mod in the load order. | -| `u28` | NewPosition | Y | [0-268M] New position of the mod in the load order. | +| Data Type | Name | Label | Description | +| ---------------------- | -------------- | ----- | ---------------------------------------------------------- | +| `u16` | Padding | `1D` | Repeats `EventType` field. Ignored. Maximizes compression. | +| `u20` ([ConfigIdx]) | [ConfigIdx] | X | [0-1M] Old position of the mod in the load order. | +| `u20` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1M] New position of the mod in the load order. | ## UpdateGameStoreManifest @@ -567,23 +530,23 @@ This event is emitted the files of the game match a known new store manifest/rev - [UPDATE_GAME_STORE_MANIFEST_XBOX_V0][update-game-store-manifest-xbox-v0] when the store is Microsoft (Xbox) Store. - [UPDATE_GAME_STORE_MANIFEST_EGS_V0][update-game-store-manifest-egs-v0] when the store is Epic Games Store. -### {01}+02: UpdateGameStoreManifest +### [1E] UpdateGameStoreManifest8 -| EventType (0-7) | NewRevision (8-15) | -| ---------------- | ------------------ | -| 42 (`{01} + 02`) | `{XXXXXXXX}` | +| EventType (0-7) | NewRevision (8-15) | +| --------------- | ------------------ | +| `1E` | `{XXXXXXXX}` | | Data Type | Name | Label | Description | | ----------------- | ----------- | ----- | ---------------------------------- | | `u8` (GameVerIdx) | NewRevision | X | [0-255] New game version revision. | -### {10}+0B: UpdateGameStoreManifest +### [1F] UpdateGameStoreManifest24 !!! note "Unlikely this will ever be used, but just in case." -| EventType (0-7) | NewRevision (8-31) | -| ---------------- | ---------------------------------- | -| 8B (`{10} + 0B`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX}` | +| EventType (0-7) | NewRevision (8-31) | +| --------------- | ---------------------------------- | +| `1F` | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX}` | | Data Type | Name | Label | Description | | ----------------- | ----------- | ----- | ------------------------------------ | @@ -602,11 +565,11 @@ This event is emitted the files of the game match a known new store manifest/rev - [UPDATE_COMMANDLINE_V0][update-commandline] -### {01}+03: UpdateCommandline8 +### [20] UpdateCommandline8 -| EventType (0-7) | Length (8-15) | -| ---------------- | ------------- | -| 43 (`{01} + 03`) | `{XXXXXXXX}` | +| EventType (0-7) | Length (8-15) | +| --------------- | ------------- | +| `20` | `{XXXXXXXX}` | | Data Type | Name | Label | Description | | --------- | ------ | ----- | ---------------------------------------------------------------------------------------------------------------- | @@ -620,11 +583,11 @@ This event is emitted the files of the game match a known new store manifest/rev - [EXTERNAL_CONFIG_UPDATED_V0][external-config-updated-v0] -### {10}+0C: ExternalConfigUpdated24 +### [21] ExternalConfigUpdated24 -| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | PackageIdIdx (44-63) | -| ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | -| 8C (`{10} + 0C`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | +| EventType (0-7) | PathIndex (8-14) | FileIndex (15-21) | PackageIdIdx (22-31) | +| --------------- | ---------------- | ----------------- | -------------------- | +| `21` | `{XXX} {XXXX}` | `{YYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ}` | | Data Type | Name | Label | Description | | ---------------------- | -------------- | ----- | ------------------------------------------------------------------------------------ | @@ -632,11 +595,11 @@ This event is emitted the files of the game match a known new store manifest/rev | `u7` | FileIndex | Y | [0-127] Index of file path in [external-config-paths.bin][external-config-paths-bin] | | `u10` ([PackageIdIdx]) | [PackageIdIdx] | Z | [0-1024] Index of package ID in [package-ids.bin][package-ids.bin] | -### {11}+07: ExternalConfigUpdated56 +### [22] ExternalConfigUpdated56 -| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | [PackageIdIdx] (44-63) | -| ---------------- | ---------------------------- | ---------------------------- | ------------------------------ | -| C7 (`{11} + 07`) | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | +| EventType (0-7) | PathIndex (8-25) | FileIndex (26-43) | [PackageIdIdx] (44-63) | +| --------------- | ---------------------------- | ---------------------------- | ------------------------------ | +| `22` | `{XXXXXXXX} {XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY} {YYYY}` | `{ZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | | Data Type | Name | Label | Description | | -------------------- | ------------ | ----- | ------------------------------------------------------------------------------------- | @@ -644,21 +607,216 @@ This event is emitted the files of the game match a known new store manifest/rev | `u18` | FileIndex | Y | [0-256K] Index of file path in [external-config-paths.bin][external-config-paths-bin] | | `u20` (PackageIdIdx) | PackageIdIdx | Z | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | -### {11}+08: ExternalConfigUpdatedFull +## PackageEnabled + +!!! info "A package has been enabled." + + This is analogous to enabling a mod in the loadout. + + This is an optimization over the full length ([PackageStatusChanged](#packagestatuschanged)) event; + +Usually a user would just enable a mod and leave it at that. So [PackageEnabled](#packageenabled) (this) +is expected to be a bit more common than [PackageDisabled](#packagedisabled). + +For this reason, the two events have been separated, in case we want to have different event type +opcode ranges for them. + +### Messages + +`PackageType` is the type of package referred to by package at [PackageIdIdx]. + +- [PACKAGE_ENABLED_V0][package-enabled-v0] `PackageType` is not known. +- [MOD_ENABLED_V0][mod-enabled-v0] `PackageType == Mod`. +- [TRANSLATION_ENABLED_V0][translation-enabled-v0] `PackageType == Translation`. +- [TOOL_ENABLED_V0][tool-enabled-v0] when `PackageType == Tool`. + +### [6B] - [9A] PackageEnabled8 + +| EventType (0-7) | [PackageIdIdx] (8-15) | +| --------------- | --------------------- | +| `6B` - `9A` | `{XXXXXXXX}` | + +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | ----------------------------------------------------------------- | +| `u8` ([PackageIdIdx]) | [PackageIdIdx] | X | [0-255] Index of package ID in [package-ids.bin][package-ids.bin] | + +The `EventType` has 48 reserved values, each for a different range of [PackageIdIdx]. + +They function as follows: + +- `6B` [PackageIdIdx] has range [0-255]. +- `6C` [PackageIdIdx] has range [256-511]. +- ... +- `9A` [PackageIdIdx] has range [12032-12287]. + +i.e. Each `EventType` has a range of 256 package IDs. + +!!! info "There is no `PackageEnabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." + +## PackageDisabled + +!!! info "A package has been disabled." + + This is analogous to disabling a mod in the loadout. + + This is an optimization over the full length ([PackageStatusChanged](#packagestatuschanged)) event; + +Please also see optimization note at start of [PackageEnabled](#packageenabled). + +### Messages + +`PackageType` is the type of package referred to by package at [PackageIdIdx]. + +- [PACKAGE_DISABLED_V0][package-disabled-v0] when `PackageType` is not known. +- [MOD_DISABLED_V0][mod-disabled-v0] when `PackageType == Mod`. +- [TRANSLATION_DISABLED_V0][translation-disabled-v0] when `PackageType == Translation`. +- [TOOL_DISABLED_V0][tool-disabled-v0] when `PackageType == Tool`. + +### [9B] - [CA] PackageDisabled8 + +| EventType (0-7) | [PackageIdIdx] (8-15) | +| --------------- | --------------------- | +| `9B` - `CA` | `{XXXXXXXX}` | + +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | ----------------------------------------------------------------- | +| `u8` ([PackageIdIdx]) | [PackageIdIdx] | X | [0-255] Index of package ID in [package-ids.bin][package-ids.bin] | + +The `EventType` has 16 reserved values, each for a different range of [PackageIdIdx]. + +They function as follows: + +- `9B` [PackageIdIdx] has range [0-255]. +- `9C` [PackageIdIdx] has range [256-511]. +- ... +- `CA` [PackageIdIdx] has range [12032-12287]. + +i.e. Each `EventType` has a range of 256 package IDs. + +!!! info "There is no `PackageDisabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." + +## PackageAdded + +!!! info "This is a specialized case for adding a new package." + + This is equivalent to adding a new package ([PackageStatusChanged](#packagestatuschanged)) and + setting its initial version ([PackageVersionChanged](#packageversionchanged)) in a single operation. + +### Messages + +- [PACKAGE_ADDED_V0][package-added-v0] when `PackageType` is not known. +- [MOD_ADDED_V0][mod-added-v0] when `PackageType == Mod`. +- [TRANSLATION_ADDED_V0][translation-added-v0] when `PackageType == Translation`. +- [TOOL_ADDED_V0][tool-added-v0] when `PackageType == Tool`. + +### [CB] - [CD] PackageAdded24 + +| EventType (0-7) | [PackageVerIdx] (8-17) | [PackageIdIdx] (18-31) | +| --------------- | ----------------------- | ---------------------- | +| `CB` - `CD` | `{XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY}` | + +| Data Type | Name | Label | Description | +| ----------------------- | --------------- | ----- | ----------------------------------------------------------------------------- | +| `u10` ([PackageVerIdx]) | [PackageVerIdx] | X | [0-1023] Index of the version in [package-versions.bin][package-versions.bin] | +| `u14` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-32767] Index of package ID in [package-ids.bin][package-ids.bin] | + +The `EventType` has 8 reserved values to act as an offset to the [PackageVerIdx] field. + +Reference: + +- `CB` [PackageVerIdx] has range [0-1023]. +- `CC` [PackageVerIdx] has range [1024-2047]. +- `CD` [PackageVerIdx] has range [2048-3071]. + +### [CE] PackageAddedFull + +| EventType (0-7) | Reserved (8-23) | NewStatus (24-43) | PackageIdIdx (44-63) | +| --------------- | --------------- | ----------------------------------- | -------------------------------- | +| `CE` | `CE CE` | `{XX} {XXXXXXXX} {XXXXXXXX}` `{XX}` | `{YYYYYY} {YYYYYYYY} {YYYYYYYY}` | + +| Data Type | Name | Label | Description | +| ----------------------- | --------------- | ----- | --------------------------------------------------------------------------- | +| `u16` | Reserved | CE | Reserved. Constant `CE` to improve compression. | +| `u20` ([PackageVerIdx]) | [PackageVerIdx] | X | [0-1M] Index of the version in [package-versions.bin][package-versions.bin] | +| `u20` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | + +The `EventType` has 8 reserved values to act as an offset to the [PackageVerIdx] field. + +## PackageAddedWithConfig + +!!! info "This is a specialized case that also updates config directly after [adding a versioned package](#packageadded)." + +### Messages + +First message is one of the following: + +- [PACKAGE_ADDED_WITH_CONFIG_V0][package-added-with-config-v0] when `PackageType` is not known. +- [MOD_ADDED_WITH_CONFIG_V0][mod-added-with-config-v0] when `PackageType == Mod`. +- [TRANSLATION_ADDED_WITH_CONFIG_V0][translation-added-with-config-v0] when `PackageType == Translation`. +- [TOOL_ADDED_WITH_CONFIG_V0][tool-added-with-config-v0] when `PackageType == Tool`. + +When the exact changes are not known, the event is [written as V1][commit-message-versioning]: + +- [PACKAGE_ADDED_WITH_CONFIG_V1][package-added-with-config-v1] when `PackageType` is not known. +- [MOD_ADDED_WITH_CONFIG_V1][mod-added-with-config-v1] when `PackageType == Mod`. +- [TRANSLATION_ADDED_WITH_CONFIG_V1][translation-added-with-config-v1] when `PackageType == Translation`. +- [TOOL_ADDED_WITH_CONFIG_V1][tool-added-with-config-v1] when `PackageType == Tool`. + +### [CF] PackageAddedWithConfig + +!!! info "This also sets the initial config file (i.e. [ConfigUpdated](#configupdated)) when the package is added." + + In the case the user downloads a package and configures it directly after. + +| EventType (0-7) | [ConfigIdx] (8-23) | NewStatus (24-43) | PackageIdIdx (44-63) | +| --------------- | --------------------- | ----------------------------------- | -------------------------------- | +| `CF` | `{XXXXXXXX XXXXXXXX}` | `{YY} {YYYYYYYY} {YYYYYYYY}` `{YY}` | `{ZZZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | + +| Data Type | Name | Label | Description | +| ----------------------- | --------------- | ----- | --------------------------------------------------------------------------- | +| `u16` ([ConfigIdx]) | [ConfigIdx] | X | [0-65535] Index of associated configuration in [config.bin][configbin] | +| `u20` ([PackageVerIdx]) | [PackageVerIdx] | Y | [0-1M] Index of the version in [package-versions.bin][package-versions.bin] | +| `u20` ([PackageIdIdx]) | [PackageIdIdx] | Z | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | + +## PackageVersion100Added + +!!! info "`1.0.0` is by far the most common version for a new package." + + Because many mods don't get updated. So we can exploit this for some additional compression wins. + +This is an optimized form of [PackageAdded](#packageadded) for the common case of adding a new +package with version `1.0.0`. + +### Messages + +- [PACKAGE_ADDED_V0][package-added-v0] when `PackageType` is not known. +- [MOD_ADDED_V0][mod-added-v0] when `PackageType == Mod`. +- [TRANSLATION_ADDED_V0][translation-added-v0] when `PackageType == Translation`. +- [TOOL_ADDED_V0][tool-added-v0] when `PackageType == Tool`. + +### [D0] - [FF] PackageAddedVersion100_8 + +| EventType (0-7) | [PackageIdIdx] (8-15) | +| --------------- | --------------------- | +| `D0` - `FF` | `{XXXXXXXX}` | + +| Data Type | Name | Label | Description | +| --------------------- | -------------- | ----- | ----------------------------------------------------------------- | +| `u8` ([PackageIdIdx]) | [PackageIdIdx] | X | [0-255] Index of package ID in [package-ids.bin][package-ids.bin] | + +The `EventType` has 48 reserved values to act as an offset to the [PackageIdIdx] field. -| EventType (0-7) | ExternalConfigIdx (8-35) | PackageIdIdx (36-63) | -| ---------------- | ----------------------------------------- | ----------------------------------------- | -| C8 (`{11} + 08`) | `{XXXXXXXX} {XXXXXXXX} {XXXXXXXX} {XXXX}` | `{YYYY} {YYYYYYYY} {YYYYYYYY} {YYYYYYYY}` | +Reference: -| Data Type | Name | Label | Description | -| ---------------------- | ----------------- | ----- | ----------------------------------------------------------------------------------------- | -| `u28` (ExtConfigIdx) | ExternalConfigIdx | X | [0-268M] Index of extended reference in [external-config-refs.bin][external-config-refs]. | -| `u28` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-268M] Index of package ID in [package-ids.bin][package-ids.bin]. | +- `D0` [PackageIdIdx] has range [0-255]. +- ... +- `FF` [PackageIdIdx] has range [12032-12287]. [configbin]: Unpacked.md#configbin [events-bin]: Unpacked.md#eventsbin [packagemetadatabin]: Unpacked.md#packages [package-ids.bin]: Unpacked.md#package-idsbin +[package-versions.bin]: ./Unpacked.md#package-versionsbin [package-added-v0]: ./Commit-Messages.md#package_added_v0 [package-removed-v0]: ./Commit-Messages.md#package_removed_v0 [package-hidden-v0]: ./Commit-Messages.md#package_hidden_v0 @@ -704,6 +862,7 @@ This event is emitted the files of the game match a known new store manifest/rev [update-commandline]: ./Commit-Messages.md#update_commandline_v0 [featuresbin]: ./About.md#featuresbin [pkgstatechange]: ./DataTypes.md#packagestatechange +[PackageStateChange]: ./DataTypes.md#packagestatechange [sortingmode]: ./DataTypes.md#sortingmode [sortorder]: ./DataTypes.md#sortorder [griddisplaymode]: ./DataTypes.md#griddisplaymode @@ -718,4 +877,23 @@ This event is emitted the files of the game match a known new store manifest/rev [external-config-paths-bin]: ./Unpacked.md#external-config-pathsbin [external-config-refs]: ./Unpacked.md#external-config-refsbin [external-config-updated-v0]: ./Commit-Messages.md#external_config_updated_v0 -[PackageIdIdx]: ./DataTypes.md \ No newline at end of file +[PackageIdIdx]: ./DataTypes.md +[PackageVerIdx]: ./DataTypes.md +[PackageEnabledStateChange]: ./DataTypes.md#packageenabledstatechange +[PackageInstallStateChange]: ./DataTypes.md#packageinstallstatechange +[Snapshots]: ./Snapshot.md +[ConfigIdx]: ./DataTypes.md +[ExternalConfigIdx]: ./DataTypes.md +[package-added-with-config-v0]: ./Commit-Messages.md#package-added-with-config-v0 +[mod-added-with-config-v0]: ./Commit-Messages.md#mod_added_with_config_v0 +[translation-added-with-config-v0]: ./Commit-Messages.md#translation_added_with_config_v0 +[tool-added-with-config-v0]: ./Commit-Messages.md#tool_added_with_config_v0 +[package-added-with-config-v1]: ./Commit-Messages.md#package_added_with_config_v1 +[mod-added-with-config-v1]: ./Commit-Messages.md#mod_added_with_config_v1 +[translation-added-with-config-v1]: ./Commit-Messages.md#translation_added_with_config_v1 +[tool-added-with-config-v1]: ./Commit-Messages.md#tool_added_with_config_v1 +[PackageAddedFull]: #ce-packageaddedfull +[GameLaunched]: #gamelaunched +[timestamps.bin]: ./Unpacked.md#timestampsbin +[GameLaunchedN]: ./Commit-Messages.md#game_launched_n_v0 +[ConfigUpdated24]: #21-externalconfigupdated24 \ No newline at end of file diff --git a/docs/Server/Storage/Loadouts/File-Format/Packed.md b/docs/Server/Storage/Loadouts/File-Format/Packed.md index 3e6deaa..79ef020 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Packed.md +++ b/docs/Server/Storage/Loadouts/File-Format/Packed.md @@ -48,7 +48,6 @@ The following tree structure represents the files included in the packed .nx arc β”œβ”€β”€ external-config.bin β”œβ”€β”€ external-config-data.bin β”œβ”€β”€ external-config-paths.bin -β”œβ”€β”€ external-config-refs.bin β”œβ”€β”€ package-ids.bin β”œβ”€β”€ package-versions-len.bin β”œβ”€β”€ package-versions.bin diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index 575561f..b5fe88e 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -185,20 +185,6 @@ Each path is a path *relative* to the path specified in [Config File Paths][conf If the `Type` is `File`. An empty path would be used instead. -### external-config-refs.bin - -!!! info "This is a buffer of `references` to [external-config-paths.bin](#external-config-pathsbin) and [external-config.bin](#external-configbin)" - -This is an array of the following structure - -| Data Type | Name | Description | -| --------- | --------- | ----------------------------------------------------------------------------------- | -| `u32` | FileIndex | [0-4M] Index of file in [external-config.bin](#external-configbin) | -| `u32` | PathIndex | [0-4M] Index of file path in [external-config-paths.bin](#external-config-pathsbin) | - -This file is used in the event that one of the components of [ExternalConfigUpdated56][event-externalconfigupdated] -is out of range and cannot fit into the encoding. - ## Packages !!! info "We refer to a 'unique package' by [XXH3(PackageID)][hashing]" @@ -478,7 +464,7 @@ Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. Which could be marked as: ``` -Added '**Super Cool Mod**' with ID '**reloaded3.utility.scmexample**' and version '**1.0.0**' +Added '**Super Cool Mod**' with ID '**reloaded3.utility.somexample**' and version '**1.0.0**' ``` #### Encoding @@ -492,7 +478,7 @@ This would be encoded as: Explanation: - 0: UTF-8 Char Array for "Super Cool Mod" - - 0: UTF-8 Char Array for "reloaded3.utility.scmexample" + - 0: UTF-8 Char Array for "reloaded3.utility.somexample" - 0: UTF-8 Char Array for "1.0.0" 2. [commit-parameters-lengths.bin][commitparam8len]: [14, 28, 5] @@ -500,13 +486,13 @@ This would be encoded as: Explanation: - 14: Length of "Super Cool Mod" - - 28: Length of "reloaded3.utility.scmexample" + - 28: Length of "reloaded3.utility.somexample" - 5: Length of "1.0.0" 3. [commit-parameters-text.bin](#parametertype): - `Super Cool Mod` - - `reloaded3.utility.scmexample` + - `reloaded3.utility.somexample` - `1.0.0` These strings are written directly to the `commit-parameters-text.bin` file, without any null @@ -529,7 +515,7 @@ This would be encoded as: Explanation: - 5: BackReference8 for `"Super Cool Mod"` - - 5: BackReference8 for `"reloaded3.utility.scmexample"` + - 5: BackReference8 for `"reloaded3.utility.somexample"` - 5: BackReference8 for `"1.0.0"` 2. [commit-parameters-backrefs-8.bin](#back-references): [0, 1, 2] @@ -537,7 +523,7 @@ This would be encoded as: Explanation: - 0: Index of `"Super Cool Mod"` - - 1: Index of `"reloaded3.utility.scmexample"` + - 1: Index of `"reloaded3.utility.somexample"` - 2: Index of `"1.0.0"` 3. [commit-parameters-versions.bin](#commit-parameters-versionsbin): [0] @@ -565,7 +551,7 @@ Let's say we want to reference all three parameters from the previous example in Explanation: - 0: Index of `"Super Cool Mod"` - - 1: Index of `"reloaded3.utility.scmexample"` + - 1: Index of `"reloaded3.utility.somexample"` - 2: Index of `"1.0.0"` 3. [commit-parameters-versions.bin](#commit-parameters-versionsbin): [0] @@ -846,7 +832,7 @@ as regular parameters in [Commit Parameters](#commit-parameters). [commit-param-backrefs]: ./Unpacked.md#back-references [commit-param-lists]: ./Unpacked.md#parameter-lists [event-packagestatuschanged]: ./Events.md#packagestatuschanged -[event-externalconfigupdated]: ./Events.md#1107-externalconfigupdated56 +[event-externalconfigupdated]: ./Events.md#22-externalconfigupdated56 [package-added]: ./Commit-Messages.md#package_added_v0 [snapshots]: ./Snapshot.md [config-files]: ../../../Packaging/Package-Metadata.md#config-files From f806d81d674249f14e47db2a4a965dd56d4c03f2 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 03:56:09 +0100 Subject: [PATCH 09/17] Changed: Shifted the opcodes down, as to leave no space. --- .../Storage/Loadouts/File-Format/Events.md | 65 ++++++++----------- 1 file changed, 26 insertions(+), 39 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index d2f63a3..6283f0e 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -630,11 +630,11 @@ opcode ranges for them. - [TRANSLATION_ENABLED_V0][translation-enabled-v0] `PackageType == Translation`. - [TOOL_ENABLED_V0][tool-enabled-v0] when `PackageType == Tool`. -### [6B] - [9A] PackageEnabled8 +### [23] - [52] PackageEnabled8 | EventType (0-7) | [PackageIdIdx] (8-15) | | --------------- | --------------------- | -| `6B` - `9A` | `{XXXXXXXX}` | +| `23` - `52` | `{XXXXXXXX}` | | Data Type | Name | Label | Description | | --------------------- | -------------- | ----- | ----------------------------------------------------------------- | @@ -644,12 +644,10 @@ The `EventType` has 48 reserved values, each for a different range of [PackageId They function as follows: -- `6B` [PackageIdIdx] has range [0-255]. -- `6C` [PackageIdIdx] has range [256-511]. +- `23` [PackageIdIdx] has range [0-255]. +- `24` [PackageIdIdx] has range [256-511]. - ... -- `9A` [PackageIdIdx] has range [12032-12287]. - -i.e. Each `EventType` has a range of 256 package IDs. +- `52` [PackageIdIdx] has range [12032-12287]. !!! info "There is no `PackageEnabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." @@ -672,26 +670,20 @@ Please also see optimization note at start of [PackageEnabled](#packageenabled). - [TRANSLATION_DISABLED_V0][translation-disabled-v0] when `PackageType == Translation`. - [TOOL_DISABLED_V0][tool-disabled-v0] when `PackageType == Tool`. -### [9B] - [CA] PackageDisabled8 +### [53] - [82] PackageDisabled8 | EventType (0-7) | [PackageIdIdx] (8-15) | | --------------- | --------------------- | -| `9B` - `CA` | `{XXXXXXXX}` | - -| Data Type | Name | Label | Description | -| --------------------- | -------------- | ----- | ----------------------------------------------------------------- | -| `u8` ([PackageIdIdx]) | [PackageIdIdx] | X | [0-255] Index of package ID in [package-ids.bin][package-ids.bin] | +| `53` - `82` | `{XXXXXXXX}` | -The `EventType` has 16 reserved values, each for a different range of [PackageIdIdx]. +The `EventType` has 48 reserved values, each for a different range of [PackageIdIdx]. They function as follows: -- `9B` [PackageIdIdx] has range [0-255]. -- `9C` [PackageIdIdx] has range [256-511]. +- `53` [PackageIdIdx] has range [0-255]. +- `54` [PackageIdIdx] has range [256-511]. - ... -- `CA` [PackageIdIdx] has range [12032-12287]. - -i.e. Each `EventType` has a range of 256 package IDs. +- `82` [PackageIdIdx] has range [12032-12287]. !!! info "There is no `PackageDisabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." @@ -709,34 +701,29 @@ i.e. Each `EventType` has a range of 256 package IDs. - [TRANSLATION_ADDED_V0][translation-added-v0] when `PackageType == Translation`. - [TOOL_ADDED_V0][tool-added-v0] when `PackageType == Tool`. -### [CB] - [CD] PackageAdded24 +### [83] - [85] PackageAdded24 | EventType (0-7) | [PackageVerIdx] (8-17) | [PackageIdIdx] (18-31) | | --------------- | ----------------------- | ---------------------- | -| `CB` - `CD` | `{XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY}` | - -| Data Type | Name | Label | Description | -| ----------------------- | --------------- | ----- | ----------------------------------------------------------------------------- | -| `u10` ([PackageVerIdx]) | [PackageVerIdx] | X | [0-1023] Index of the version in [package-versions.bin][package-versions.bin] | -| `u14` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-32767] Index of package ID in [package-ids.bin][package-ids.bin] | +| `83` - `85` | `{XXXXXXXX} {XX}` | `{YYYYYY} {YYYYYYYY}` | -The `EventType` has 8 reserved values to act as an offset to the [PackageVerIdx] field. +The `EventType` has 3 reserved values to act as an offset to the [PackageVerIdx] field. Reference: -- `CB` [PackageVerIdx] has range [0-1023]. -- `CC` [PackageVerIdx] has range [1024-2047]. -- `CD` [PackageVerIdx] has range [2048-3071]. +- `83` [PackageVerIdx] has range [0-1023]. +- `84` [PackageVerIdx] has range [1024-2047]. +- `85` [PackageVerIdx] has range [2048-3071]. -### [CE] PackageAddedFull +### [86] PackageAddedFull | EventType (0-7) | Reserved (8-23) | NewStatus (24-43) | PackageIdIdx (44-63) | | --------------- | --------------- | ----------------------------------- | -------------------------------- | -| `CE` | `CE CE` | `{XX} {XXXXXXXX} {XXXXXXXX}` `{XX}` | `{YYYYYY} {YYYYYYYY} {YYYYYYYY}` | +| `86` | `86 86` | `{XX} {XXXXXXXX} {XXXXXXXX}` `{XX}` | `{YYYYYY} {YYYYYYYY} {YYYYYYYY}` | | Data Type | Name | Label | Description | | ----------------------- | --------------- | ----- | --------------------------------------------------------------------------- | -| `u16` | Reserved | CE | Reserved. Constant `CE` to improve compression. | +| `u16` | Reserved | 86 | Reserved. Constant `CE` to improve compression. | | `u20` ([PackageVerIdx]) | [PackageVerIdx] | X | [0-1M] Index of the version in [package-versions.bin][package-versions.bin] | | `u20` ([PackageIdIdx]) | [PackageIdIdx] | Y | [0-1M] Index of package ID in [package-ids.bin][package-ids.bin] | @@ -762,7 +749,7 @@ When the exact changes are not known, the event is [written as V1][commit-messag - [TRANSLATION_ADDED_WITH_CONFIG_V1][translation-added-with-config-v1] when `PackageType == Translation`. - [TOOL_ADDED_WITH_CONFIG_V1][tool-added-with-config-v1] when `PackageType == Tool`. -### [CF] PackageAddedWithConfig +### [87] PackageAddedWithConfig !!! info "This also sets the initial config file (i.e. [ConfigUpdated](#configupdated)) when the package is added." @@ -770,7 +757,7 @@ When the exact changes are not known, the event is [written as V1][commit-messag | EventType (0-7) | [ConfigIdx] (8-23) | NewStatus (24-43) | PackageIdIdx (44-63) | | --------------- | --------------------- | ----------------------------------- | -------------------------------- | -| `CF` | `{XXXXXXXX XXXXXXXX}` | `{YY} {YYYYYYYY} {YYYYYYYY}` `{YY}` | `{ZZZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | +| `87` | `{XXXXXXXX XXXXXXXX}` | `{YY} {YYYYYYYY} {YYYYYYYY}` `{YY}` | `{ZZZZZZ} {ZZZZZZZZ} {ZZZZZZZZ}` | | Data Type | Name | Label | Description | | ----------------------- | --------------- | ----- | --------------------------------------------------------------------------- | @@ -794,11 +781,11 @@ package with version `1.0.0`. - [TRANSLATION_ADDED_V0][translation-added-v0] when `PackageType == Translation`. - [TOOL_ADDED_V0][tool-added-v0] when `PackageType == Tool`. -### [D0] - [FF] PackageAddedVersion100_8 +### [88] - [B7] PackageAddedVersion100_8 | EventType (0-7) | [PackageIdIdx] (8-15) | | --------------- | --------------------- | -| `D0` - `FF` | `{XXXXXXXX}` | +| `88` - `B7` | `{XXXXXXXX}` | | Data Type | Name | Label | Description | | --------------------- | -------------- | ----- | ----------------------------------------------------------------- | @@ -808,9 +795,9 @@ The `EventType` has 48 reserved values to act as an offset to the [PackageIdIdx] Reference: -- `D0` [PackageIdIdx] has range [0-255]. +- `88` [PackageIdIdx] has range [0-255]. - ... -- `FF` [PackageIdIdx] has range [12032-12287]. +- `B7` [PackageIdIdx] has range [12032-12287]. [configbin]: Unpacked.md#configbin [events-bin]: Unpacked.md#eventsbin From 1fe6f0f2bba6352d992ab0095c432c251c275ed1 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 15:54:16 +0100 Subject: [PATCH 10/17] Updated: Read/Write Events Examples --- .../Storage/Loadouts/File-Format/Events.md | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index 6283f0e..77e41e6 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -30,46 +30,43 @@ We'll use a mix of different event types to show various scenarios. ### Writing Events -!!! info "The event types used here are be outdated. This is just an example only." - Suppose we want to write the following sequence of events: -1. [GameLaunched][GameLaunched] (1 byte) -2. `PackageStatusChanged8` (2 bytes) -3. [ConfigUpdated8](#0101-configupdated8) (2 bytes) -4. [PackageUpdated16](#1005-packageupdated16) (4 bytes) +1. [GameLaunched] (1 byte) +2. [PackageStatusChanged24] (4 bytes) +3. [ConfigUpdated24] (2 bytes) +4. [PackageUpdated16] (4 bytes) Here's how these events would be written to the file: -| Byte 0-7 | Meaning | -| ------------- | -------------------------------------------------- | -| `01` | [GameLaunched][GameLaunched] event | -| `40 ??` | `PackageStatusChanged8` event | -| `41 ??` | [ConfigUpdated8](#0101-configupdated8) event | -| `00 00 00` | NOP padding to align next event to 8-byte boundary | -| `85 85 ?? ??` | [PackageUpdated16](#1005-packageupdated16) event | +| Byte 0-7 | Meaning | +| ------------- | ------------------------------ | +| `02` | [GameLaunched] event | +| `01 ?? ?? ??` | [PackageStatusChanged24] event | +| `00 00 00` | NOP padding | +| `04 ?? ?? ??` | [ConfigUpdated24] event | +| `17 ?? ?? ??` | [PackageUpdated24] event | Explanation: -- The [GameLaunched][GameLaunched] event (`01`) takes 1 byte. -- The `PackageStatusChanged8` event (`40 XX`) takes 2 bytes. -- The [ConfigUpdated8](#0101-configupdated8) event (`41 XX`) takes 2 bytes. -- At this point, we've written 5 bytes. To ensure the next 4-byte event ([PackageUpdated16](#1005-packageupdated16)) starts on an 8-byte boundary, we add 3 bytes of [NOP](#00-nop) padding (`00 00 00`). -- Finally, we write the [PackageUpdated16](#1005-packageupdated16) event (`85 85 XX XX`), which takes 4 bytes. +- The [GameLaunched] event (`02`) takes 1 byte. +- The [PackageStatusChanged24] event (`01 XX XX XX`) takes 4 bytes. +- After these 5 bytes, we add 3 bytes of [NOP] padding (`00 00 00`) to align to the 8-byte boundary. +- The [ConfigUpdated24] event (`04 XX XX XX`) takes 4 bytes. +- The [PackageUpdated24] event (`17 XX XX XX`) takes 4 bytes. ### Reading Events When reading these events, the system would perform full 8-byte reads: -1. First read (8 bytes): `01 40 ?? 41 ?? 00 00 00` - - Processes [GameLaunched][GameLaunched] (1 byte) - - Processes `PackageStatusChanged8` (2 bytes) - - Processes [ConfigUpdated8](#0101-configupdated8) (2 bytes) +1. First read (8 bytes): `02 01 XX XX XX 00 00 00` + - Processes [GameLaunched] (1 byte) + - Processes [PackageStatusChanged24] (4 bytes) - Skips NOP padding (3 bytes) -2. Second read (8 bytes): `85 85 ?? ?? ?? ?? ?? ??` - - Processes [PackageUpdated16](#1005-packageupdated16) (4 bytes) - - The last 2 bytes (XX XX) would be the start of the next event or additional padding +2. Second read (8 bytes): `04 XX XX XX 17 XX XX XX` + - Processes [ConfigUpdated24] (4 bytes) + - Processes [PackageUpdated24] (4 bytes) ## Optimizing for Compression @@ -649,7 +646,7 @@ They function as follows: - ... - `52` [PackageIdIdx] has range [12032-12287]. -!!! info "There is no `PackageEnabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." +!!! info "There is no `PackageEnabled24` variant, instead use [PackageStatusChanged24][PackageStatusChanged24]." ## PackageDisabled @@ -685,7 +682,7 @@ They function as follows: - ... - `82` [PackageIdIdx] has range [12032-12287]. -!!! info "There is no `PackageDisabled24` variant, instead use [PackageStatusChanged24](#packagestatuschanged)." +!!! info "There is no `PackageDisabled24` variant, instead use [PackageStatusChanged24]." ## PackageAdded @@ -883,4 +880,7 @@ Reference: [GameLaunched]: #gamelaunched [timestamps.bin]: ./Unpacked.md#timestampsbin [GameLaunchedN]: ./Commit-Messages.md#game_launched_n_v0 -[ConfigUpdated24]: #21-externalconfigupdated24 \ No newline at end of file +[ConfigUpdated24]: #21-externalconfigupdated24 +[PackageStatusChanged24]: #packagestatuschanged +[PackageUpdated16]: #1005-packageupdated16 +[NOP]: #00-nop \ No newline at end of file From d483412e4615fbd512b27005b9ad841182dce517 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 16:04:04 +0100 Subject: [PATCH 11/17] Added: PackageUpdated24 event to example --- docs/Server/Storage/Loadouts/File-Format/Events.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Events.md b/docs/Server/Storage/Loadouts/File-Format/Events.md index 77e41e6..99a50e9 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Events.md +++ b/docs/Server/Storage/Loadouts/File-Format/Events.md @@ -35,7 +35,7 @@ Suppose we want to write the following sequence of events: 1. [GameLaunched] (1 byte) 2. [PackageStatusChanged24] (4 bytes) 3. [ConfigUpdated24] (2 bytes) -4. [PackageUpdated16] (4 bytes) +4. [PackageUpdated24] (4 bytes) Here's how these events would be written to the file: @@ -221,6 +221,8 @@ This event has no extra data. ### [03] GameLaunchedN +!!! info "a.k.a. `GameLaunched8`" + !!! info "This is equivalent to repeating [GameLaunched] event `N` times." | EventType (0-7) | Length (8-15) | @@ -882,5 +884,5 @@ Reference: [GameLaunchedN]: ./Commit-Messages.md#game_launched_n_v0 [ConfigUpdated24]: #21-externalconfigupdated24 [PackageStatusChanged24]: #packagestatuschanged -[PackageUpdated16]: #1005-packageupdated16 -[NOP]: #00-nop \ No newline at end of file +[NOP]: #00-nop +[PackageUpdated24]: #17-packageupdated24 \ No newline at end of file From 73373f2add1ee362a0f17f03e8a5999182ec5ced Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 16:21:30 +0100 Subject: [PATCH 12/17] Fixed: Incorrect event types in `commit-parameter-types.bin` example. --- docs/Server/Storage/Loadouts/File-Format/Unpacked.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index b5fe88e..f1ea8bc 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -510,7 +510,7 @@ This would be encoded as: !!! info "Suppose you wanted to repeat the earlier parameter, we would use [back references](#back-references)." -1. [commit-parameter-types.bin](#commit-parameters-typesbin): [7, 7, 7] +1. [commit-parameter-types.bin](#commit-parameters-typesbin): [5, 5, 5] Explanation: From 82a81e99a0f91cdac0a864302316554724fb383f Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 18:59:00 +0100 Subject: [PATCH 13/17] Fixed: Commit Messages Encoding Example Not Accounting Contextual Parameter --- .../Storage/Loadouts/File-Format/Unpacked.md | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index f1ea8bc..80f2046 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -467,33 +467,34 @@ Which could be marked as: Added '**Super Cool Mod**' with ID '**reloaded3.utility.somexample**' and version '**1.0.0**' ``` +!!! note "The `Version` is a ['Contextual Parameter'][commit-messages-contextual], and thus is derived from context." + + It is not stored in the commit parameters. + #### Encoding -!!! note "Parameters are encoded in the order in which they appear in the template!!" +!!! tip "Parameters are encoded in the order in which they appear in the template!!" This would be encoded as: -1. [commit-parameter-types.bin](#commit-parameters-typesbin): [0, 0, 0] +1. [commit-parameter-types.bin](#commit-parameters-typesbin): [0, 0] Explanation: - - 0: UTF-8 Char Array for "Super Cool Mod" - - 0: UTF-8 Char Array for "reloaded3.utility.somexample" - - 0: UTF-8 Char Array for "1.0.0" + - 0: UTF-8 Char Array for "Super Cool Mod" (Name) + - 0: UTF-8 Char Array for "reloaded3.utility.somexample" (ID) -2. [commit-parameters-lengths.bin][commitparam8len]: [14, 28, 5] +2. [commit-parameters-lengths.bin][commitparam8len]: [14, 28] Explanation: - 14: Length of "Super Cool Mod" - 28: Length of "reloaded3.utility.somexample" - - 5: Length of "1.0.0" 3. [commit-parameters-text.bin](#parametertype): - `Super Cool Mod` - `reloaded3.utility.somexample` - - `1.0.0` These strings are written directly to the `commit-parameters-text.bin` file, without any null terminator or padding. @@ -510,21 +511,19 @@ This would be encoded as: !!! info "Suppose you wanted to repeat the earlier parameter, we would use [back references](#back-references)." -1. [commit-parameter-types.bin](#commit-parameters-typesbin): [5, 5, 5] +1. [commit-parameter-types.bin](#commit-parameters-typesbin): [5, 5] Explanation: - 5: BackReference8 for `"Super Cool Mod"` - 5: BackReference8 for `"reloaded3.utility.somexample"` - - 5: BackReference8 for `"1.0.0"` -2. [commit-parameters-backrefs-8.bin](#back-references): [0, 1, 2] +2. [commit-parameters-backrefs-8.bin](#back-references): [0, 1] Explanation: - 0: Index of `"Super Cool Mod"` - 1: Index of `"reloaded3.utility.somexample"` - - 2: Index of `"1.0.0"` 3. [commit-parameters-versions.bin](#commit-parameters-versionsbin): [0] @@ -546,13 +545,12 @@ Let's say we want to reference all three parameters from the previous example in - 11: BackReference3_8 for all three parameters -2. [commit-parameters-backrefs-8.bin](#back-references): [0, 1, 2] +2. [commit-parameters-backrefs-8.bin](#back-references): [0, 1] Explanation: - 0: Index of `"Super Cool Mod"` - 1: Index of `"reloaded3.utility.somexample"` - - 2: Index of `"1.0.0"` 3. [commit-parameters-versions.bin](#commit-parameters-versionsbin): [0] From a5b9bd7222b6769b97555335fe111254c64226e7 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 19:07:04 +0100 Subject: [PATCH 14/17] Fixed: Message Encode Example Points at Correct Param Lengths File --- docs/Server/Storage/Loadouts/File-Format/Unpacked.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index 80f2046..df63e38 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -484,7 +484,7 @@ This would be encoded as: - 0: UTF-8 Char Array for "Super Cool Mod" (Name) - 0: UTF-8 Char Array for "reloaded3.utility.somexample" (ID) -2. [commit-parameters-lengths.bin][commitparam8len]: [14, 28] +2. [commit-parameters-lengths-8.bin][commitparam8len]: [14, 28] Explanation: From fced52da4d1c249fdac53cfffa157e5b5853ca10 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 19:18:41 +0100 Subject: [PATCH 15/17] Changed: Reordered Sections of Commit Parameters for Easier Understanding --- .../Storage/Loadouts/File-Format/Unpacked.md | 165 +++++++++--------- 1 file changed, 87 insertions(+), 78 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md index df63e38..91a7ce3 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Unpacked.md +++ b/docs/Server/Storage/Loadouts/File-Format/Unpacked.md @@ -491,7 +491,7 @@ This would be encoded as: - 14: Length of "Super Cool Mod" - 28: Length of "reloaded3.utility.somexample" -3. [commit-parameters-text.bin](#parametertype): +3. [commit-parameters-text.bin][ParameterType]: - `Super Cool Mod` - `reloaded3.utility.somexample` @@ -558,7 +558,7 @@ Let's say we want to reference all three parameters from the previous example in - 0: Version of the commit message. -This optimized approach uses a single [ParameterType](#parametertype) (`11: BackReference3_8`) to +This optimized approach uses a single [ParameterType] (`11: BackReference3_8`) to reference all three parameters at once, reducing the overall size of the encoded data. It's particularly efficient when you need to reference multiple consecutive parameters from a @@ -585,8 +585,8 @@ previous event. 2. Read the parameter types from [commit-parameter-types.bin][commit-param-types]. 3. Based on the parameter types, retrieve the actual parameter data from the appropriate locations: - Contextual parameters like `EventTime` can be inferred from the event context. - - Text data from [commit-parameters-text.bin](#parametertype) - - Timestamps from [commit-parameters-timestamps.bin](#parametertype) + - Text data from [commit-parameters-text.bin][ParameterType] + - Timestamps from [commit-parameters-timestamps.bin][ParameterType] - Back references from the appropriate [commit-parameters-backrefs-*.bin][commit-param-backrefs] file - Lists from [commit-parameters-lists.bin][commit-param-lists] @@ -597,74 +597,11 @@ previous event. This is an array of: -| Data Type | Name | Description | -| --------- | ------------------------------- | ---------------------- | -| `u8` | [ParameterType](#parametertype) | Type of the parameter. | +| Data Type | Name | Description | +| --------- | --------------- | ---------------------- | +| `u8` | [ParameterType] | Type of the parameter. | -### commit-parameters-versions.bin - -!!! info "This enables versioning, ensuring that different variations of the same commit message can coexist." - -!!! warning "There should be 1 entry for each event!! Regardless of whether it has a message or not!!" - -This is an array of: - -| Data Type | Name | Description | -| --------- | ------- | ------------------------------ | -| `u8` | Version | Version of the commit message. | - -The version number corresponds to the version suffix in the message key. - -For example: - -- If the message key is `PACKAGE_ADDED_V0`, the version would be `0`. -- If the message key is `MOD_CONFIG_UPDATED_V1`, the version would be `1`. - -This array contains `u8` values which correspond to the version of the commit message last issued -for each event. - -For example, if the message for an event like [PackageStatusChanged][message-packagestatuschanged] -is encoded with the key `PACKAGE_ADDED_V0`, it would be written as `0`: - -``` -Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. -``` - -However, if a new version of the message is introduced with a ***different meaning***, ***order of -parameters***, or ***number of parameters***, it would use a new key like `PACKAGE_ADDED_V1`, and -the version number would be `1`. - -In practice, expect to see mostly `0`, as the text for most commit messages is unlikely to change -often. When changes are needed, a new version of the message is created with an incremented version -number in its key. - -!!! note "Compressing 1M zeroes with zstd yields file size of ~50 bytes." - -### commit-parameters-lengths-8.bin - -This is an array of: - -| Data Type | Name | Description | -| --------- | --------------- | --------------------------------- | -| `u8` | ParameterLength | Length of the parameter in bytes. | - -### commit-parameters-lengths-16.bin - -This is an array of: - -| Data Type | Name | Description | -| --------- | --------------- | --------------------------------- | -| `u16` | ParameterLength | Length of the parameter in bytes. | - -### commit-parameters-lengths-32.bin - -This is an array of: - -| Data Type | Name | Description | -| --------- | --------------- | --------------------------------- | -| `u32` | ParameterLength | Length of the parameter in bytes. | - -### ParameterType +#### ParameterType `ParameterType` is defined as: @@ -719,6 +656,77 @@ Here is a listing of which parameter types go where: | `16` | `u32, u32` [(BackReference2_32)](#back-references) | `commit-parameters-backrefs-32.bin` | | `17` | `u32, u32, u32` [(BackReference3_32)](#back-references) | `commit-parameters-backrefs-32.bin` | +### Commit Parameter Lengths + +!!! info "The following files store the parameter lengths." + +These files are only used whenever the used [ParameterType] requires it. + +See the `description` section of each [ParameterType] for more information. + +#### commit-parameters-lengths-8.bin + +This is an array of: + +| Data Type | Name | Description | +| --------- | --------------- | --------------------------------- | +| `u8` | ParameterLength | Length of the parameter in bytes. | + +#### commit-parameters-lengths-16.bin + +This is an array of: + +| Data Type | Name | Description | +| --------- | --------------- | --------------------------------- | +| `u16` | ParameterLength | Length of the parameter in bytes. | + +#### commit-parameters-lengths-32.bin + +This is an array of: + +| Data Type | Name | Description | +| --------- | --------------- | --------------------------------- | +| `u32` | ParameterLength | Length of the parameter in bytes. | + +### commit-parameters-versions.bin + +!!! info "This enables versioning, ensuring that different variations of the same commit message can coexist." + +!!! warning "There should be 1 entry for each event!! Regardless of whether it has a message or not!!" + +This is an array of: + +| Data Type | Name | Description | +| --------- | ------- | ------------------------------ | +| `u8` | Version | Version of the commit message. | + +The version number corresponds to the version suffix in the message key. + +For example: + +- If the message key is `PACKAGE_ADDED_V0`, the version would be `0`. +- If the message key is `MOD_CONFIG_UPDATED_V1`, the version would be `1`. + +This array contains `u8` values which correspond to the version of the commit message last issued +for each event. + +For example, if the message for an event like [PackageStatusChanged][message-packagestatuschanged] +is encoded with the key `PACKAGE_ADDED_V0`, it would be written as `0`: + +``` +Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'. +``` + +However, if a new version of the message is introduced with a ***different meaning***, ***order of +parameters***, or ***number of parameters***, it would use a new key like `PACKAGE_ADDED_V1`, and +the version number would be `1`. + +In practice, expect to see mostly `0`, as the text for most commit messages is unlikely to change +often. When changes are needed, a new version of the message is created with an incremented version +number in its key. + +!!! note "Compressing 1M zeroes with zstd yields file size of ~50 bytes." + ### Back References !!! info "Back References are a Special Type of Parameter that references a previous item." @@ -731,7 +739,7 @@ Here is a listing of which parameter types go where: This improves loadout sizes by reducing existing previous data. Back References are defined as 1 or more `ParameterIndex` fields, whose location and data type -depends on [ParameterType](#parametertype). +depends on [ParameterType]. A `ParameterIndex` of `0` means 'the first commit parameter' in file. `1` means 'the second commit parameter' etc. @@ -773,11 +781,11 @@ This is where `Parameter Lists` come in. A `Parameter List` is defined as: -| Data Type | Name | Description | -| --------- | ------------------------------- | ------------------------------------- | -| `u8` | [ParameterType](#parametertype) | Type of the parameter. | -| `u4` | Version | [Event Specific] version of the list. | -| `u20` | NumParameters | Number of parameters. | +| Data Type | Name | Description | +| --------- | --------------- | ------------------------------------- | +| `u8` | [ParameterType] | Type of the parameter. | +| `u4` | Version | [Event Specific] version of the list. | +| `u20` | NumParameters | Number of parameters. | For the example above, we can treat each `Change` as 2 parameters. In which case, if we had 2 changes, we would set `NumParameters` to `4`. @@ -837,4 +845,5 @@ as regular parameters in [Commit Parameters](#commit-parameters). [config-file-paths]: ../../../Packaging/Package-Metadata.md#path [config-file]: ../../../Packaging/Package-Metadata.md#configfile [community-repository]: ../../../../Services/Community-Repository.md -[nexus]: https://www.nexusmods.com \ No newline at end of file +[nexus]: https://www.nexusmods.com +[ParameterType]: #parametertype \ No newline at end of file From 071ad8fcc9ed55ebb5059ea24d3a8952d8748342 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 19:21:52 +0100 Subject: [PATCH 16/17] Improve: Updated External Config is a Contextual Parameter --- docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md index cc2610e..ed16082 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md +++ b/docs/Server/Storage/Loadouts/File-Format/Commit-Messages.md @@ -578,7 +578,7 @@ to - `Name` [0]: The name of the package (mod, tool, etc.). [String][commit-param-type] - `ID` [1]: The package ID. [String][commit-param-type] -- `ConfigPath` [2]: The path of the updated configuration file. [String][commit-param-type] +- `ConfigPath` [-1]: The path of the updated configuration file. [String [Contextual]](#contextual-parameters) ### EXTERNAL_CONFIG_UPDATED_V0 From 292a1b5a727e4803de030de965fcb6d7624389c8 Mon Sep 17 00:00:00 2001 From: Sewer56 Date: Sun, 25 Aug 2024 20:52:39 +0100 Subject: [PATCH 17/17] Added: Info of external configs in snapshot. --- .../Storage/Loadouts/File-Format/Snapshot.md | 27 ++++++++++++++++--- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md index 38221bc..418c024 100644 --- a/docs/Server/Storage/Loadouts/File-Format/Snapshot.md +++ b/docs/Server/Storage/Loadouts/File-Format/Snapshot.md @@ -22,6 +22,10 @@ struct Snapshot { configurations: Vec>, config_hashes: HashMap, // Hash to index in 'configurations' + // External Configurations + external_configs: Vec, + external_config_hashes: HashMap, // Hash to index in 'external_configs' + // Mod Load Order mod_load_order: Vec, // Vec of indices into packages @@ -42,6 +46,12 @@ struct PackageInfo { configuration_index: Option, // Index into configurations Vec } +struct ExternalConfigInfo { + package_id: XXH3, // XXH3(PackageID) + path: String, // Relative path to the external config file + data: Vec, // Raw content of the external config file +} + struct LoadoutDisplaySettings { loadout_grid_enabled_sort_mode: SortingMode, loadout_grid_disabled_sort_mode: SortingMode, @@ -113,17 +123,25 @@ struct MicrosoftStoreData { - Corresponds to [config.bin & config-data.bin][configbin]. - Configurations are not version specific. They persist across package versions. -4. **Mod Load Order** +4. **External Configurations** + - A list of external configuration information for packages, each containing: + - Package ID: The [XXH3 hash][hashing] of the package ID this config belongs to. + - Path: The relative path to the external config file. + - Data: The raw content of the external config file. + - Corresponds to [external-config.bin, external-config-data.bin, & external-config-paths.bin][externalconfigbin]. + - External configurations are tracked separately from internal configurations. + +5. **Mod Load Order** - An ordered list of indices representing the current load order of mods. - These indices correspond to the positions in the `packages` list. -5. **Loadout Display Settings** +6. **Loadout Display Settings** - Enabled Grid Sort Mode: How enabled mods are sorted in the mod view ([`SortingMode`][sortingmode]). - Disabled Grid Sort Mode: How disabled mods are sorted in the mod view ([`SortingMode`][sortingmode]). - Mod Load Order Sort: Whether mods are shown `top to bottom` or `bottom to top` for load ordering ([`SortOrder`][sortorder]). - Grid Display Style: The visual style of the grid that displays enabled mods ([`GridDisplayMode`][griddisplaymode]). -6. **Game Store Manifest** +7. **Game Store Manifest** - Store Type: Which store the game is from ([`StoreType`][storetype]). - Store Data: A structure containing common fields and store-specific data: - Common fields for all store types: @@ -138,7 +156,7 @@ struct MicrosoftStoreData { The `store_type` field determines which store-specific struct is populated and should be used. -7. **Commandline Parameters** +8. **Commandline Parameters** - A string containing the current commandline parameters for the game. ## How Snapshots are Used @@ -193,4 +211,5 @@ there will be no migration code; as to avoid bloating the binary. [loadout-location]: ../About.md#location [bitcode]: ../../../../Research/Library-Sizes/Serializers.md#bitcode [configbin]: ./Unpacked.md#configbin +[externalconfigbin]: ./Unpacked.md#external-configbin [packagereferenceidsbin]: ./Unpacked.md#package-idsbin \ No newline at end of file