Notes and Design Info for the Service Refactor

[dave-gray101]

In Defense of Service Structs

In order to more properly document my goals with the service refactor, this document will break down the "rings and layers" of functionality I'm hoping to achieve. Let's hammer down where we stand on it finally.

pkg/anything:

Code in these folders should not really be specific to LocalAI at all. Mostly, things here are already the way they should be - utilities, downloader code, library wrappers etc. While some of these like functions and models aren't likely to have particular use outside of LocalAI, they all share the defining characteristic of not referencing any code in core (only external libraries and other pkg packages.)

Starting from the "Front" of LocalAI as a request would see it:

core/http/middleware

The basics of core/http aren't scheduled to change, but I plan to centralize existing middleware and create more under this package. Auth will be applied to all endpoints by default rather than specifying it on specific ones. I've already tested a config option to enable freeing the webui to users without an auth header. Metrics needs zero changes. The bulk of the interesting code in this folder will be a series of centralized fiber context ==> request model retrieval middlewares to reduce code duplication in the next section

core/http/endpoints... AND ... core/mqtt/endpoints:

Once a request is recieved by LocalAI, code in these layers has an extremely limited set of responsibilities. The ONLY task of this layer is to parse the request to a standard request structure, invoke the next service below, and format the response for output. There should be no logic here whatsoever - this is the layer that differentiates between a request being transmitted over http, mqtt or a future transport, and all functionality should be identical between the transports. Ideally, these classes should hold a reference to one or two core/services structs described below, and nothing else other than what they need for their transport.

The existing "service structures" ModelLoader and BackendConfigLoader:

These two structures are extremely important to the function of LocalAI as we know it, and therefore in these early stages of the refactor, I want to minimize changes to them. My current plan for these is to leave them in the exact places they stand for now - but to essentially consider these "private" or low level implementation detail services. Once an endpoint is "migrated" and fully refactored, I expect that no endpoint (or middleware) will directly call into these services - the relevant core/services or core/backend structure will call into these as required. Specifically, in the "interim" period of incremental refactoring, an endpoint that still directly uses these will indicate that it hasn't been completed yet.

core/services - such as ListModels etc:

The structures here make up the "public api" of LocalAI itself for fellow developers and will exist to hold all of our existing buisness logic / actual functionality that isn't provided by an external gRPC backend. Code here should be specific to a usecase like ListModels or Gallery - this is where the calls to ModelLoader and BackendConfigLoader will continue to be made, and expose a limited function surface to the endpoints above - or potentially other core/backend structures described below.

core/backend structures:

Currently, this folder has "loose" functions that manage the LocalAI-provided portions of backend calls: loading models, handling templating and formatting responses, etc. I plan to provide slim structures here similar to core/services to avoid the need for endpoints to directly work with "private" structures like ModelLoader or ApplicationConfig. Existing logic code from the endpoints will be moved to this layer to allow reuse between transports.

gRPC backends:

Optional Proposal here only: #2749

[dave-gray101]

@mudler - to clarify: I'm expressly not sure that structures are the best way to accomplish this type of separation of concerns in golang, but it's what comes to my mind. This discussion item was created to give me a place to jot notes and my current plan in a central place. Despite the header joke about the core/services package's fate across merges, I care more about the overall division of responsibilities than implementation style.