Support multiple tokens locally

[Wauplin]

Originally from @osanseviero on slack (private)

With the big push for fine-grained tokens, I'm wondering if we should have a way for users to have multiple tokens easily in their environment and switch between them rather than overriding each time

Originally from @julien-c in reply

aws-cli has something like this

we could add a flag for --profile token_name or --token-name "My token to list private models"

Originally from @osanseviero as well

On my side I stopped using Colab secret management entirely as it doesn't make sense in the fine-grained tokens world. I'm again signing in each time with a new key

---

We would need:

• to be able to define a name per token in login
• a new command / helper to list existing local tokens
• a new command / helper to switch the "current" local token

Google Colab secret management is a separate topic but still relevant.

[lappemic]

Hey @Wauplin, i would love to work on this one! To be aligned this would be my approach:

• Adding HF_TOKENS_PATH to constants.py
• Updating the CLI interface in huggingface_hub/commands/user.py with subcommands list-tokens and switch-tokens
• Updating the huggingface_hub/utils.py with _load_tokens(), _save_tokens(), get_token(), list_tokens(), and switch_token()
• And then bringing it together in huggingface_hub/_login.py adding the token_name as new param.

Does this sound in general correct?

Looking forward to your feedback and having finally time to work on another hf issue!

[Wauplin]

Hi @lappemic, nice to hear back from you :) Thanks for the suggestions. Unfortunately we've decided internally to move forward on it with @hanouticelina who joined us last week and will maintain core parts of the library. She's already done a comparison work with aws-cli, gh, etc. and has started to work on a PR. Maybe @hanouticelina you could share here or on a first PR the direction you are taking? (once you have something consolidated).

@lappemic in the meantime, would you be interested to work on the huggingface-cli delete-cache command instead. It's a command line tool to clean your HF cache locally that has been implemented ~18 months ago. It has become quite practical in the ecosystem but can still be improved. We've got great feedback in #1997 and also discussed improvements in #1065. The goal is not to change everything but to add a few improvements to save users some time when they clean-up their cache. Typically:

• as mentioned in huggingface-cli delete-cache --disable-tui improvements #1997, when TUI is disabled, we should not use a temporary file but a fixed one (./tmp_hf_cleaner.txt?)
• we should provide an option to select all "detached" revisions at once (e.g. only keep main or specific tags)
• change the wording when confirming the delete (confusing at the moment).

Would you like to look into it and work on some improvements? Ofc, not everything has to be done at once. If yes, let's continue the discussion in a #1997 :)

[hanouticelina]

Hi 👋 As discussed internally, we drafted a first version of the CLI usage documentation. The idea is to keep using HF_TOKEN and HF_TOKEN_PATH (i.e. ~/cache/huggingface/token) when switching between tokens.

Usage

Login

huggingface-cli login [--token TOKEN] [--profile PROFILE_NAME]

• Adds or updates the specified profile with the given token.
• If --token is not provided, prompts the user for token interactively. This is already the case.
• If --profile is not specified, uses the "default" profile.

List Profiles

huggingface-cli auth list

• Lists all saved profiles.

Logout

huggingface-cli logout [--profile PROFILE_NAME --all]

• Removes the specified profile and its associated token.
• If --profile is not specified, logs out from the default one.
• --all : A total clean-up option to ensure all tokens are deleted.

Switching the current active profile

huggingface-cli auth switch PROFILE_NAME

• Sets the specified profile as the active profile for subsequent commands.
• This command will override the token written in ~/.cache/huggingface/token (defined by HF_TOKEN_PATH env variable).

How it Works

Token storage

Tokens will be stored in ~/.cache/huggingface/profiles (or HF_PROFILES_PATH env variable if set), which is INI format file:

[default] 
hf_token = hf_XXXXXXX 
[profile1] 
hf_token = hf_YYYYYYY
[profile2] 
hf_token = hf_ZZZZZZZ

Token Selection Priority

We will keep the same token retrieval priority order

1. HF_TOKEN env variable.
2. Token file ~/.cache/huggingface/token. The token stored in this file will be overrided when switching between profiles.

Implementation Plan

I already started some experimentation locally with the following implementation (still need to be refined and discussed in the PR):

• Add a new en variable that points to the path of a file that contains the different profiles : HF_PROFILES_PATH=~/.cache/huggingface/profiles.
• Add profile parameter (default to 'default') to login() function.
• Update _login() and add a couple of helper functions:

def _login(
    token: str,
    add_to_git_credential: bool,
    write_permission: bool = False,
    profile: Optional[str] = None,
) -> None:
    # Existing validation code...

    profile_name = profile or "default"
    # Save token to profiles
    _save_token_to_profiles(token, profile_name)
    # Set active profile
    _set_active_profile(profile_name)

    logger.info("Login successful")


def _save_token_to_profiles(token: str, profile_name: str) -> None:
    """
    Save the token to the profile. Parses the ini file, if the profile does not exist,
    it creates it. Otherwise, it updates the token.
    """
    profiles_path = Path(constants.HF_PROFILES_PATH)
    ...


def _get_token_from_profiles(profile_name: str) -> str:
    """
    Get the token from the profile. Parses the ini file and returns the token.
    If the token is not found, it raises an error.
    Otherwise, it returns the token.
    """
    ...

def _set_active_profile(profile_name: str) -> None:
    token = _get_token_from_profiles(profile_name)
    # Write token to HF_TOKEN_PATH
    path = Path(constants.HF_TOKEN_PATH)
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(token)
    print(f"Your token has been saved to {constants.HF_TOKEN_PATH}")

Other function helpers will be added to write/parse the profiles file.

• Update logout() to add a profile and a all boolean parameter:

def logout(profile_name: Optional[str] = None, all: bool = False) -> None:
    profiles = ...  # Read profiles from file
    profile_name = profile_name or "default"
    if all:
        # delete profiles file and token file
        return

    if profile_name in profiles:
        # Remove profile from profiles file
        # If the active token matches this profile's token, delete it + Add a warning to the user that the active token will be deleted
        if profiles[profile_name] == get_token():
            warnings.warn(f"Active Profile '{profile_name}' will be deleted.")
        print(f"Profile '{profile_name}' has been deleted.")
    else:
        # Raise error profile '{profile_name}' does not exist.

• Add a profile switching function that will be used for the huggingface-cli auth switch command:

def auth_switch(profile_name: str) -> None:
    """Switch to the specified profile."""
    token = _get_token_from_profiles(profile_name)
    if token:
        # Write token to ~/.cache/huggingface/token
        _set_active_profile(profile_name)
        print(f"Switched to profile '{profile_name}'.")
        # Warn if HF_TOKEN environment variable is set
        if _get_token_from_environment():
            warnings.warn(
                "The environment variable 'HF_TOKEN' is set and will override " "the token from the profile."
            )
    else:
        # Raise error profile '{profile_name}' does not exist.

• Add profile listing function that will be used for the huggingface-cli auth list command:

def auth_list():
    """List all saved profiles. The formatting still needs to be defined"""
    ...

• Add the new commands.
• Add tests. 🤗
• Update cli and login guides.
• Update documentation.

[Wauplin]

Logout

If --profile is not specified, logs out from the default one.

I'd say, if --profile is not specified, logs out from the current one.

[Wauplin]

Otherwise code looks good but next time don't hesitate to open the PR directly even if the code is not complete (draft PRs exist for that^^). It makes it easier to try things and suggest small changes.