Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] run any model from huggingface #358

Draft
wants to merge 473 commits into
base: main
Choose a base branch
from
Draft

[WIP] run any model from huggingface #358

wants to merge 473 commits into from

Conversation

komikat
Copy link

@komikat komikat commented Oct 16, 2024

ref: #357

TODO

  • parse user input
  • download gguf file
  • decide backend to run this .gguf file (currently thinking of llama.cpp)

@risingsunomi
fix main debug error
48f2de6
@risingsunomi
fix main debug error
68d9c46
@risingsunomi
finish error
7306cfc
@risingsunomi
finish error
09eb58a
@risingsunomi
finish error
9c18737
@risingsunomi
trying to manipulate sampling
bf8ed2a
@risingsunomi
using resolve tokenizer from api
ae413d2
@risingsunomi
using resolve tokenizer from api
5e8663d
@risingsunomi
working on eot error
9e3a4a7
@risingsunomi
working on eot error
6f90f43
@risingsunomi
working on eot error
07c146c
@risingsunomi
working on eot error
da2a97b
@risingsunomi
working on eot error
f54990a
@risingsunomi
working on eot error
e96579f
@risingsunomi
working on eot error
25361ec
@risingsunomi
eot issue, update sampling
84a0323
@risingsunomi
ensure 2d
fffc034
@risingsunomi
ensure 2d
3fbc7d2
@risingsunomi
doing a fraction of temp for top p
df2144e
@risingsunomi
doing a fraction of temp for top p
9b37ee1
@risingsunomi
trying other sampling
49eead3
@risingsunomi
trying other sampling
1192971
@risingsunomi
trying other sampling
8bc583b
@risingsunomi
trying other sampling
9afe07f
@risingsunomi
trying other sampling
954682b
@risingsunomi
fix typo
0547fa3
@risingsunomi
use input embeds
88c94a8
@risingsunomi
use position ids
4160c40
@risingsunomi
use position ids
a6e3c15
@risingsunomi
use position ids
6f3435e
AlexCheema and others added 30 commits October 11, 2024 16:39
@AlexCheema
in torch, explicitly set the device when initilaizing the model
830d33d
@AlexCheema
spacing
074dfe3
@AlexCheema
add model mlx-community/Qwen2-0.5B-Instruct-4bit
d9cfcc4
@risingsunomi
Merge branch 'exo-explore:main' into main
c3e1934
@risingsunomi
code changes from PR feedback, working on splitting of weights
2c056b4
@risingsunomi
Merge branch 'exo-explore:main' into pr139-dev-oct24
da5c28d
@risingsunomi
doing more work toward individual safetensor loading, adding back dev…
83a723b 
…ice mapping auto
@risingsunomi
working on split model, moving to server for more vram
47be250
@risingsunomi
change to hf downloader as was not getting all safetensor files
ea0d4b1
@risingsunomi
splitting model still work in progress as transformers still seems to…
30b7991 
… try to load more than needed even witha modified safetensor json file, finished up PR main updates but will continue on this one
@risingsunomi
updating readme
3a2c431
@risingsunomi
Merge branch 'main' into pr139-dev-oct24
4def538
@risingsunomi
Merge pull request exo-explore#24 from risingsunomi/pr139-dev-oct24
b35224c 
Pr139 dev oct24
@risingsunomi
successful splitting model test with only loading needed weights, imp…
6c6e7b2 
…lementing it in main inference code
@risingsunomi
Merge branch 'pr139-dev-oct24' of github.com:risingsunomi/exo-nvidia …
55ffdc7 
…into pr139-dev-oct24
@risingsunomi
adding model sharding to inference engine, doing testing with inferen…
aacdeb5 
…ce engine and sharding
@risingsunomi
fixing layer range issue
ce702d1
@risingsunomi
fixing layer range issue
e387a79
@risingsunomi
fixing layer range issue
e0ba2bb
@risingsunomi
checking if ram over usaage even if reducing layers on large models
5b9638f
@risingsunomi
half layer inference engine testing
664f29f
@risingsunomi
fixing layer amount with sharded modeling
2591fab
@risingsunomi
adding qwen2.5 3B for testing
99dac57
@risingsunomi
Merge pull request exo-explore#25 from risingsunomi/pr139-dev-oct24
c12526f 
Pr139 dev oct24
@risingsunomi
updating inference engine test
493cd3e
@risingsunomi
cleaning up utils and split model
de23294
@risingsunomi
Merge pull request exo-explore#26 from risingsunomi/pr139-dev-oct24
d5a02be 
Pr139 dev oct24
@dtnewman
bugfix in llm setup
e7470b1
@risingsunomi
Merge pull request exo-explore#27 from dtnewman/main
fa24f46 
bugfix in llm setup
@komikat
parsing and downloading .gguf from hf
7cc4471
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@komikat @risingsunomi @AlexCheema @dtnewman