Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace code generation with generics #61

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

lwc
Copy link

@lwc lwc commented Apr 4, 2022

Mostly done for my own edification.

Intentionally kept the API as similar as possible to demonstrate the existing test cases still pass (with one exception, noted).

Should keep the migration path very straightforward from generated to generic.

dl.Prime(user.ID, &user)
u := &user
cpy := *u
dl.Prime(user.ID, &cpy)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one breaking change, tests are updated to reflect new behaviour.

@n33pm
Copy link

n33pm commented Apr 5, 2022

code looks good to me...

  • added test on your pr. we dont need tests on the example anymore.
  • README.md needs updates too.
  • version update because its a breaking change

@n33pm
Copy link

n33pm commented Apr 11, 2022

code looks good to me...

  • added test on your pr. we dont need tests on the example anymore.
  • README.md needs updates too.
  • version update because its a breaking change

https://github.com/numero33/dataloaden

@frederikhors
Copy link

Hi guys, I would like to bring to your attention the following project: https://github.com/vikstrous/dataloadgen which has great benchmarks!

What do you think?

@n33pm
Copy link

n33pm commented Apr 11, 2022

Hi guys, I would like to bring to your attention the following project: https://github.com/vikstrous/dataloadgen which has great benchmarks!

What do you think?

benchmarks are similar and on my hand i use concurrently far more often than cached values.
don't get me wrong but I don't think it's optimal on vikstrous/dataloadgen if you cache the whole function.

dl := dataloaden.NewLoader(dataloaden.LoaderConfig[int, benchmarkUser]{
		Wait:     500 * time.Nanosecond,
		MaxBatch: 100,
		Fetch: func(keys []int) ([]benchmarkUser, []error) {
			users := make([]benchmarkUser, len(keys))
			errors := make([]error, len(keys))

			for i, key := range keys {
				if key%100 == 1 {
					errors[i] = fmt.Errorf("user not found")
				} else if key%100 == 1 {
					users[i] = benchmarkUser{}
				} else {
					users[i] = benchmarkUser{ID: strconv.Itoa(key), Name: "user " + strconv.Itoa(key)}
				}
			}
			return users, errors
		},
	})
cpu: Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
BenchmarkDataloadgen
BenchmarkDataloadgen/caches
BenchmarkDataloadgen/caches-8         	17150414	        65.49 ns/op
BenchmarkDataloadgen/random_spread
BenchmarkDataloadgen/random_spread-8  	 2686813	       649.7 ns/op
BenchmarkDataloadgen/concurently
BenchmarkDataloadgen/concurently-8    	   89816	     14618 ns/op
BenchmarkN33Dataloaden
BenchmarkN33Dataloaden/caches
BenchmarkN33Dataloaden/caches-8       	12214902	        92.20 ns/op
BenchmarkN33Dataloaden/random_spread
BenchmarkN33Dataloaden/random_spread-8         	 2403955	       532.4 ns/op
BenchmarkN33Dataloaden/concurently
BenchmarkN33Dataloaden/concurently-8           	   82574	     13577 ns/op

@frederikhors
Copy link

@StevenACoffman can you operate on this repo?

Instead of a new version on this package, can we add this to gqlgen? Like gqlgen/dataloaden or gqlgen/dataloadgen?

We could also write in Readme that the project comes from this repo and new credits to @lwc.

@StevenACoffman
Copy link
Collaborator

StevenACoffman commented Apr 11, 2022

I do have access to merge pull requests in this repository, and I don't mind merging this PR if @lwc cannot.

I don't think that this repo should be folded into gqlgen though.

I would rather than the community coalesced and collaborated on a single dataloader implementation, I don't think we are quite there yet.

Currently graph-gophers/graphql-go and 99designs/gqlgen can pick from any of the three mentioned dataloaders:

At Khan Academy, we currently use (and prefer) graph-gophers/dataloader so I would personally rather not add vektah/dataloaden to the gqlgen repository. Also, we are stuck on Go 1.16 until GCP AppEngine supports a newer version, so Khan can't use generics yet.

For those reasons, I don't think I can allocate time to maintain this repository until Khan Academy was actively using it.

@frederikhors
Copy link

@lwc what do you think about creating a new repo since it’s all new?

@Warashi
Copy link

Warashi commented Jul 25, 2022

hello, I created a repository Warashi/dataloaden. which contains generics version of vektah/dataloaden.
If you want to use generics version of dataloaden, please consider to use Warashi/dataloaden.

@StevenACoffman
Copy link
Collaborator

@Warashi Have you compared your project to https://github.com/vikstrous/dataloadgen ? I wonder if you could collaborate.

Copy link
Collaborator

@wilhelmeek wilhelmeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loving this 😍

// Fetch is a method that provides the data for the loader
Fetch func(keys []K) ([]T, []error)

// Wait is how long wait before sending a batch
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a typo here? i.e.

Suggested change
// Wait is how long wait before sending a batch
// Wait is how long to wait before sending a batch

return l.LoadThunk(key)()
}

// LoadThunk returns a function that when called will block waiting for a User.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// LoadThunk returns a function that when called will block waiting for a User.
// LoadThunk returns a function that when called will block waiting for a T.

return users, errors
}

// LoadAllThunk returns a function that when called will block waiting for a Users.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// LoadAllThunk returns a function that when called will block waiting for a Users.
// LoadAllThunk returns a function that when called will block waiting for Ts.

@Warashi
Copy link

Warashi commented Jul 26, 2022

@StevenACoffman oh, I had missed dataloadgen. I would love to collaborate with you on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants