recursive version of just
#293
Replies: 4 comments 12 replies
-
Hi @Jasha10, thanks so much for taking the time to write this up! I like I am currently playing around with your proposed implementation and am working on a write-up. I wanted to post in the meantime to let you know that I am working on this. I am teaching an intensive course this month, which is pulling me away from hydra-zen for the time being, so my responses may be more delayed than I would like them to be 😅 Thanks again for sharing this idea! |
Beta Was this translation helpful? Give feedback.
-
Thanks for sharing @Jasha10! Would this be related to #257 with the addition of recursion? |
Beta Was this translation helpful? Give feedback.
-
ResponseHi @Jasha10 ! I am finished teaching and am back in the swing of things at work! Thanks for your patience and understanding for the past few weeks. A Brief Review of This ProposalLet me first summarize my understanding of the novel functionality that is being proposed here. Wherease To demonstrate the utility of this in practice, let's design a structured config using a pydantic-based dataclass, which we can make compatible with Hydra while retaining the runtime type-checking and rich types associated with our original dataclass. import dataclasses
from typing import List, Optional, Literal
from pydantic.dataclasses import dataclass as pydantic_dataclass
from pydantic import PositiveInt
@pydantic_dataclass
class User:
name: str
age: PositiveInt # typically Hydra users would have to use `int` here
@pydantic_dataclass
class Match:
user_a: User = User(name="Bob", age=25)
user_b: User = User(name="Alice", age=27) Now we can use from hydra.errors import InstantiationException
conf = recursive_just((User(name="Bob", age=20)))
# Attempting to overwrite age with a negative number..
with raises(InstantiationException):
# Pydantic's runtime type-checking raises an error during the instantiation process.
instantiate(conf, age=-10) # error: age < 0
profile: User = instantiate(conf, age=11)
assert isinstance(profile, User)
assert profile == User("Bob", 11) And here is the recursion in action: from hydra_zen import to_yaml
sanitized_match = recursive_just(Match())
assert to_yaml(sanitized_match) == """\
_target_: __main__.Match
user_a:
_target_: __main__.User
name: Bob
age: 25
user_b:
_target_: __main__.User
name: Alice
age: 27
"""
assert instantiate(sanitized_match) == Match() I like this proposal a lot! I think that this reduces the barrier to entry for new Hydra users while also enabling advanced patterns (e.g., the demonstrated cross-compatibility with pydantic!). The improved ergonomics afforded by the recursion is also a big win, and the gains here will grow as hydra-zen's auto-config support continues to improve (e.g., see #294). Implementation DetailsOne change to this prototype that I have identified thus far is that I would like to support dataclass-types in addition to the proposed dataclass-instances; this is motivated by the fact that Hydra largely treats dataclass types and instances on an equal footing. from hydra_zen import just, builds, to_yaml
from hydra_zen.structured_configs._type_guards import is_builds
from dataclasses import is_dataclass, fields
def recursive_just(obj):
if is_dataclass(obj) and not is_builds(obj):
# obj is a dataclass type or instance that does not have a _target_ field
type_ = obj if isinstance(obj, type) else type(obj)
converted_fields = {}
for field in fields(obj):
if field.init and hasattr(obj, field.name):
value = getattr(obj, field.name)
converted_fields[field.name] = recursive_just(value)
Conf = builds(
type_,
populate_full_signature=True,
**converted_fields
)
return Conf if type_ is obj else Conf()
else:
return just(obj) Thus # support for dataclass types (in addition to instances)
assert to_yaml(recursive_just(User)) == """\
_target_: __main__.User
name: ???
age: ???
"""
# `recursive_just(User)` returns dataclass type
assert isinstance(recursive_just(User), type) # Builds_User
# `recursive_just(User("bob", 12))` returns dataclass instance
assert not isinstance(recursive_just(User("bob", 12)), type) # Builds_User("bob", 12) This is effectively identical to Supporting dataclass types impacts the nice property of idempotence that Jasha's implementation possesses. To remedy this, from dataclasses import dataclass
# demonstrating idempotence
compat_user = recursive_just(User("bob", 12))
assert hasattr(compat_user, "_target_")
assert recursive_just(compat_user) is compat_user
@dataclass
class HasTarget:
_target_: str = "hi"
conf = HasTarget()
assert recursive_just(conf) is conf The APInote: I ended up changing course on the API design that I laid out here, see #293 (reply in thread) for revised details The name We will also need to allow users to pipe through def create_hydra_compat_config(
config: Union[Dataclass, Type[Dataclass]],
hydra_convert=None, # Sent to all recursive calls of `create_hydra_compat_config`
# these only apply to the resulting top-level dataclass
dataclass_name: Optional[str] = None,
bases: Tuple[Type[Dataclass], ...] = ()
)
-> Builds[Type[Dataclass]]:
"""Given a dataclass type or instance, constructs a Hydra-compatible structured config that,
when instantiated, returns an instance of the original dataclass. This applies recursively to
nested dataclasses"""
... I am hoping that the name conveys to the user that this function is meant to operate specifically on structured configs / dataclasses, and that a new, Hydra-compatible, structured config is returned. Next stepsI would love to get general feedback on this (including possible alternative names to Footnotes
|
Beta Was this translation helpful? Give feedback.
-
@Jasha10 @addisonklinke @jgbos I have just released Read more about it here: #301 I'm excited about this new feature. I think it is quite powerful! |
Beta Was this translation helpful? Give feedback.
-
cc @pixelb @jieru-hu
Edit: fixed a bug above
Beta Was this translation helpful? Give feedback.
All reactions