Heterogeneous array creation #109

treeowl · 2018-03-25T22:13:13Z

Create arbitrarily many arrays of arbitrarily many types from one
ST action.

Closes #103

treeowl · 2018-03-25T22:17:40Z

Question: I made the heterogeneous creation functions take traversals that are polymorphic over their Applicatives. Is that necessary to ensure the safety of the unsafeFreeze calls, or would it be okay to specialize to ST s? The rest of the polymorphism seems to be necessary.

Create arbitrarily many arrays of arbitrarily many types from one `ST` action. Closes haskell#103

treeowl · 2018-03-27T04:06:42Z

FYI, the higher-rank traversals are basically the ones from Rank2.Traversable in rank2classes and from Data.Comp.Multi.HTraversable in compdata. They're type-flipped from Data.Vinyl.Core.rtraverse in vinyl (we definitely don't want them flipped like that). The only difference is that the traversals can be type-changing. I don't think that likely lets anyone break things (though I haven't proven that), and it seems to be somewhat useful.

andrewthad · 2018-03-29T00:32:17Z

Data/Primitive/SmallArray.hs

+runSmallArrays m = runST $ m >>= traverse unsafeFreezeSmallArray
+
+-- | Create arbitrarily many arrays that may have different types. For
+-- a simpler but less general version, see 'runArrays'.


Should say "see runSmallArrays".

andrewthad · 2018-03-29T00:40:57Z

I cannot verify the soundness of this, but I like it. Also, it would be good to know the answer of your question about Applicative possibly being specialized to ST instead.

andrewthad · 2018-03-29T01:00:52Z

Also, I dislike the names with Het in them, but at the moment, I don’t have any better suggestions.

treeowl · 2018-03-29T01:12:54Z

The names are awful.

…

On Thu, Mar 29, 2018, 4:00 AM Andrew Martin ***@***.***> wrote: Also, I dislike the names with Het in them, but at the moment, I don’t have any better suggestions. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABzi_Qd-4jwDJn4wHEwTJV8_ko_rtuo-ks5tjDJFgaJpZM4S6VN1> .

RyanGlScott

I, too, cannot offer an opinion on whether the extra polymorphism is necessary or not, but these seem like they're in the spirit of vector's createT, so I generally support them. (I also can't offer better naming suggestions.)

Two minor comments inline.

RyanGlScott · 2018-04-06T16:04:51Z

Data/Primitive/Array.hs

@@ -1,6 +1,9 @@
 {-# LANGUAGE CPP, MagicHash, UnboxedTuples, DeriveDataTypeable, BangPatterns #-}
 {-# LANGUAGE RankNTypes #-}
 {-# LANGUAGE TypeFamilies #-}
+#if __GLASGOW_HASKELL__ >= 708


Shouldn't this be 706? (That's the first GHC version that reliably supported PolyKinds.)

I mistakenly though it didn't stabilize till later. Fixed!

RyanGlScott · 2018-04-06T16:05:00Z

Data/Primitive/SmallArray.hs

@@ -7,6 +7,9 @@
 {-# LANGUAGE DeriveDataTypeable #-}
 {-# LANGUAGE GeneralizedNewtypeDeriving #-}
 {-# LANGUAGE BangPatterns #-}
+#if __GLASGOW_HASKELL__ >= 708


Similarly, 706 here.

treeowl · 2018-04-13T08:02:02Z

Using Data.Functor.Sum, I've convinced myself that the type change from t to u shouldn't be a problem. I'm still not so sure about whether we can have trouble with unlawful traversals in general. I think the very most we might be able to get away with is this:

runArraysHetOfThen
  :: Applicative q
  => (forall s1 s2.
       ((forall x. MutableArray s2 x -> Compose (ST s1) q (r x)) -> t (MutableArray s2) -> Compose (ST s1) q (u r)))
  -> (forall x. Array x -> q (r x))
  -> (forall s. ST s (t (MutableArray s)))
  -> q (u r)

This reveals almost everything to the traversal function, but doesn't give it quite enough to actually monkey with the MutableArrays itself (since it doesn't know it's working in the same state thread they live in).

treeowl · 2018-04-14T21:47:23Z

I've pretty much convinced myself that these are safe, even generalized considerably. See my answer to my question on SO.

RyanGlScott · 2018-04-14T21:53:33Z

Do you mind including a summary of the rationale in that SO post somewhere in the comments? Once that's included, I'd say this is ready to be merged.

treeowl · 2018-04-15T01:06:31Z

@RyanGlScott, I added some notes, changed some types, and removed PolyKinds (I don't think it actually does anything useful here). Could you take another glance?

treeowl · 2018-04-15T01:45:52Z

Oh, and a note I meant to leave earlier:

I've generalized the types of runArraysHetOf and runSmallArraysHetOf. Thanks to that generalization, runArraysHetOfThen and runSmallArraysHetOfThen can be implemented (efficiently) using them, so I've removed the latter two functions to reduce clutter. I've added runArraysOf and runSmallArraysOf for consistency and because not everyone who might want to supply a custom traversal will also want to figure out rank-2 traversals.

* Add notes. * Improve some of the types. * Drop `PolyKinds`. I added it because of unclear thinking, and I don't think any of this benefits meaningfully from having it.

treeowl · 2018-04-15T08:41:09Z

One more question: would runHetArraysOf roll more easily off the tongue?

treeowl · 2018-04-15T08:41:46Z

I think it would; if other's don't disagree, I'll make that change.

RyanGlScott

I don't have a strong opinion on the names, so pick whichever sounds best to you.

cartazio · 2018-04-15T17:47:47Z

I’ll have a read through after I finish my other bits of review work

…

On Sun, Apr 15, 2018 at 11:02 AM Ryan Scott ***@***.***> wrote: ***@***.**** approved this pull request. I don't have a strong opinion on the names, so pick whichever sounds best to you. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#109 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQwtEVkGVM4urmTkggEaTNCFOCf8raks5to2ERgaJpZM4S6VN1> .

Without `PolyKinds`, this is needed to actually make it work.

treeowl · 2018-04-15T21:04:33Z

Data/Primitive/Array.hs

+  :: (forall s1 s2.
+       ((forall x. MutableArray s1 x -> ST s2 (Array x)) -> t (mut s1) -> ST s2 u))
+     -- ^ A rank-2 traversal
+  -> (forall s. ST s ((t :: (* -> *) -> *) (mut s)))


With PolyKinds, this can be t :: (j -> k) -> *. Maybe we can do that once we drop support for 7.4; until then I think it's probably a bit too much CPP for what it offers.

(Remember that mut doesn't necessarily have to be used at all; it's just there to make the type match that of htraverse. So if someone wants to use a container of MutableArray#, say, they can still do that with a bit of type trickery.)

treeowl · 2018-04-15T22:22:57Z

One thing that's still missing is a way to produce Arrays, SmallArrays, and ByteArrays in one ST action. There's actually a pretty clean way to do that (using an abstract record of freezing functions), but I don't yet know how to make it extensible in an Olegian sense. ByteArray throws a bit of a wrench into most nice approaches, since its s parameter is its last rather than second to last. I think we can leave that as an open question for future research.

treeowl · 2018-04-16T23:52:39Z

I deleted my last comment because it was a bit wrong, thankfully.

I've been thinking some more, and there is another heterogeneous option:

class Freezable m i | m -> i where
  type GetS m :: *
  uf :: m -> ST (GetS m) i

instance Freezable (MutableByteArray s) ByteArray where
  type GetS (MutableByteArray s) = s
  uf = unsafeFreezeByteArray

instance a ~ b => Freezable (MutableArray s a) (Array b) where
  type GetS (MutableArray s a) = s
  uf = unsafeFreezeArray

instance a ~ b => Freezable (SmallMutableArray s a) (SmallArray b) where
  type GetS (SmallMutableArray s a) = s
  uf = unsafeFreezeSmallArray

runVariousHetArraysOf
  :: (forall s1 s2.
       ((forall m i. (Freezable m i, GetS m ~ s1) => m -> ST s2 i) -> t (mut s1) -> ST s2 u))
  -> (forall s. ST s ((t :: k -> *) (mut s)))
  -> u
runVariousHetArraysOf trav m = runST $ m >>= trav uf

The nice thing is that you can run one ST action and get all sorts of different results all at once: Arrays, SmallArrays, ByteArrays, etc. Since the Freezable class is open, it could easily be instantiated for types living in other packages.

treeowl · 2018-04-19T18:46:24Z

Aha! I just realized something. If I modify runArraysOf very slightly, I can actually implement runHetArraysOf on top of it! Specifically, if I change the type signature of runArraysOf to

runArraysOf
  :: (forall s1 s2.
       (forall x. MutableArray s1 x -> ST s2 (Array x)) -> t (mut s1 a) -> ST s2 u)
  -> (forall s. ST s (t (mut s a)))
  -> u

then I can write

runHetArraysOf trav m = runArraysOf (\f -> trav f . unT) $ T <$> m

newtype T (t :: (* -> *) -> *) (msa :: *) = T {unT :: t (Fun msa)}
type family Fun x where
  Fun (f x) = f

cartazio · 2018-04-19T20:53:33Z

One thing I’ve been thinking about is maybe this set of combinators would make more sense to live in lens. A lot of the utility seems like it’d be highest for folks who use lense a lot. Especially the whole higher rank traversal bit. I guess what I mean is that comfort using these combinators seems predicated on using lens, which is down stream of primitive. At a philosophical level, my perspective on the primitive package is that it’s about making the various magic hash mutable data structures ghc provides usable from types in kind Star. Vector and similar higher level libraries are the clients / users of primitive for their data structure needs The only exception is We can’t do that safely for STM Data structures in primitive because of the no nesting transactions requirement of stm precluding execution in any setting / prim monad aside from those with io as the base monad. So I guess what I’m trying to say is that I’m currently inclined to say no to adding these traversal combinators to primitive. Though I’ll ponder this a bit more before I take a strong stance

…

On Thu, Apr 19, 2018 at 2:46 PM David Feuer ***@***.***> wrote: Aha! I just realized something. If I modify runArraysOf very slightly, I can actually implement runHetArraysOf on top of it! Specifically, if I change the type signature of runArraysOf to runArraysOf :: (forall s1 s2. (forall x. MutableArray s1 x -> ST s2 (Array x)) -> t (mut s1 a) -> ST s2 u) -> (forall s. ST s (t (mut s a))) -> u then I can write runHetArraysOf trav m = runArraysOf (\f -> trav f . unT) $ T <$> m newtype T (t :: (* -> *) -> *) (msa :: *) = T {unT :: t (Fun msa)}type family Fun x where Fun (f x) = f — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQwmpclIulqGPz_AJ8czjZxoHIyoYaks5tqNuAgaJpZM4S6VN1> .

treeowl · 2018-04-19T21:02:51Z

That might be reasonable. On the other hand, one of the versions I described uses a Freezable class to be able to freeze anything with a notion of freezing. That Freezable class, to be maximally useful, should live at the lowest level of the module hierarchy that it can. Optimally, that would probably mean GHC.Arr, but starting it somewhere in primitive might be a good compromise. With that class available, anyone can write a Trustworthy version of runArraysOf with their desired type details. On Thu, Apr 19, 2018 at 4:53 PM, Carter Tazio Schonwald < [email protected]> wrote:

…

One thing I’ve been thinking about is maybe this set of combinators would make more sense to live in lens. A lot of the utility seems like it’d be highest for folks who use lense a lot. Especially the whole higher rank traversal bit. I guess what I mean is that comfort using these combinators seems predicated on using lens, which is down stream of primitive. At a philosophical level, my perspective on the primitive package is that it’s about making the various magic hash mutable data structures ghc provides usable from types in kind Star. Vector and similar higher level libraries are the clients / users of primitive for their data structure needs The only exception is We can’t do that safely for STM Data structures in primitive because of the no nesting transactions requirement of stm precluding execution in any setting / prim monad aside from those with io as the base monad. So I guess what I’m trying to say is that I’m currently inclined to say no to adding these traversal combinators to primitive. Though I’ll ponder this a bit more before I take a strong stance On Thu, Apr 19, 2018 at 2:46 PM David Feuer ***@***.***> wrote: > Aha! I just realized something. If I modify runArraysOf very slightly, I > can actually implement runHetArraysOf on top of it! Specifically, if I > change the type signature of runArraysOf to > > runArraysOf > :: (forall s1 s2. > (forall x. MutableArray s1 x -> ST s2 (Array x)) -> t (mut s1 a) -> ST s2 u) > -> (forall s. ST s (t (mut s a))) > -> u > > then I can write > > runHetArraysOf trav m = runArraysOf (\f -> trav f . unT) $ T <$> m > newtype T (t :: (* -> *) -> *) (msa :: *) = T {unT :: t (Fun msa)}type family Fun x where > Fun (f x) = f > > — > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > <#109 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAAQwmpclIulqGPz_ AJ8czjZxoHIyoYaks5tqNuAgaJpZM4S6VN1> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABzi_fUI42nfH3yYqdhb4wxAHjEmQWpdks5tqPlOgaJpZM4S6VN1> .

treeowl · 2018-04-19T23:11:06Z

Let met take that back a bit... The ergonomics of the extreme class-based version don't seem that great. Maybe I can fix that, but maybe not. The array-type-specific versions, on the other hand, have great ergonomics, and there doesn't seem to be anywhere more sensible to put them than here.

…

On Thu, Apr 19, 2018 at 5:02 PM, David Feuer ***@***.***> wrote: That might be reasonable. On the other hand, one of the versions I described uses a Freezable class to be able to freeze anything with a notion of freezing. That Freezable class, to be maximally useful, should live at the lowest level of the module hierarchy that it can. Optimally, that would probably mean GHC.Arr, but starting it somewhere in primitive might be a good compromise. With that class available, anyone can write a Trustworthy version of runArraysOf with their desired type details. On Thu, Apr 19, 2018 at 4:53 PM, Carter Tazio Schonwald < ***@***.***> wrote: > One thing I’ve been thinking about is maybe this set of combinators would > make more sense to live in lens. A lot of the utility seems like it’d be > highest for folks who use lense a lot. Especially the whole higher rank > traversal bit. > > I guess what I mean is that comfort using these combinators seems > predicated on using lens, which is down stream of primitive. > > At a philosophical level, my perspective on the primitive package is that > it’s about making the various magic hash mutable data structures ghc > provides usable from types in kind Star. Vector and similar higher level > libraries are the clients / users of primitive for their data structure > needs > > > The only exception is We can’t do that safely for STM Data structures in > primitive because of the no nesting transactions requirement of stm > precluding execution in any setting / prim monad aside from those with io > as the base monad. > > So I guess what I’m trying to say is that I’m currently inclined to say no > to adding these traversal combinators to primitive. Though I’ll ponder > this a bit more before I take a strong stance > > On Thu, Apr 19, 2018 at 2:46 PM David Feuer ***@***.***> > wrote: > > > Aha! I just realized something. If I modify runArraysOf very slightly, I > > can actually implement runHetArraysOf on top of it! Specifically, if I > > change the type signature of runArraysOf to > > > > runArraysOf > > :: (forall s1 s2. > > (forall x. MutableArray s1 x -> ST s2 (Array x)) -> t (mut s1 a) -> ST > s2 u) > > -> (forall s. ST s (t (mut s a))) > > -> u > > > > then I can write > > > > runHetArraysOf trav m = runArraysOf (\f -> trav f . unT) $ T <$> m > > newtype T (t :: (* -> *) -> *) (msa :: *) = T {unT :: t (Fun msa)}type > family Fun x where > > Fun (f x) = f > > > > — > > You are receiving this because you commented. > > > > > > Reply to this email directly, view it on GitHub > > <#109 (comment)>, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/ > AAAQwmpclIulqGPz_AJ8czjZxoHIyoYaks5tqNuAgaJpZM4S6VN1> > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#109 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABzi_fUI42nfH3yYqdhb4wxAHjEmQWpdks5tqPlOgaJpZM4S6VN1> > . >

andrewthad · 2018-04-20T13:12:21Z

In my mind, runArrays is clearly useful and should be included. I doubt that I personally would ever remember to use anything beyond runArrays. I would probably forget and just do it without the combinators. Here are my general documentation-related concerns:

What is a rank-2 traversal? Are there laws that such a function must satisfy? A google search for "haskell rank 2 traversal" turns up nothing relevant. The docs should probably link to resources or other haskell libraries.
An example of how to use runHetArraysOf is provided. This is good. But, is this function intended to be used with functions from some other library? Are the arguments taken in a particular order for a particular reason? Is this intended to be used with lens or something like lens? If so, it would be nice to include the lens-type-synonym translation of the type signature in the docs.

My other concern is that we have the following implementations:

runArraysOf trav m = runST $ m >>= trav unsafeFreezeArray
runHetArraysOf trav m = runST $ m >>= trav unsafeFreezeArray

These two functions have identical implementations and slightly different type signatures. It bothers me that the type signatures of these two cannot somehow be unified, but there seems to be no way around this.

Also, there's a annoying asymmetry going on. We don't have a runHetArrays function (with a TraversableRank2 typeclass constraint or whatever the appropriate typeclass is named). This makes me more inclined to say that runHetArraysOf belongs in some other package, but it does seem to be pretty useful in the example given, so I guess I like it here.

cartazio · 2018-04-20T15:58:35Z

These are valid points, but let’s make sure we don’t scope creep the role of the primitive package :)

…

On Fri, Apr 20, 2018 at 9:12 AM Andrew Martin ***@***.***> wrote: In my mind, runArrays is clearly useful and should be included. I doubt that I personally would ever remember to use anything beyond runArrays. I would probably forget and just do it without the combinators. Here are my general documentation-related concerns: - What is a rank-2 traversal? Are there laws that such a function must satisfy? A google search for "haskell rank 2 traversal" turns up nothing relevant. The docs should probably link to resources or other haskell libraries. - An example of how to use runHetArraysOf is provided. This is good. But, is this function intended to be used with functions from some other library? Are the arguments taken in a particular order for a particular reason? Is this intended to be used with lens or something like lens? If so, it would be nice to include the lens-type-synonym translation of the type signature in the docs. My other concern is that we have the following implementations: runArraysOf trav m = runST $ m >>= trav unsafeFreezeArray runHetArraysOf trav m = runST $ m >>= trav unsafeFreezeArray These two functions have identical implementations and slightly different type signatures. It bothers me that the type signatures of these two cannot somehow be unified, but there seems to be no way around this. Also, there's a annoying asymmetry going on. We don't have a runHetArrays function (with a TraversableRank2 typeclass constraint or whatever the appropriate typeclass is named). This makes me more inclined to say that runHetArraysOf belongs in some other package, but it does seem to be pretty useful in the example given, so I guess I like it here. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQwuB7HFEjnTyleLuzQOrcwBi-ceP3ks5tqd61gaJpZM4S6VN1> .

treeowl force-pushed the gencreate branch from 6139ae1 to 9f9894c Compare March 25, 2018 22:34

Heterogeneous array creation

d01fdd2

Create arbitrarily many arrays of arbitrarily many types from one `ST` action. Closes haskell#103

treeowl force-pushed the gencreate branch from 9f9894c to d01fdd2 Compare March 25, 2018 23:03

Add more bells and whistles

26ce85b

treeowl mentioned this pull request Mar 27, 2018

Think about namespaces #110

Closed

andrewthad reviewed Mar 29, 2018

View reviewed changes

RyanGlScott reviewed Apr 6, 2018

View reviewed changes

treeowl force-pushed the gencreate branch from 9d0f7a5 to 1c5ccb5 Compare April 14, 2018 21:46

Expand, reduce

e8685d6

treeowl force-pushed the gencreate branch from 1c5ccb5 to e8685d6 Compare April 14, 2018 21:50

More changes

8e71351

* Add notes. * Improve some of the types. * Drop `PolyKinds`. I added it because of unclear thinking, and I don't think any of this benefits meaningfully from having it.

treeowl force-pushed the gencreate branch from 44e4bc9 to 8e71351 Compare April 15, 2018 03:33

RyanGlScott approved these changes Apr 15, 2018

View reviewed changes

treeowl added 3 commits April 15, 2018 14:47

Rename

fe9fef0

Add necessary kind signature

fd2f964

Without `PolyKinds`, this is needed to actually make it work.

Update extensions in Cabal file; update changelog

f783ec9

treeowl commented Apr 15, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Heterogeneous array creation #109

Heterogeneous array creation #109

treeowl commented Mar 25, 2018

treeowl commented Mar 25, 2018

treeowl commented Mar 27, 2018 •

edited

Loading

andrewthad Mar 29, 2018

andrewthad commented Mar 29, 2018

andrewthad commented Mar 29, 2018

treeowl commented Mar 29, 2018 via email

RyanGlScott left a comment •

edited

Loading

RyanGlScott Apr 6, 2018

treeowl Apr 14, 2018

RyanGlScott Apr 6, 2018

treeowl commented Apr 13, 2018

treeowl commented Apr 14, 2018

RyanGlScott commented Apr 14, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

RyanGlScott left a comment

cartazio commented Apr 15, 2018 via email

treeowl Apr 15, 2018

treeowl Apr 15, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 16, 2018

treeowl commented Apr 19, 2018

cartazio commented Apr 19, 2018 via email

treeowl commented Apr 19, 2018 via email

treeowl commented Apr 19, 2018 via email

andrewthad commented Apr 20, 2018

cartazio commented Apr 20, 2018 via email

Heterogeneous array creation #109

Are you sure you want to change the base?

Heterogeneous array creation #109

Conversation

treeowl commented Mar 25, 2018

treeowl commented Mar 25, 2018

treeowl commented Mar 27, 2018 • edited Loading

andrewthad Mar 29, 2018

Choose a reason for hiding this comment

andrewthad commented Mar 29, 2018

andrewthad commented Mar 29, 2018

treeowl commented Mar 29, 2018 via email

RyanGlScott left a comment • edited Loading

Choose a reason for hiding this comment

RyanGlScott Apr 6, 2018

Choose a reason for hiding this comment

treeowl Apr 14, 2018

Choose a reason for hiding this comment

RyanGlScott Apr 6, 2018

Choose a reason for hiding this comment

treeowl commented Apr 13, 2018

treeowl commented Apr 14, 2018

RyanGlScott commented Apr 14, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

treeowl commented Apr 15, 2018

RyanGlScott left a comment

Choose a reason for hiding this comment

cartazio commented Apr 15, 2018 via email

treeowl Apr 15, 2018

Choose a reason for hiding this comment

treeowl Apr 15, 2018

Choose a reason for hiding this comment

treeowl commented Apr 15, 2018

treeowl commented Apr 16, 2018

treeowl commented Apr 19, 2018

cartazio commented Apr 19, 2018 via email

treeowl commented Apr 19, 2018 via email

treeowl commented Apr 19, 2018 via email

andrewthad commented Apr 20, 2018

cartazio commented Apr 20, 2018 via email

treeowl commented Mar 27, 2018 •

edited

Loading

RyanGlScott left a comment •

edited

Loading