Async Iterator Helpers on Async Generators

[nikolaybotev]

Hi @zloirock,

I have been thinking hard about what the best way is to make Async Iterator Helpers (through your polyfills) as widely available in an as easy manner as possible.

Just for context, as you describe in the README, the problem is that unlike iterator helpers, for async iterator helpers it is not possible to retrieve a reference to the (currently hidden) async iterator prototype object (without using modern syntax, which core-js is constrained from using). So the following does not work out of the box:

require("core-js/proposals/async-iterator-helpers");

async function* gen() {
  yield* [1, 2, 3, 4, 5];
}

gen()
  .drop(1)
  .take(3)
  .filter(n => n != 3)
  .map(n => n * 10)
  .forEach(n => console.log(n));

On modern runtimes with async generator syntax, this can be fixed by leveraging the configurator, with the following snippet at the top of the script:

{
  const asyncGeneratorInstancePrototype = Object.getPrototypeOf(async function*(){}());
  const AsyncGeneratorPrototype = Object.getPrototypeOf(asyncGeneratorInstancePrototype);
  const AsyncIteratorPrototype = Object.getPrototypeOf(AsyncGeneratorPrototype);
  require("core-js/configurator")({ AsyncIteratorPrototype });
}

On older runtimes, when transpiling using babel's standard preset-env, the above does not work, because the transpiler does not install the full standard prototype chain on async generator objects. Luckily, this can be worked around fairly simply, like so:

{
  const asyncGeneratorInstancePrototype = Object.getPrototypeOf(async function*(){}());
  const AsyncGeneratorPrototype = Object.getPrototypeOf(asyncGeneratorInstancePrototype);
  let AsyncIteratorPrototype;
  if (AsyncGeneratorPrototype === Object.prototype) {
    // Fix-up for babel's transform-async-generator-functions
    AsyncIteratorPrototype = {};
    Object.setPrototypeOf(asyncFunctionPrototype, AsyncIteratorPrototype);
  } else {
    AsyncIteratorPrototype = Object.getPrototypeOf(AsyncGeneratorPrototype);
  }
  require("core-js/configurator")({ AsyncIteratorPrototype });
}

Also, it should not be too difficult to fix the transpiler to provide the proper prototype chain, especially since this has already been done for regular generators in the regenerator library, whose approach can be adopted for async generators. Then the above setup code makes async iterator helpers readily available on async generators in all environments. It is also future-proof, in that when async iterator helpers become implemented by runtimes, the native AsyncIterator constructor will be used.

My question then is this: do you think it is reasonable to add a babel plugin (say proposal-async-iterator-helpers, which installs the above snippet in code during transpilation, and runs before the async generator function transformer)? Or do you think there is a better approach?

Thanks for the attention.

[nikolaybotev]

For context, here is some history:

Many years ago, during my career as a Java engineer, I had the opportunity to welcome the benefits of a lazy pull streams library when Java Streams was released as part of the Java Standard Library.

Later, after working for some time using Node.js, I missed Java Streams, and in 2000, not being aware of the Iterator Helpers proposal or your polyfill, which was released in the fall of 2019, I endeavored to quickly prototype a basic lazy pull streams library for JavaScript (and maybe overly ambitiously, called it streams and was graciously granted the npm streams name by its original owner who had abandoned his library under that name at version 0.1 may years prior). That library is essentially identical to Async Iterator Helpers with the addition of a couple of operators, and a handful of utility async iterator sources mixed in. For the last four years, I kept the prototype alive by updating (dev) dependencies every few months, until I was able to start investing time in the project a couple of weeks ago. Luckily, I immediately stumbled across Iterator Helpers while scrolling through node.green and then your polyfills. My goal was always to make the benefits of lazy pull streams available in the JavaScript ecosystem for everyone, so then I realized that the library I started to build had lost its original purpose (or half of it at least).

Since Iterator Helpers are here now, my goal is to produce a library that augments (Async) Iterator Helpers with additional sources and operators in a way seamlessly integrates into Iterator and AsyncIterator abstract classes/constructors from the proposal. I already have implementations of a few additions in the library as I mentioned, and these are only lacking some unit tests and documentation.

But before I even focus on augmenting Iterator Helpers, I wanted to make sure Iterator Helpers in their current form (as implementations in the latest Node.js and browsers, and as core-js polyfills of the Async versions) are as easy to use as that streams library prototype would be on its own, namely:

• Type definitions are available;
• Polyfills work seamlessly across the latest current, and old legacy runtimes with transpilation, and are future-proof ready for future runtimes with a native AsyncIterator constructor;
• Polyfills work on all of built-in iterators (ala Array.values), generators, async generators, and custom classes that extend the abstract Iterator/AsyncIterator constructors;
• Third-party libraries have a recipe for augmenting the Iterator and AsyncIterator abstract classes with additional sources and operators in a way that is also a) backwards compatible with transpiling for older runtimes, b) compatible with core-js polyfills for those two abstract classes/constructors, and c) future-proof and would work on new runtimes with native implementations as they come out.

Type Definitions

On the type definitions front, TypeScript just introduced the Iterator abstract class (and fixed the type definitions of functions that return built-in iterators to be able to support augmenting the Iterator and AsyncIterator classes without polluting the core next/return/throw iterator and async iterator protocols) in 5.6.2 released just last week, and an AsyncIterator abstract class type definition can be derived fairly trivially from the Iterator version. I already have done it and am thinking of attempting to push that to DefinitelyTyped at @types/core-js. I see that the type definitions for core-js at DefinitelyTyped are very incomplete, and I gather from the discussions here that you would like to generate type definitions for core-js from JavaScript comments in the future, which is awesome. It also appears to me that core-js 4 is almost around the corner. Do you have any recommendations with respect to my effort to bring async generator helper typedefs to wide availability - would you like me to put together a PR for core-js to integrate the type definitions in js comments? Is there utility in me working on that at this time, given the huge v4 PR which might get merged anytime and would that cause me grief if I then have to re-integrate the changes on top of v4? Or maybe I could start adding comments on a branch based off of the v4 branch? What do you recommend?

Polyfills Backwards Compatibility and Future-proofing

Iterator

I have come to the conclusion that in the case of Iterator, all outlined requirements are met with the current state of core-js and babel and TypeScript type definitions. I can also reliably retrieve a reference to an Iterator abstract class (from the runtime or core-js) and fall-back to polyfilling it myself, before augmenting it with additional capabilities.

Note: Unlike core-js, which is restricted from using modern syntax, a library providing additional iterator helpers would be best written using modern syntax and then be transpiled for older runtimes if needed by its clients in their app bundling phase. Would you agree?

Side note

One goal of mine with the streams library would be to provide a template for others who want to write additional iterator helpers to be able to do so with ease, and in a way that is also easy to consume in all possible environments. There may be value in that - rxjs has something like 120 operators and if I understand correctly, they maybe had even more in earlier versions, and tried to move away from being a kitchen-sink library... now granted, some of those do not seem applicable to pull streams, but still - it would be nice if it was easy to inject operators in Iterator and AsyncIterator and publish those as a library and have a clear path for consuming those operators in all runtimes via at least the babel +corejs (+webpack?) pipeline.

AsyncIterator

AsyncIterator is a bit more challenging (and at the same time simpler since there are no built-in iterators that predated generators, but those complexities I have gladly discovered are solved in a very robust manner by core-js today - Kudos for that @zloirock this is hard work and core-js makes very difficult problems go away in a very through comprehensive manner! Thank you! And you can quote me on that). This where my question comes from that I put out to you via this github issues channel.

[nikolaybotev]

I tried the above example under ESM and discovered another caveat! ESM loads and initializes all imported modules before executing any code, and so both places in the example have to use dynamic imports (await import()) to make sure the AsyncIteratorPrototype is shared with core-js. This throws the first wrench in my idea of creating a babel transform plugin to seamlessly integrate Async Iterator Helpers.

[nikolaybotev]

Maybe these are not real-world scenarios, but I am also discovering that babel does not seem to transpile ESM await import() to valid require syntax that Node.js (22.8.0) can understand:

  (await Promise.resolve().then(function () {
         ^^^^^^^
    return _interopRequireWildcard(require("core-js/configurator.js"));

SyntaxError: Unexpected identifier 'Promise'
    at wrapSafe (node:internal/modules/cjs/loader:1469:18)

[nikolaybotev]

... I am starting to wonder how deno will behave in this context...

[nikolaybotev]

So the solution to the ESM non-dynamic import problem is to pull the block of code into its own file. This is something to be done from the entry point of individual applications. An npm package could be published that can be simply imported and then included in transpilation. I really hoped this could be done somehow as part of core-js or a combination of core-js and babel utilities, which is why I opened this issue here. However, this does not seem to be the case and I guess that is okay.

I have published two gists with the code in CJS and ESM form for easy adoption in projects via copy-paste:

https://gist.github.com/nikolaybotev/acab33b7387e4773bc1749f1c556efbb

and

https://gist.github.com/nikolaybotev/56e916666c214e0cf501672414e90b1d

[nikolaybotev]

There is a specific babel configuration needed to make this work as well, in terms of ensuring runtime helpers are installed only once on the entire bundle to ensure a single AsyncGenerator function shared across all modules after transpiling. I put together a sample at https://github.com/nikolaybotev/async-iterator-helpers-demo

[nikolaybotev]

There is now a commonjs version of the sample as well in te commonjs branch here: https://github.com/nikolaybotev/async-iterator-helpers-demo/tree/commonjs