Pre-proposal: Incorporate jupyter-book as a Jupyter sub-project

[choldgraf]

Context

For the last several years, Jupyter Book has been developed as part of the executablebooks/ project. It has grown considerably over that time in usage, and the technical stack behind it has evolved considerably.

In addition, in the last two years there's been an effort within executablebooks to create a document engine that aligns more directly with Jupyter's infrastructure, and that could serve as a back-end for Jupyter Book, rather than Sphinx. This work is at https://mystmd.org and has made significant progress.

The executablebooks/ grant is winding down, and we are thinking about the best "home" for various technical pieces that we've created as part of the project. Some will stay within the executablebooks/ org, and others may be moved to other orgs. This brings me to our proposal:

Proposal summary

We'd like to move several repositories within executablebooks/ into a new GitHub organization called jupyter-book, and incorporate this as a sub-project within Jupyter. See the linked document below for the full proposal, feel free to comment and/or suggest edits.

Proposal text

There are a few documents that can help others understand this proposal, and we invite feedback on the first in particular.

1. Proposal to incorporate a jupyter-book subproject. This is structured as a JEP, and provides the information needed to help the community make a decision. It includes a list of the repositories we intend to incorporate under jupyter-book/, and which will remain where they are.
2. A blog post describing our intent to use MyST as Jupyter Book's document engine. Our hope is to bring the MyST document engine to feature parity with Sphinx, and to then offer it as a replacement back-end for Jupyter Book. This will align Jupyter Book's technical stack with MyST, and offer the unified technical product that drives much of the jupyter-book/ sub-project.
3. List of repositories we propose incorporating under jupyter-book. These are the subset of repositories in executablebooks/ that we believe are within the strategic scope of the proposed jupyter-book/ sub-project.

Proposed authors

• @choldgraf
• @gregcaporaso
• @jstac
• @rowanc1

To do

• Create initial draft
• Iterate with current executablebooks/ steering council -> v0.1
• Create GitHub issue and e-mail the Jupyter EC -> broad-level feedback
• Iterate per feedback -> v1.0alpha
• Create a JEP and post our draft in the Jupyter "enhancement proposals" repository.
• Iterate with EC/SSC feedback -> v1.0
• Vote on the proposal

[choldgraf]

cc @gregcaporaso, and @jstac @rowanc1, who are the other Steering Council members of executablebooks/ and co-proposers here.

[fperez]

This is exciting, many thanks team!!

We had a chance to briefly discuss this during today's EC team meeting. Basic feedback is "please go ahead and move forward", aka 🚀 :)

Specifically, we encourage you to clarify in the proposal the current state of the JupyterBook community/team/maintainers, so it's clear this is a net benefit to the project not just in terms of code/technology, but also of bandwidth to work as part of the overall Project Jupyter extended team.

I am personally very happy to see this development, so full steam ahead with a full proposal! As a reminder, once ready it will need a vote from both the EC and the SSC, which is required for all new project incorporations.

[tonyfast]

cool to see the progress in jupyter books. i can imagine this proposal maturing jupyter book to effectively, or eventually, become another jupyter front end with thebe in the mix. this will add another front end for the accessibility group to consider, and where there are accessibility concerns there are security concerns. i looked through the proposal for both of those topics, but couldn't find any information. do y'all have any plans accessibility or security plans that could be added to the proposal? i think this would help the @jupyter/accessibility-council understand the impact of this decision.

[rowanc1]

Thanks @fperez and @tonyfast, excited to hear the enthusiasm from the EC -- I will aim to add a section to the document in the next few days on accessibility/security, and we will also document the current team+contributors+community.

[rpwagner]

From a security perspective, the first thing I would like to ask is that you consider creating the new repositories in an existing Jupyter GitHub organization. It’s very challenging to manage security policies, roles, and permissions across multiple GitHub organizations. Whenever possible, please to consolidate repositories under existing organizations.

Thanks

[fperez]

@rpwagner - thanks for that input, though I want to make you aware of something that just happened this morning :) GitHub apparently has a new feature called "Enterprises", and today we upgraded Jupyter's org to be an enterprise.

There's still details being worked out, and I didn't follow all the steps - @jasongrout and @blink1073 were closer to the process in case you have questions.

But without in any way contradicting you, I wanted to mention it because it's precisely supposed to help complex projects that are spread out across many orgs. We planned to reach out shortly to various teams to help with the implications of this, I have already seen various subprojects accepting the invitation to join.

And obviously, you and the rest of the security team will have first say on managing security matters under this new setup!

[rpwagner]

Thanks, @fperez! I just saw the email from @jasongrout about the Enterprise upgrade. That will help immensely with applying uniform policies across the orgs. And it reduces my concern about a new org being created.

[fperez]

Yup, we crossed messages - I hadn't seen Jason's email when I replied above. I'm very happy with this development, I suspect it's going to be super useful for us overall.

[choldgraf]

Does that mean we no longer need to address the "separate organization or not" question in the document? I'll admit that I have a strong preference to have a standalone GitHub organization, in order to allow the Jupyter Book community to define its own norms and identity as a project, have more flexibility over the repositories in the org, and be able to give users permissions without worrying they'll spill over into neighboring project repositories. That said, I also don't want to expose Jupyter to unnecessary security risks and I'm happy to discuss pros / cons with the security team if needed.

[krassowski]

thanks for that input, though I want to make you aware of something that just happened this morning :) GitHub apparently has a new feature called "Enterprises", and today we upgraded Jupyter's org to be an enterprise.

Hooray, that's amazing! I was highlighting that going Enterprise would ease organising security managers roles for a long time (jupyter-governance/ec-team-compass#25 (comment), jupyter-governance/ec-team-compass#12 (comment)), pushing back against the security team here.

[Carreau]

any link about this new "enterprise" feature ? googling for it return the classic "github enterprise" cloud hosting. Or is it the same but on github.com ?

Thanks for doing this.

it also would be great to have 1) a channel for such announce 2) a top level issue that explain this is happening, as it's hard to find this in a subthread

[tonyfast]

seems like there are two threads going on here. might make sense to open another issue about github enterprise or rename this one? otherwise, this pre-proposal won't get the attention it needs.

[jasongrout]

it also would be great to have 1) a channel for such announce 2) a top level issue that explain this is happening, as it's hard to find this in a subthread

See jupyter/governance#219. Sorry for the confusion around this.

[fperez]

Agreed @tonyfast - for all further discussion of the Enterprise features, please see jupyter/governance#219 as per @jasongrout's comment. Sorry for any confusion we caused, we got excited about this opportunity knowing the implications, and it happened to be so timely that we mentioned it in various threads before having a chance to make a formal announcement.

Let's get this thread back on track and focused on the jupyter-book conversation.

[rowanc1]

@fperez, @tonyfast @choldgraf I have written a first cut at the accessibility/security in JupyterBook and MyST, as well as including a few links to where we have documented this previously. Feel free to make any comments or anything else that you would like to see us discuss in the document. The sections are in the FAQ section at the bottom of the doc.

[JohanMabille]

Thanks for your work on this project, it's amazing! We will discuss this pre-proposal during the next SSC meetings, my personal feeling is that it should be a JEP.

[choldgraf]

@JohanMabille if it's helpful and possible time-wise, I could attend the meeting and answer any questions people might have. Would that help? (and if so, when is the meeting?)

[minrk]

I support this plan and the creation of the jupyter-book org. Thanks @tonyfast for raising accessibility and @rowanc1 for adding it to the text.

[JohanMabille]

@JohanMabille if it's helpful and possible time-wise, I could attend the meeting and answer any questions people might have. Would that help? (and if so, when is the meeting?)

I think that would definitely help. The SSC working hours is every Monday at 8am Pacific time.

[choldgraf]

I'll plan to join the SSC meeting this coming Monday. If anybody has questions they'd like to ask ahead of time I am happy to answer them here or in another channel.

[choldgraf]

I had a discussion with the SSC today, here are my notes:

• We discussed the relationship between Jupyter Book / MyST and the Jupyter Frontends sub-group (the one that manages jupyterlab+notebook). The SSC asked if it made sense for MyST to be a part of that group. My suggestion was that most of the complexity of MyST is on the back-end and document engine, not front-end. But we did have a front-end component to MyST that would benefit from more connections with the JupyterLab world. Folks generally agreed with this.
• The SSC suggested that we do a little "world tour" of Jupyter if possible, by attending as many sub-group meetings as possible one time to introduce Jupyter Book to the community. I think it's a great idea, and the Jupyter Book team should discuss how to do this.
• The SSC asked if we were going to have something like a weekly / monthly / etc community meeting. I told them we'd be setting up community resources in general, didn't commit to a specific meeting but suggested we'd probably want to do something like this.
• The SSC asked if this had been discussed with the Executable Books team in general, and I confirmed that we've got approval from all of the steering council of that project. We've also had a number of community conversations touching on this over the years (none super recently, but I think the steering council is the key decider here).
• They also made a point I hadn't considered, which is that being part of Jupyter might unlock some developer capacity for teams that are employed at larger companies. So that's a nice added bonus if we are incorporated as a sub-project :-)

Next steps:

• Open a JEP
• Put together a team compass for Jupyter Book and start filling it out
• I've also got a quick PR to the governance repo here: Add e-mails for SSC and EC governance#222

[minrk]

Thanks for the summary, sounds like a great plan!

[fperez]

Fantastic update, many thanks @choldgraf for taking the time to engage with the SSC, and to the entire SSC team for their thoughtful input. I'm delighted with the plan.

[vidartf]

The proposal was accepted, so I'm going to close this pre-proposal. Feel free to reopen if there are still tasks to be done here!

[rowanc1]

Thank you -- it was fantastic to announce this at SciPy last week with some of the Jupyter team!