Add an __init__.py in repo root to prevent import confusion

[maresb]

It makes me a little bit uneasy to put an __init__.py file outside of a legit Python package, but that's also sort of the point. I'm interested to know what people think.

Description

Related Issue

• Closes BUG: Unable to install pytensor as a package with pip install -e #947

Checklist

• Checked that the pre-commit linting/style checks pass
• Included tests that prove the fix is effective or that the new feature works
• Added necessary documentation (docstrings and/or example notebooks)
• If you are a pro: each commit corresponds to a relevant logical change

Type of change

• New feature / enhancement
• Bug fix
• Documentation
• Maintenance
• Other (please specify):

[ricardoV94]

So why are we special to need this? Should the inner pytensor directory be called src?

[maresb]

We are not at all special; this is a completely general Python annoyance.

I do prefer the src layout since it reduces confusion with the Python package names. However, it wouldn't have prevented #947.

Note: While I prefer src/ I'm also annoyed because src/ is a misnomer. It is actually "the root directory for Python packages", which frustratingly is quite a few too many words to turn into a sensible directory name. I have a slight preference for python/ which is more accurate but slightly less conventional than src/. All options are pretty bad IMHO.

[maresb]

Hmm, 362b1b3 seems to be a workaround for a pytest bug

[maresb]

Oh weird, it works for me locally

[maresb]

Oh, this is pretty funny...

I have pytest v8.1.1 locally, but when I upgrade to v8.2.2 I get the same behavior as the CI. That's annoying!

[maresb]

I'm not entirely serious about actually merging this one. The root of this problem is Python itself, so a workaround like this is likely to introduce more problems than it solves.

This is more of a proof-of-concept, to probe for misconfigurations, and to explore what nasty hacks are being used by mypy and pytest. Indeed as I'm discovering, pytest and mypy use some very nasty hacks.

[maresb]

Going a bit deeper, the PR that causes the root __init__.py to be imported is pytest-dev/pytest#12208. The reason is that this change causes parent modules to be loaded.

But why does the root __init__.py qualify as a parent module? It seems like a few things are going on:

This block is responsible for converting a path into a pair (pkg_root, module_name), e.g. ~/repos/pytensor/pytensor/tensor/math.py gets converted to ("~/repos/pytensor", "pytensor.tensor.math"). But pkg_root is determined by walking up directories as long as __init__.py exists. So with the extra __init__.py, things get wrongly split into ("~/repos", "pytensor.pytensor.tensor.math").

Additionally, it seems that having conftest.pyin the repo root causes __init__.py in the same directory to get imported. I'm actually pretty confused.