is putting the package files directly in src/ supported?

[calestyo]

Hey.

I read through:

• https://setuptools.pypa.io/en/latest/userguide/quickstart.html#package-discovery
• https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

about the various layouts, but couldn't really find out whether the following is properly supported or not:

foo/
├── .git
├── src
│   ├── __init__.py
│   └── foo.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

So the idea would be, that the whole project will ever only consist of one module (with the same name as the project, namely foo).

I do not want to have the *.py source files directly in foo/, which I think would be the single-module distribution.

Neither would I want the flat layout as that would give me a "duplicate" named directory foo, which I think looks ugly, as in:

foo/
├── .git
├── foo
│   ├── __init__.py
│   └── foo.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

For that simple case of having only one package in the project, the src-layout seems also a bit ugly to me:

foo/
├── .git
├── src
│   └── foo
│       ├── __init__.py
│       └── foo.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

I will never have any "sibling" packages of foo so for all pathnames, there is just one more useless .../foo/ in between.

I tried the layout from the very top, where all *.py files are directly in src/. It seems to work (with an --editable I was able to import the package), even with auto-discovery, though - as said - I wasn't able to find any reference that this is actually supported.

Contrary, #3872 seems to even imply that it's not really supported.

However, for me:

$ python3 -m src
/usr/bin/python3: No module named src

seems to fail, as it should, while:

$ python3 -m foo
$

does seem to work.

So question would be:

• Is that supported?
• Are there any things that will possibly not work as expected, with such a layout (like in the sdist, wheel, egg, etc. pp.)?

And if it works, shouldn't that be documented as a supported layout? Guess some people would want to use it :-)

Thanks,
Chris.

[abravalheri]

Hi @calestyo, to be honest, what you called "ugly" is standard practice in the community. I don't think it is worth messing with that for 2 reasons:

1. It will make the project structure/build process more difficult for any eventual collaborator to interact with (as they might be familiar with the community practices).
2. It will require more configuration and complexity in you pyproject.toml.

What really puzzles me is why do you have a folder called foo with a foo.py file inside of it... I mean, this kind of file structure seems to imply that you want an statement like import foo.foo to be possible...

Do you really need that __init__.py file there? Wouldn't the following work for you?

foo/
├── .git
├── src
│   └── foo.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

Or maybe

foo/
├── .git
├── src
│   └── foo
│       ├── __init__.py
│       └── other_submodule.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

[calestyo]

btw: Did I understand this right, that the schema of having projectname/src/modulename.py (where modulename may then be identical to projectname "by coincidence") is supported and proper use of setuptools (without having an __init__.py)?

So the src -layout would actually be:

foo/
├── .git
├── src
│   ├── module_foo.py
│   ├── module_bar.py
│   ├── package1
│   |  ├── __init__.py
│   |  └── submodule1.py
│   └── package2
│       ├── __init__.py
│       └── submoduleA.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

Is that expected to work and proper use?

[webknjaz]

Yes, that should work. Though, I don't remember if you need to declare those modules explicitly, you might want to experiment with that.
Note that things under src/ are going to be installed directly into site-packages/ si make sure those top-level module names are unique and won't collide with someone else's distribution package.

[calestyo]

Well, as written above, it did seem to work. :-)

As for collisions,... that's in principle always possible, isn't it, at least if I name my package like an already existing one. In my case, the Debian packaging on top of the Python-packaging should however prevent (i.e. error out) in such cases.

Would you guys accept a patch that expands:

setuptools/docs/userguide/package_discovery.rst

Lines 152 to 173 in 6083e18

	src-layout
	----------
	The project should contain a ``src`` directory under the project root and
	all modules and packages meant for distribution are placed inside this
	directory::
	project_root_directory
	├── pyproject.toml # AND/OR setup.cfg, setup.py
	├── ...
	└── src/
	└── mypkg/
	├── __init__.py
	├── ...
	├── module.py
	├── subpkg1/
	│   ├── __init__.py
	│   ├── ...
	│   └── module1.py
	└── subpkg2/
	├── __init__.py
	├── ...
	└── module2.py

to also include a top-level module?

[calestyo]

what you called "ugly" is standard practice in the community

😇 Didn't mean to step on anyone's toes.

It's just that with the flat layout, I'd have foo/foo/<files> and with the documented src layout I'd have foo/src/foo/<files>, which both seems unnecessary to me, if there will ever be only one package.

I though every package would need an __init__.py... that's the only reason why I have it there.

You're first suggestion would work for me... as far as I can see it's the same as my first one above, just except for the __init__.py?

The second, AFAICS, would be the src layout as documented here, i.e. with the "superfluous" foo/src/foo/....

I mean it works... but I will only every have one file (the directory foo/ within src/).

Thanks :-)

[abravalheri]

The thing is that the name of the root directory does not really matter and does not have to match the name in PyPI... The mental model is something like project/src/package. The name under src is what is actually relevant.

In general it is indeed recomended that you have only one top-level folder or .py file under src; this way the name on PyPI can match the name on the import statement and you have a 1:1 mapping between distribution and top-level package.

It is possible to have multiple top-level packages under a single distribution in PyPI, but it is not very common, and may be a bit confusing (setuptools itself distributes 2 top-level packages- setuptools and pkg_resources in the same wheel, but that is a bit confusing because you never get to explicitly install pkg_resources - in the future we are looking on ways on how to deprecate pkg_resources and remove it from the wheel).

[calestyo]

What I don't understand then is, how did it get the package name in my case above, where I had that layout:

foo/
├── .git
├── src
│   ├── __init__.py
│   └── foo.py
├── tests
├── LICENCE
├── README
└── pyproject.toml

Actually I'm using this mini-package:

https://gitlab.com/calestyo/python-monitoring-plugin

That's in the src/package/*-layout.

If I just move src/monitoring_plugin/* to src/ and build (python3 -m build) the package again it just seems to work. ^^

The only differences of the generated .tar.gz are:

1. The location of the two *.py files:
   
   In the src/package/*-layout these are at:

src/monitoring_plugin/__init__.py
src/monitoring_plugin/monitoring_plugin.py

In the src/*-layout they're at:
src/init.py
src/monitoring_plugin.py

2. All other files are identical except for top_level.txt which in the first case is:

monitoring_plugin

and in the 2nd:

__init__
monitoring_plugin

When installing it --editable I can just import it under it's name and seemingly use it exactly the same way.

Thanks :-)

[calestyo]

Oh and the generated *.whl is also different, in the first case the pathnames are:

./monitoring_plugin
./monitoring_plugin/__init__.py
./monitoring_plugin/monitoring_plugin.py
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/COPYING
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/LICENCE
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/METADATA
./monitoring_?lugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/WHEEL
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/top_level.txt
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/RECORD

and in the second:

./__init__.py
./monitoring_plugin.py
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/COPYING
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/LICENCE
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/METADATA
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/WHEEL
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/top_level.txt
./monitoring_plugin-1.0.1.dev1+gc0eabe9.d20230708.dist-info/RECORD

Is this the problem? That the *.py files shouldn't be in the "root"?

[calestyo]

I guess this might actually be the problem, cause with the 2nd layout, Python seems to think __init__ would be importable... (despite that failing then):

>>> import __init__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/calestyo/prj/Tf/src/__init__.py", line 6, in <module>
    from .monitoring_plugin import (
ImportError: attempted relative import with no known parent package

whereas in the 1st layout:

>>> import __init__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named '__init__'

[abravalheri]

I guess this might actually be the problem, cause with the 2nd layout, Python seems to think init would be importable... (despite that failing then):

The __init__.py file would indeed be importable given the project layout + configuration.

With that project layout and configuration, what setuptools/python understand is that you want to have 2 top-level single-file importable modules: foo and __init__¹. As top-level single-file modules, you cannot write relative imports (that explains the error you are seeing when you try to import __init__, if you don't have relative import statements, importing __init__ would be perfectly valid¹).

The advantage of both layouts as suggested in the setuptools docs, is that the contents in your .whl file will mimic the file structure under your project root/source folder. If you attempt to do something custom with src/__init__.py, you loose that mapping; which means that things are going to be a bit messy and require more configuration².

Footnotes

1. Despite the fact that having a __init__ top-level module is a messed up thing, since __init__.py have special meaning for Python and is not meant to be used in that way. But if the Python interpreter itself don't complain about that, who am I to complain 😅. ↩ ↩²

2. And yet, you can pursue that route, but then you have to set the tool.setuptools.package-dir configuration mapping accordingly. ↩