Having an empty setuptools directory under /lib/python/site-packages makes python misbehave

[sshedi]

Please check python/cpython#118510

Bug description:
I have a machine where there was a leftover setuptools-57.4.0.dist-info directory after package removal.

/lib/python3.10/site-packages/setuptools-57.4.0.dist-info --> this directory is empty

And newer setuptools /lib/python3.10/site-packages/setuptools-65.5.1-py3.10.egg-info/ is also present and this has content in it.

And I tried building a python project using python3 setup.py build
For example, this project https://github.com/Supervisor/supervisor

And it throws error like below:

root@photon [ ~/supervisor-4.2.2 ]# python3 setup.py build
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'extras_require'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'tests_require'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'include_package_data'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'zip_safe'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'test_suite'
warnings.warn(msg)
/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py:264: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
running build
running build_py
And if I do:
rmdir /lib/python3.10/site-packages/setuptools-57.4.0.dist-info

And then try the build, it works fine.

root@photon [ ~/supervisor-4.2.2 ]# python3 setup.py build
running build
running build_py
running egg_info
writing supervisor.egg-info/PKG-INFO
writing dependency_links to supervisor.egg-info/dependency_links.txt
writing entry points to supervisor.egg-info/entry_points.txt
writing requirements to supervisor.egg-info/requires.txt
writing top-level names to supervisor.egg-info/top_level.txt
reading manifest file 'supervisor.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'docs/.build'
adding license file 'LICENSES.txt'
writing manifest file 'supervisor.egg-info/SOURCES.txt'
/usr/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'supervisor.scripts' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################

...
...
is there a known issue in python3 or a way to tell python to ignore such empty directories?
This seems to be a regression.

PS:
This issue is not seen in python-3.11.7

CPython versions tested on:
3.10

Operating systems tested on:
Linux

[sshedi]

This is happening with setuptools-65.x and I have another vm with python-3.11 & setuptools-69.x and it doesn't happen there.

Can somebody please point me to the fix if there was anything done to solve this issue?

[webknjaz]

What did you do exactly to get your env into a broken state?

[sshedi]

Please see my response below.

[abravalheri]

That happens because importlib.metadata reads from such directories (*.dist-info) and they are not supposed to be empty according to the specification.

Do you know how it became empty in the first place? If everything was installed/removed/updated by pip, I suppose that should not be a problem because pip should clean after itself...

The solution in the end is to indeed remove the empty folder. If you manage to identify how the empty folder was left behind and if that was caused by a specific tool misbehaving, then it can be a good idea to contact that tool developers about the problem.

The second warning you see (once you remove the directory) is because the package you are trying to install has a misconfiguration on the packages field.

[webknjaz]

It is also best to use https://src.fedoraproject.org/rpms/pyproject-rpm-macros nowadays (looks like that document mentions it in part). I think it's even available in EPEL and the latest RHEL streams..

[webknjaz]

If setup.py should not be used, then how to build the package?

That used to be the way in the past and it's been deprecated for a while. The above macros is what most RPM specs migrated in the past years.

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

[webknjaz]

Oh, and if you got leftovers from the previous package version, that indicates a problem with the RPM spec of that version, not with setuptools itself — it must never be used to install things. Some of the latest efforts and the ecosystem came up with way to prevent accidentally managing the same things with several different package managers. But setuptools doesn't know about the INSTALLER file they put on disk, for example. So if someone were reckless enough to install with it outside of the normal RPM packaging process, theoretically, they could get some files on disk that RPM doesn't know about (and so dnf/yum wouldn't be able to delete them).

[sshedi]

Oh, and if you got leftovers from the previous package version, that indicates a problem with the RPM spec of that version

Partially agree with you here but still setuptools considering an empty directory as source of truth and misbehaving seems odd.

[webknjaz]

@sshedi it's not setuptools that does "considering" but Python's built-in import machinery works like this for any importables, not just setuptools.

[sshedi]

It might be the case that the file you linked is using deprecated functionality.

If setup.py should not be used, then how to build the package?

This is the most up-to-date fedora recommendation: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/

@abravalheri This documentation is really confusing.
Check https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_deprecated_macros

And check the sample spec in:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_example_python_spec_file

At one place it says %py3_build is deprecated and in the other page same is used.

[abravalheri]

Hi @sshedi

This documentation is really confusing.

I am sorry to hear that. I am sure that the Fedora folks are trying their best to create good documentation.

Sometimes opensource has its limitations 😅. Maybe it is a good idea to consider contributing to those docs so that future users will have a better time?

It seems that this is the website that the fedora community organise the works on the docs: https://pagure.io/packaging-committee/blob/master/f/guidelines/modules/ROOT/pages/Python.adoc

---

At the top of https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/ there is a disclaimer:

These guidelines are being replaced by a newer version.

So it should be safe to assume that https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_deprecated_macros is the most up-to-date document and the information in there should be taken as the most precise regarding deprecations.

But again, if these docs are problematic, it might be useful to create a conversation with the fedora community that maintains them.

[sshedi]

There are still unknowns.
Check https://src.fedoraproject.org/rpms/cloud-init/blob/rawhide/f/cloud-init.spec#_82

And
https://kojipkgs.fedoraproject.org//packages/cloud-init/24.1.4/2.fc40/data/logs/noarch/build.log

...
+ /usr/bin/python3 setup.py build '--executable=/usr/bin/python3 -sP'
...

%py3_build is still used and works as expected in projects.

And if you take supervisorctl as example,
https://github.com/Supervisor/supervisor/blob/main/docs/installing.rst

The documentation still says to use python3 setup.py install

Is there any documentation that says not to use python3 setup.py install anymore?
I'm sure there are thousands of projects out there which use this approach for build and install.

[abravalheri]

Is there any documentation that says not to use python3 setup.py install anymore?

This is described in https://setuptools.pypa.io/en/latest/userguide/quickstart.html.

.. admonition:: Info: Using ``setup.py``
  :class: seealso

  Setuptools offers first class support for ``setup.py`` files as a configuration
  mechanism.

  It is important to remember, however, that running this file as a
  script (e.g. ``python setup.py sdist``) is strongly **discouraged**, and
  that the majority of the command line interfaces are (or will be) **deprecated**
  (e.g. ``python setup.py install``, ``python setup.py bdist_wininst``, ...).

  We also recommend users to expose as much as possible configuration in a
  more *declarative* way via the :doc:`pyproject.toml <pyproject_config>` or
  :doc:`setup.cfg <declarative_config>`, and keep the ``setup.py`` minimal
  with only the dynamic parts (or even omit it completely if applicable).

  See `Why you shouldn't invoke setup.py directly`_ for more background.

.. _Why you shouldn't invoke setup.py directly: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

[abravalheri]

%py3_build is still used and works as expected in projects.

Sure, but setuptools no longer provides first-class support for this approach (this capability may be removed at any point in time). Also, we don't ensure it plays nice with the rest of the environment, and we don't ensure it follows all Python's standards.

[sshedi]

Thanks a lot for all the info and patiently answering all my queries. Have a nice time ahead.

[sshedi]

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

[webknjaz]

Yes, calling it directly is deprecated, declaring metadata there is not.