I am currently so confused about installation of my own Python packages... Between setup.pys, sdists and wheels, no matter what I do, I can't seem to achieve what I want - which is to have a bunch of non-.py data files installed and kept in the virtual environment with the same structure I have in my project folder after the installation.

I've read all kinds of documentations, and created a setup.py file that has a data_files field that contains all the data files I need in my installation.

I have the following structure:

.
|__ requirements.txt
    setup.py
    hr_datapool
    |__ __init__.py
    |__ data_needed
        |__ needed_folder
           |__ needed_file.notpy
    |__ one_module
        |__ __init__.py
            |__ misc_tools.py
            |__tests
                |__ test_tools.py
    |__ other_module
            ...

And data_needed contains non-.py data files that are needed for misc_tools.py (and thus tests.py) to run.

Because of this, I added a data_files into my setup.py that contains all the folders and files I need. This I confirmed, everything is there what should be.

And yet, if I do any variation of pip install ., python setup.py install or the likes, the data_files are completely ignored, and/or placed in the wrong directory and don't appear anywhere in the created build or dist folders. And because of this, obviously all my tests fail, since they can't load files that are not there. Neither are they stored in the installation folder on the venv when I sometimes do succeed in copying them, but rather in the root of the venv.

The funny thing is, the files are handled while installing, I keep getting console output when installing with python setup.py install like:

copying data_needed/needed_folder/needed_file.notpy-> /Users/.../venv/hr_datapool/data_needed/needed_folder/

but only if I use python setup.py install, (not when using pip install .).

According to the documentation:

The directory should be a relative path. It is interpreted relative to the installation prefix (Python’s sys.prefix for system installations; site.USER_BASE for user installations). Distutils allows directory to be an absolute installation path, but this is discouraged since it is incompatible with the wheel packaging format. No directory information from files is used to determine the final location of the installed file; only the name of the file is used.

Notice the highlights. However, in my example, it doesn't install relative to the directory containing the package, but it installs into its own folder in the root of the virtual environment, making it practically unreachable from within my code. I made sure I se relative paths in my setup.py, but still this happens.

How can I make sure the required data_files install within the target directory of the module, and not separately into the root of the virtual environment?