bitflippin.com
Steven's personal website and bit flipping laboratory
2024-07-31

Starting a python project

Unable to find a suitable alternative ecosystem despite searching very hard, I started a new python project. I navigated choices around package management, directory structure, and unit testing.

I documented my process setting up the project myproject as a sequence of instructions. Note though that many different or better choices may exist at each step.

Installation and virtual environment

For PopOS, install python-related packages with apt if necessary. To avoid conflicts with other locally run python projects that may require different versions of the same PyPI dependencies, set up a virtual environment.

# Install python
sudo apt install python3-pip

# Support starting python with the 'python' command
sudo apt install python-is-python3

# Python dependency manager
sudo apt install python3-pip

# Python virtual environment management tool
sudo apt install python3-venv

# Optionally make a new directory
mkdir ~/.venvs

# Create a virtual environment for the project
python -m venv ~/.venvs/myproject

Before working with myproject activate the project's virtual envornmnent. The appearance of the name of the virtual environment in the prompt indicates that subsequent commands will run in the context of the virtual environment. After working with myproject deactivate its virtual environment.

# Activate the virtual environment
source ~/.venvs/myproject/bin/activate

# Deactivate the virtual environment
deactivate

Activate the virtual environment if not already activated.

PyPI dependencies

Create a directory to house myproject and navigate there. Optionally make the directory a git repository. List all the myproject dependencies and their versions in a file called requirements.txt in the top level of the repository.

myproject/requirements.txt
mpyc==0.10
numpy==2.0.1
scikit-learn==1.5.1
matplotlib==3.9.1
pandas==2.2.2
pytest==8.3.2

Note the pytest dependency. Be sure to include pytest to follow along with the unit testing section. All the other dependencies are for example purposes. Install the dependencies all at once into the virtual environment using pip. This approach allows others to easily start using myproject too.

python -m pip install -r requirements.txt

Directory structure

Create a directory called myproject to house the project's source code - even if the project directory is already called myproject create another myproject directory inside it. Naming this new directory myproject will cause the pip package to be named after the project too. Thus the imports will be more intuitive, especially in the unit tests which live externally to the package. Add demo.py to the new myproject directory.

myproject/myproject/demo.py
def return_true():
    return True

if __name__ == '__main__':
    print('You ran the demo')

Back in the top level directory create a directory called tests. Inside the new directory create a file called test_demo.py. Note at this stage the import is broken because tests can only see subpackages.

myproject/tests/test_demo.py
import myproject.demo

def test_return_true():
    assert myproject.demo.return_true() == True

Back in the top level directory create a file called setup.py. This file helps pip make myproject into a package.

myproject/setup.py
from setuptools import setup
from setuptools import find_namespace_packages

setup(name='myproject',
    version='1.0',
    packages=find_namespace_packages(include=['myproject']))

Install myproject into the virtual environment as a package with pip. Subsequent changes to the source code of myproject automatically propagate to the installed package without needing to run pip again.

pip install -e .

Useful commands

That completes the setup of the demo project. Optionally interact with the project using any of these commands from the top level directory.

# Activate the virtual environment
source ~/.venvs/myproject/bin/activate

# Run the demo project
python ./myproject/demo.py

# Run the test suite
pytest ./tests

# Deactivate the virtual environment
deactivate

Linter

I abandoned the idea of integrating a linter into the build process. It adds complication, and I expect only minimal participation by others. VSCode offers practical on-the-fly linter options intead.

References