Unlock the Power of Pip: Create and Share Installable Python Packages!

Photo on iStockphoto by Diy13

Hi friends, and welcome to another edition where we provide actionable tips and guides for immediate use.

The goal of this guide is to help you create your own Python package and make it installable using pip.

The motivation for this topic arose from our previous discussion on “Communication in Microservices,” where we encountered the need to duplicate certain code across different microservices. If you haven’t already, feel free to check it out. And don’t forget to subscribe to receive notifications whenever a new post is published.

Without further ado, let’s dive right in.

By the end of this guide, you will have the knowledge and skills to accomplish the following:

• Create your own Python package that can be easily installed using pip.
• Configure your package to be private, ensuring that only authorized users can install and download it.
• Configure your package to be publicly available for anyone to use.

Source code:
shared package or library
private-library-compatible-project
public-libray-compatible-project

#Creating your first package

At the core of turning your Python modules or packages into installable packages is the pyproject.toml file. This file serves multiple purposes, including specifying the build system to be used for creating distribution packages for your project. It also contains essential metadata such as the package name, version, and dependencies for your application.

Here is the folder structure for the reusable library we want to build:

├── .gitignore
├── LICENSE
├── pyproject.toml
├── README.md
└── src

pyproject.toml

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "micro_shared_lib"
version = "0.0.1"
authors = [
  { name="Ridwan Yusuf", email="alabarise@gmail.com" },
]
description = "A shared library used in Microservices"
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

[project.urls]
"Homepage" = "https://github.com/ridwanray/micro_shared_lib"

pyproject.toml is a configuration file in Python projects that specifies various project settings and metadata. In addition to the build system, it can contain other information such as the package name, version, description, and more.

For more details on the available metadata fields in the pyproject.toml file, you can refer to the following link: link.
In our case, the name of the package is micro_shared_lib

src

Inside the “src” folder, there is another folder named “micro_shared_lib” that hosts all the files and modules for the package. The structure of the “micro_shared_lib” folder is as follows:

└── micro_shared_lib
    ├── authentication
    │   ├── auth.py
    │   ├── __init__.py
    │   └── permissions.py
    ├── __init__.py
    ├── models.py
    ├── tests
    └── utils.py

Essentially, the content of each file in the “micro_shared_lib” package has been taken from the previous tutorial on communication between microservices (Authentication and Product services). These files contain the necessary code and logic related to authentication, permissions, models, and utilities that are relevant to the microservices.

LICENSE

The “LICENSE” file included in your package provides the licensing information, guiding users on how to utilize your package. You can select a suitable license from the options available on this website: https://choosealicense.com/.

#Use package privately

To use the package privately and restrict access to authorized users only, it is necessary to make the repository for the project on GitHub private. This approach is beneficial for proprietary or internal packages that are intended for specific users or organizations rather than the general public.

Based on the previous project, we have created a separate branch called “private-shared-lib-compatible” specifically for the Product service. This branch is designed to be compatible with the installable library and supports private usage.

├── docker
│   ├── dev
│   │   ├── Dockerfile
│   │   └── entrypoint.sh

To update the Dockerfile and include ‘git’ in the package installation step, modify the relevant line as follows:

RUN  apt-get update \
    && apt-get install -y gcc python3-dev musl-dev libmagic1 libffi-dev git netcat \
    && pip install Pillow

The entrypoint.sh file needs to be updated to include the installation process for the package.

pip install git+https://${GITHUB_TOKEN}@github.com/ridwanray/micro_shared_lib.git@main

The GITHUB_TOKEN needs to be read from the .env file, so make sure to include it as well. The token is generated on GitHub and is used as a means of authentication.

Next, the package is added as part of the installed apps in the Django settings.

core/settings/base.py

INSTALLED_APPS = [
    
    #Third-party Apps
    'micro_shared_lib',
]

With all these in place, we can now reuse modules and utilities from the package wherever they are needed in the application.

core/settings/base.py

REST_FRAMEWORK = {

    "DEFAULT_AUTHENTICATION_CLASSES": (
        "micro_shared_lib.authentication.auth.CustomJWTAuthentication",
    ),
}

product/views.py

from micro_shared_lib.authentication.permissions import IsAdmin

class ProductViewSet(viewsets.ModelViewSet):
    permission_classes = [IsAuthenticated]
    queryset = Product.objects.all()
    pagination_class = None
    http_method_names = ["get", "post", "put", "delete"]
    serializer_class = ProductSerializer

    def perform_create(self, serializer):
        serializer.save(user_id=self.request.user.id)

    def get_permissions(self):
        permission_classes = self.permission_classes
        if self.action == "destroy":
            permission_classes = [IsAdmin]
        return [permission() for permission in permission_classes]

Finally, rebuild and start the container using the following command:

docker compose -f docker-compose.dev.yml up --build

Running the automated test cases again shows that nothing is broken. This ensures that the changes made to the code and the package integration have not introduced any regressions or issues, and the application continues to function as expected.

Test cases

#Use package publicly

To make the package available to the public, we first need to publish it to PyPI (Python Package Index). A new branch named ‘public-compatible-package’ contains the source code for this section.

Run this command to install build (used for building distribution packages) and twine (used for uploading the distribution package)

python -m pip install build twine

From the same directory where the pyproject.toml file is located, execute the following command:

python -m build

The build commands generates two files in the dist directory

├── dist
│   ├── micro_shared_lib-0.0.1-py3-none-any.whl
│   └── micro_shared_lib-0.0.1.tar.gz

Next, upload the contents of the dist directory to PyPI.

twine upload dist/*

You will be prompted to enter your username and password during the upload process.

Finally, the package is uploaded to PyPi and can be accessed through the displayed URL: https://pypi.org/project/micro-shared-lib/

This time, there is no need to make any changes to the Dockerfile since we are not installing the package from a remote repository.

Update the requirements file to include the newly published package and import and use the package wherever it is needed in the app.

app/requirements/base.txt

#other packages
micro-shared-lib==0.0.1 

Update the Django settings file to include the package

core/settings/base.py

INSTALLED_APPS = [
    
    #Third-party Apps
    'micro_shared_lib',
]


REST_FRAMEWORK = {

    "DEFAULT_AUTHENTICATION_CLASSES": (
        "micro_shared_lib.authentication.auth.CustomJWTAuthentication",
    ),
}

Also update views.py

product/views.py

from micro_shared_lib.authentication.permissions import IsAdmin

class ProductViewSet(viewsets.ModelViewSet):
    permission_classes = [IsAuthenticated]
    queryset = Product.objects.all()
    pagination_class = None
    http_method_names = ["get", "post", "put", "delete"]
    serializer_class = ProductSerializer

    def perform_create(self, serializer):
        serializer.save(user_id=self.request.user.id)

    def get_permissions(self):
        permission_classes = self.permission_classes
        if self.action == "destroy":
            permission_classes = [IsAdmin]
        return [permission() for permission in permission_classes]

product/tests/conftest.py

from micro_shared_lib.authentication.auth import UserData

Remember to rebuild the image and start the container using the following command:

docker compose -f docker-compose.dev.yml up --build

Running the test cases again shows that nothing is broken.

 docker exec -it <container_name> pytest -rP -vv

Github repos:
shared package or library
private-library-compatible-project
public-libray-compatible-project

Thanks for reading and feel free to explore my video collection on YouTube for more educational content. Don’t forget to subscribe to the channel to stay updated with future releases.

References:
https://packaging.python.org/en/latest/tutorials/packaging-projects/
https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#declaring-project-metadata
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html