Categories
Python

Django Friday Tips: Custom Admin Pages

One of the great builtin features of Django is the admin app. It lets you, among other things, execute the usual CRUD operations on your data, search, filter and execute bulk actions on many records.

However the interface is a bit rigid, by default you have the “dashboard” with the list of models, the page listing your records for each model and the details page for each individual item.

What if you want to display other information to the admins? Such an overview of the system, aggregate data, some statistics, etc.

In this post I will address the steps required to add custom pages to the admin and include them in the main menu of the administration.

The starting point will be the website I used in a previous tip/post:

Regular Django admin dashboard, without any custom page on the menu.
Admin without any custom pages

1. Add a custom Admin Site

The first step is to create a new custom admin website, so we can easily modify its contents and functionality.

from django.contrib import admin

class YourCustomAdminSite(admin.AdminSite):
    pass

Now we will have to use this admin site across your project, which means changing your current admin settings to use this “site” (such as your ModelAdmins and your urls.py.

If the above is too much trouble and requires to many changes, this small “hack” before including the admin URLs will also work:

# urls.py

admin.site.__class__ = YourCustomAdminSite

2. Add a new view

In our new admin site we can now create views as methods, to handle different requests. Like this:

from django.contrib import admin
from django.template.response import TemplateResponse

class YourCustomAdminSite(admin.AdminSite):
    def custom_page(self, request):
        context = {"text": "Hello Admin", 
                   "page_name": "Custom Page"}
        return TemplateResponse(request,
                                "admin/custom_page.html", 
                                context)

3. Add a new template

As you can see in the above python snippet, we are using a template that doesn’t exist yet, so lets create it:

{# templates/admin/custom_page.html #}

{% extends 'admin/change_list.html' %}

{% block pagination %}{% endblock %}
{% block filters %}{% endblock filters %}
{% block object-tools %}{% endblock object-tools %}
{% block search %}{% endblock %}

{% block breadcrumbs %}
<div class="breadcrumbs">
  <a href="{% url 'admin:index' %}">Home</a>
  {% if page_name %} &rsaquo; {{ page_name }}{% endif %}
</div>
{% endblock %}

{% block result_list %}
{{text}}
{% endblock result_list %}

(I chose to extend admin/change_list.html, but the content of this template is up to you, no restrictions here. If you decide to not extend any existing admin template, the last steps of 5. will not apply to your case)

4. Extend the admin URLs

To make this new endpoint accessible, we now have to edit the method that returns all existing URLs of the administration and append our new page.

from django.urls import path
...

class YourCustomAdminSite(admin.AdminSite):
    ...

    def get_urls(
        self,
    ):
        return [
            path(
                "custom_page/",
                self.admin_view(self.custom_page),
                name="custom_page",
            ),
        ] + super().get_urls()

At this point the custom page should be available at /admin/custom_page/, but users will not be able to find it, since no other link or button points to it.

5. Extend the menu items

The final step is to include a link to the new page in the existing menu, in order for it to be easily accessible. First we need to replace the current index_template:

...

class YourCustomAdminSite(admin.AdminSite):
    index_template = "admin/custom_index.html"
    ...

And add the following content to your new template:

{% extends "admin/index.html" %}

{% block content %}
<div id="content-main">
  {% include "admin/custom_app_list.html" with app_list=app_list show_changelinks=True %}
</div>
{% endblock %}

The included custom_app_list.html should look like this:

<div id="extra_links_wrapper" class="module">
    <table>
        <caption>
            <a class="section" title="Custom Pages">Custom Pages</a>
        </caption>
        <tr>
            <th scope="row">
                <a href="{% url 'admin:custom_page' %}">
                    Custom Page
                </a>
            </th>
            <td></td>
        </tr>
    </table>
</div>

{% include 'admin/app_list.html' %}

Basically we added a new section to the list containing a link to our new page.

Screenshot of the index page, with links to the new custom page
Links were added to the index page
Screenshot of the custom page that was created.
The custom page we just created

Looking good, the page is there with our content and is accessible through the index page. However many traditional elements on the page are missing (side menu, logout button, etc). To add them we just need some small changes to our view:

...

class YourCustomAdminSite(admin.AdminSite):
    ...

    def custom_page(self, request):
        context = {
            "text": "Hello Admin",
            "page_name": "Custom Page",
            "app_list": self.get_app_list(request),
            **self.each_context(request),
        }
        return TemplateResponse(request, "admin/custom_page.html", context)

Now it looks a lot better:

Screenshot of the custom page now with the other standard elements included.

But something is still missing. Where are the links to our custom pages?

It seems the standard app_list.html is being used. Lets replace it with our custom one by overriding nav_sidebar.html:

{# templates/admin/nav_sidebar.html #}

{% load i18n %}
<button class="sticky toggle-nav-sidebar" id="toggle-nav-sidebar"
  aria-label="{% translate 'Toggle navigation' %}"></button>
<nav class="sticky" id="nav-sidebar">
  {% include 'admin/custom_app_list.html' with app_list=available_apps show_changelinks=False %}
</nav>

Note: The app where you put the above template must be placed before the “admin” app in your “INSTALLED_APPS” list, otherwise the default template will be used anyway.

Screenshot of the custom page, with all elements and the correct links displayed on the sidebar
Custom admin page with all the elements

And for today, this is it. With the above changes you can add as many custom pages to your admin as you need and have full controls over their functionality.

Categories
Technology and Internet

worker-planet

Some time ago I wrote about using “Cloudflare Workers”, first as a way of adding Content-Security-Policy headers to your pages (when you don’t have other easy ways of doing it) and after I even built an easy to deploy DDNS solution on top of them.

This time I just used this tool to build a “planet” software. You might be wondering about what is a planet, so I will try to explain it in a simple and practical way: A planet is a web page (and RSS feed) that gathers content about a specific topic from multiple other sources and displays them in a chronological order, using the feeds provided by those sources.

If you prefer, you can check Wikipedia’s entry about it.

This is specially useful if you want to cover all the content and activity of a given community in an open way (without extra intermediaries or curators). A few examples are:

There are other software available for this purpose, such as moonmoon, but if you don’t want to manage a server just for this purpose, Cloudflare workers can be very useful.

So I built worker-planet in order to easily create these community pages without having to worry with managing servers. There are many improvements that can be added, but the base functionality is there, and many configurations and theming options are already supported.

The project is open-source and free (as in freedom) software, so please test it and use it as you wish.

Categories
Python Software Development

Django Friday Tips: Password validation

This time I’m gonna address Django’s builtin authentication system, more specifically the ways we can build custom improvements over the already very solid foundations it provides.

The idea for this post came from reading an article summing up some considerations we should have when dealing with passwords. Most of those considerations are about what controls to implement (what “types” of passwords to accept) and how to securely store those passwords. By default Django does the following:

  • Passwords are stored using PBKDF2. There are also other alternatives such as Argon2 and bcrypt, that can be defined in the setting PASSWORD_HASHERS.
  • Every Django release the “strength”/cost of this algorithm is increased. For example, version 3.1 applied 216000 iterations and the last version (3.2 at the time of writing) applies 260000. The migration from one to another is done automatically once the user logs in.
  • There are a set of validators that control the kinds of passwords allowed to be used in the system, such as enforcing a minimum length. These validators are defined on the setting AUTH_PASSWORD_VALIDATORS.

By default when we start a new project these are the included validators :

  • UserAttributeSimilarityValidator
  • MinimumLengthValidator
  • CommonPasswordValidator
  • NumericPasswordValidator

The names are very descriptive and I would say a good starting point. But as the article mentions the next step is to make sure users aren’t reusing previously breached passwords or using passwords that are known to be easily guessed (even when complying with the other rules). CommonPasswordValidator already does part of this job but with a very limited list (20000 entries).

Improving password validation

So for the rest of this post I will show you some ideas on how we can make this even better. More precisely, prevent users from using a known weak password.

1. Use your own list

The easiest approach, but also the more limited one, is providing your own list to `CommonPasswordValidator`, containing more entries than the ones provided by default. The list must be provided as a file with one entry in lower case per line. It can be set like this:

{
  "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
  "OPTIONS": {"password_list_path": "<path_to_your_file>"}
}

2. Use zxcvbn-python

Another approach is to use an existing and well-known library that evaluates the password, compares it with a list of known passwords (30000) but also takes into account slight variations and common patterns.

To use zxcvbn-python we need to implement our own validator, something that isn’t hard and can be done this way:

# <your_app>/validators.py

from django.core.exceptions import ValidationError
from zxcvbn import zxcvbn


class ZxcvbnValidator:
    def __init__(self, min_score=3):
        self.min_score = min_score

    def validate(self, password, user=None):
        user_info = []
        if user:
            user_info = [
                user.email, 
                user.first_name, 
                user.last_name, 
                user.username
            ]
        result = zxcvbn(password, user_inputs=user_info)

        if result.get("score") < self.min_score:
            raise ValidationError(
                "This passoword is too weak",
                code="not_strong_enough",
                params={"min_score": self.min_score},
            )

    def get_help_text(self):
        return "The password must be long and not obvious"

Then we just need to add to the settings just like the other validators. It’s an improvement but we still can do better.

3. Use “have i been pwned?”

As suggested by the article, a good approach is to make use of the biggest source of leaked passwords we have available, haveibeenpwned.com.

The full list is available for download, but I find it hard to justify a 12GiB dependency on most projects. The alternative is to use their API (documentation available here), but again we must build our own validator.

# <your_app>/validators.py

from hashlib import sha1
from io import StringIO

from django.core.exceptions import ValidationError

import requests
from requests.exceptions import RequestException

class LeakedPasswordValidator:
    def validate(self, password, user=None):
        hasher = sha1(password.encode("utf-8"))
        hash = hasher.hexdigest().upper()
        url = "https://api.pwnedpasswords.com/range/"

        try:
            resp = requests.get(f"{url}{hash[:5]}")
            resp.raise_for_status()
        except RequestException:
            raise ValidationError(
                "Unable to evaluate password.",
                code="network_failure",
            )

        lines = StringIO(resp.text).readlines()
        for line in lines:
            suffix = line.split(":")[0]

            if hash == f"{hash[:5]}{suffix}":
                raise ValidationError(
                    "This password has been leaked before",
                    code="leaked_password",
                )

    def get_help_text(self):
        return "Use a different password"

Then add it to the settings.

Edit: As suggested by one reader, instead of this custom implementation we could use pwned-passwords-django (which does practically the same thing).

And for today this is it. If you have any suggestions for other improvements related to this matter, please share them in the comments, I would like to hear about them.

Categories
Python

Django Friday Tips: Subresource Integrity

As you might have guessed from the title, today’s tip is about how to add “Subresource integrity” (SRI) checks to your website’s static assets.

First lets see what SRI is. According to the Mozilla’s Developers Network:

Subresource Integrity (SRI) is a security feature that enables browsers to verify that resources they fetch (for example, from a CDN) are delivered without unexpected manipulation. It works by allowing you to provide a cryptographic hash that a fetched resource must match.

Source: MDN

So basically, if you don’t serve all your static assets and rely on any sort of external provider, you can force the browser to check that the delivered contents are exactly the ones you expect.

To trigger that behavior you just need to add the hash of the content to the integrity attribute of the <script> and/or <link> elements in question.

Something like this:

<script src="https://cdn.jsdelivr.net/npm/vue@2.6.12/dist/vue.min.js" integrity="sha256-KSlsysqp7TXtFo/FHjb1T9b425x3hrvzjMWaJyKbpcI=" crossorigin="anonymous"></script>

Using SRI in a Django project

This is all very nice but adding this info manually isn’t that fun or even practical, when your resources might change frequently or are built dynamically on each deployment.

To help with this task I recently found a little tool called django-sri that automates these steps for you (and is compatible with whitenoise if you happen to use it).

After the install, you just need to replace the {% static ... %} tags in your templates with the new one provided by this package ({% sri_static .. %}) and the integrity attribute will be automatically added.

Categories
Software Development

Documentation done right

One critical piece of the software development process that often gets neglected by companies and also by many open-source projects is explaining how it works and how it can be used to solve the problem in question.

Documentation is often lacking and people have an hard time figuring out how they can use or contribute to a particular piece of software. I think most developers and users have faced this situation at least once.

Looking at it from the other side, it isn’t always easy to pick and share the right information so others can hit the ground running. The fact that not everybody is starting from the same point and have the same goal, makes the job a bit harder.

One approach to solve this problem that I like is Divio’s documentation system.

Divio's documentation system explained. Showing the 4 quadrants and their relations.
The components of Divio’s documentation system.

It splits the problems in 4 areas targeting different stages and needs of the person reading the documentation. Django uses this system and is frequently praised for having great documentation.

From a user point of view it looks solid. You should take a look and apply it on your packages/projects, I surely will.

Categories
Python

Django Friday Tips: Permissions in the Admin

In this year’s first issue of my irregular Django quick tips series, lets look at the builtin tools available for managing access control.

The framework offers a comprehensive authentication and authorization system that is able to handle the common requirements of most websites without even needing any external library.

Most of the time, simple websites only make use of the “authentication” features, such as registration, login and logout. On more complex systems only authenticating the users is not enough, since different users or even groups of users will have access to distinct sets of features and data records.

This is when the “authorization” / access control features become handy. As you will see they are very simple to use as soon as you understand the implementation and concepts behind them. Today I’m gonna focus on how to use these permissions on the Admin, perhaps in a future post I can address the usage of permissions on other situations. In any case Django has excellent documentation, so a quick visit to this page will tell you what you need to know.

Under the hood

Simplified Entity-Relationship diagram of Django's authentication and authorization features.
ER diagram of Django’s “auth” package

The above picture is a quick illustration of how this feature is laid out in the database. So a User can belong to multiple groups and have multiple permissions, each Group can also have multiple permissions. So a user has a given permission if it is directly associated with him or or if it is associated with a group the user belongs to.

When a new model is added 4 permissions are created for that particular model, later if we need more we can manually add them. Those permissions are <app>.add_<model>, <app>.view_<model>, <app>.update_<model> and <app>.delete_<model>.

For demonstration purposes I will start with these to show how the admin behaves and then show how to implement an action that’s only executed if the user has the right permission.

The scenario

Lets image we have a “store” with some items being sold and it also has a blog to announce new products and promotions. Here’s what the admin looks like for the “superuser”:

Admin view, with all models being displayed.
The admin showing all the available models

We have several models for the described functionality and on the right you can see that I added a test user. At the start, this test user is just marked as regular “staff” (is_staff=True), without any permissions. For him the admin looks like this:

A view of Django admin without any model listed.
No permissions

After logging in, he can’t do anything. The store manager needs the test user to be able to view and edit articles on their blog. Since we expect in the future that multiple users will be able to do this, instead of assigning these permissions directly, lets create a group called “editors” and assign those permissions to that group.

Only two permissions for this group of users

Afterwards we also add the test user to that group (in the user details page). Then when he checks the admin he can see and edit the articles as desired, but not add or delete them.

Screenshot of the Django admin, from the perspective of a user with only "view" and "change" permissions.
No “Add” button there

The actions

Down the line, the test user starts doing other kinds of tasks, one of them being “reviewing the orders and then, if everything is correct, mark them as ready for shipment”. In this case, we don’t want him to be able to edit the order details or change anything else, so the existing “update” permissions cannot be used.

What we need now is to create a custom admin action and a new permission that would let specific users (or groups) execute that action. Lets start with the later:

class Order(models.Model):
    ...
    class Meta:
        ...
        permissions = [("set_order_ready", "Can mark the order as ready for shipment")]

What we are doing above, is telling Django there is one more permission that should be created for this model, a permission that we will use ourselves.

Once this is done (you need to run manage.py migrate), we can now create the action and ensure we check that the user executing it has the newly created permission:

class OrderAdmin(admin.ModelAdmin):
    ...
    actions = ["mark_as_ready"]

    def mark_as_ready(self, request, queryset):
        if request.user.has_perm("shop.set_order_ready"):
            queryset.update(ready=True)
            self.message_user(
                request, "Selected orders marked as ready", messages.SUCCESS
            )
        else:
            self.message_user(
                request, "You are not allowed to execute this action", messages.ERROR
            )

    mark_as_ready.short_description = "Mark selected orders as ready"

As you can see, we first check the user as the right permission, using has_perm and the newly defined permission name before proceeding with the changes.

And boom .. now we have this new feature that only lets certain users mark the orders as ready for shipment. If we try to execute this action with the test user (that does not have yet the required permission):

No permission assigned, no action for you sir

Finally we just add the permission to the user and it’s done. For today this is it, I hope you find it useful.

Categories
Software Development Technology and Internet

Mirroring GitHub Repositories

Git by itself is a distributed version control system (a very popular one), but over the years organizations started to rely on some internet services to manage their repositories and those services eventually become the central/single source of truth for their code.

The most well known service out there is GitHub (now owned by Microsoft), which nowadays is synonymous of git for a huge amount of people. Many other services exist, such as Gitlab and BitBucked, but GitHub gained a notoriety above all others, specially for hosting small (and some large) open source projects.

These centralized services provide many more features that help managing, testing and deploying software. Functionality not directly related to the main purpose of git.

Relying on these central services is very useful but as everything in life, it is a trade-off. Many large open source organizations don’t rely on these companies (such as KDE, Gnome, Debian, etc), because the risks involved are not worth the convenience of letting these platforms host their code and other data.

Over time we have been witnessing some of these risks, such as your project (and all the related data) being taken down without you having any chance to defend yourself (Example 1 and Example 2). Very similar to what some content creators have been experiencing with Youtube (I really like this one).

When this happens, your or your organizations don’t lose the code itself since you almost certainly have copies on your own devices (thanks to git), but you lose everything else, issues, projects, automated actions, documentation and essentially the known used by URL of your project.

Since Github is just too convenient to collaborate with other people, we can’t just leave. In this post I explain an easy alternative to minimize the risks described above, that I implemented myself after reading many guides and tools made by others that also tried to address this problem before.

The main idea is to automatically mirror everything in a machine that I own and make it publicly available side by side with the GitHub URLs, the work will still be done in Github but can be easily switched over if something happens.

The software

To achieve the desired outcome I’ve researched a few tools and the one that seemed to fit all my requirements (work with git and be lightweight) was “Gitea“. Next I will describe the steps I took.

The Setup

This part was very simple, I just followed the instructions present on the documentation for a docker based install. Something like this:

version: "3"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest
    container_name: gitea
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"

If you are doing the same, don’t copy the snippet above. Take look here for updated instructions.

Since my website is not supposed to have much concurrent activity, using an SQLite database is more than enough. So after launching the container, I chose this database type and made sure I disabled the all the functionality I won’t need.

Part of Gitea's configuration page. Server and Third-Party Service Settings.
Part of the Gitea’s configuration page

After this step, you should be logged in as an admin. The next step is to create a new migration on the top right menu. We just need to choose the “Github” option and continue. You should see the below screen:

Screenshot of the page that lets the users create a new migration/mirror in Gitea.
Creating a new Github migration/mirror in Gitea.

If you choose This repository will be a mirror option, Gitea will keep your repository and wiki in sync with the original, but unfortunately it will not do the same for issues, labels, milestones and releases. So if you need that information, the best approach is to uncheck this field and do a normal migration. To keep that information updated you will have to repeat this process periodically.

Once migrated, do the same for your other repositories.

Conclusion

Having an alternative with a backup of the general Github data ended up being quite easy to set up. However the mirror feature would be much more valuable if it included the other items available on the standard migration.

During my research for solutions, I found Fossil, which looks very interesting and something that I would like to explore in the future, but at the moment all repositories are based on Git and for practical reasons that won’t change for the time being.

With this change, my public repositories can be found in:

Categories
Python

Django Friday Tips: Inspecting ORM queries

Today lets look at the tools Django provides out of the box to debug the queries made to the database using the ORM.

This isn’t an uncommon task. Almost everyone who works on a non-trivial Django application faces situations where the ORM does not return the correct data or a particular operation as taking too long.

The best way to understand what is happening behind the scenes when you build database queries using your defined models, managers and querysets, is to look at the resulting SQL.

The standard way of doing this is to set the logging configuration to print all queries done by the ORM to the console. This way when you browse your website you can check them in real time. Here is an example config:

LOGGING = {
    ...
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
        },
        ...
    },
    'loggers': {
        ...
        'django.db.backends': {
            'level': 'DEBUG',
            'handlers': ['console', ],
        },
    }
}

The result will be something like this:

...
web_1     | (0.001) SELECT MAX("axes_accessattempt"."failures_since_start") AS "failures_since_start__max" FROM "axes_accessattempt" WHERE ("axes_accessattempt"."ip_address" = '172.18.0.1'::inet AND "axes_accessattempt"."attempt_time" >= '2020-09-18T17:43:19.844650+00:00'::timestamptz); args=(Inet('172.18.0.1'), datetime.datetime(2020, 9, 18, 17, 43, 19, 844650, tzinfo=<UTC>))
web_1     | (0.001) SELECT MAX("axes_accessattempt"."failures_since_start") AS "failures_since_start__max" FROM "axes_accessattempt" WHERE ("axes_accessattempt"."ip_address" = '172.18.0.1'::inet AND "axes_accessattempt"."attempt_time" >= '2020-09-18T17:43:19.844650+00:00'::timestamptz); args=(Inet('172.18.0.1'), datetime.datetime(2020, 9, 18, 17, 43, 19, 844650, tzinfo=<UTC>))
web_1     | Bad Request: /users/login/
web_1     | [18/Sep/2020 18:43:20] "POST /users/login/ HTTP/1.1" 400 2687

Note: The console output will get a bit noisy

Now lets suppose this logging config is turned off by default (for example, in a staging server). You are manually debugging your app using the Django shell and doing some queries to inspect the resulting data. In this case str(queryset.query) is very helpful to check if the query you have built is the one you intended to. Here’s an example:

>>> box_qs = Box.objects.filter(expires_at__gt=timezone.now()).exclude(owner_id=10)
>>> str(box_qs.query)
'SELECT "boxes_box"."id", "boxes_box"."name", "boxes_box"."description", "boxes_box"."uuid", "boxes_box"."owner_id", "boxes_box"."created_at", "boxes_box"."updated_at", "boxes_box"."expires_at", "boxes_box"."status", "boxes_box"."max_messages", "boxes_box"."last_sent_at" FROM "boxes_box" WHERE ("boxes_box"."expires_at" > 2020-09-18 18:06:25.535802+00:00 AND NOT ("boxes_box"."owner_id" = 10))'

If the problem is related to performance, you can check the query plan to see if it hits the right indexes using the .explain() method, like you would normally do in SQL.

>>> print(box_qs.explain(verbose=True))
Seq Scan on public.boxes_box  (cost=0.00..13.00 rows=66 width=370)
  Output: id, name, description, uuid, owner_id, created_at, updated_at, expires_at, status, max_messages, last_sent_at
  Filter: ((boxes_box.expires_at > '2020-09-18 18:06:25.535802+00'::timestamp with time zone) AND (boxes_box.owner_id <> 10))

This is it, I hope you find it useful.

Categories
Random Bits Technology and Internet

Giving a new life to old phones

Nowadays, in some “developed” countries, it is very common for people to have a bunch of old phones stored somewhere in a drawer. Ten years have passed since smartphones became ubiquitous and those devices tend to become unusable very quickly, at least for their primary purpose. Either a small component breaks, the vendor stops providing updates, newer apps don’t support those older versions, etc.

The thing is, these phones are still powerful computers. It would be great if we could give them another life once they are no longer fit for regular day to day use or the owner just wants to try a shiny new device.

I never had many smartphones, mines tend to last many years, but I still have one or two lying around. Recently I started thinking of new uses for them, make them work instead of just gathering dust. A quick search on the internet tells me that many people already had the same idea (I’m quite late to the party) and have been working on cool things to do with these devices.

However, most of these articles just throw the idea at you, without telling you how to do it. Others assume that your device is relatively recent.

Of course the difficulty increases with the age of the phone, in my case the software that I will be able to run on a 10 year old Samsung Galaxy S will not be as easy to find as the software that I can run on another device with just one or two years.

Bellow is a list posts I found online with cool things you can do with your old phones. What sets this list apart from other results is that all the items aren’t just ideas, they contain step by step instructions of how to achieve the end result.

You don’t have to follow the provided instructions rigorously and you should introduce some variations that are more appropriate to your use case.

Have fun and reuse your old devices.

Categories
Python Software Development

Why you shouldn’t remove your package from PyPI

Nowadays most software developed using the Python language relies on external packages (dependencies) to get the job done. Correctly managing this “supply-chain” ends up being very important and having a big impact on the end product.

As a developer you should be cautious about the dependencies you include on your project, as I explained in a previous post, but you are always dependent on the job done by the maintainers of those packages.

As a public package owner/maintainer, you also have to be aware that the code you write, your decisions and your actions will have an impact on the projects that depend directly or indirectly on your package.

With this small introduction we arrive to the topic of this post, which is “What to do as a maintainer when you no longer want to support a given package?” or ” How to properly rename my package?”.

In both of these situations you might think “I will start by removing the package from PyPI”, I hope the next lines will convince you that this is the worst you can do, for two reasons:

  • You will break the code or the build systems of all projects that depend on the current or past versions of your package.
  • You will free the namespace for others to use and if your package is popular enough this might become a juicy target for any malicious actor.

TLDR: your will screw your “users”.

The left-pad incident, while it didn’t happen in the python ecosystem, is a well known example of the first point and shows what happens when a popular package gets removed from the public index.

Malicious actors usually register packages using names that are similar to other popular packages with the hope that a user will end up installing them by mistake, something that already has been found multiple times on PyPI. Now imagine if that package name suddenly becomes available and is already trusted by other projects.

What should you do it then?

Just don’t delete the package.

I admit that in some rare occasions it might be required, but most of the time the best thing to do is to leave it there (specially for open-source ones).

Adding a warning to the code and informing the users in the README file that the package is no longer maintained or safe to use is also a nice thing to do.

A good example of this process being done properly was the renaming of model-mommy to model-bakery, as a user it was painless. Here’s an overview of the steps they took:

  1. A new source code repository was created with the same contents. (This step is optional)
  2. After doing the required changes a new package was uploaded to PyPI.
  3. Deprecation warnings were added to the old code, mentioning the new package.
  4. The documentation was updated mentioning the new package and making it clear the old package will no longer be maintained.
  5. A new release of the old package was created, so the user could see the deprecation warnings.
  6. All further development was done on the new package.
  7. The old code repository was archived.

So here is what is shown every time the test suite of an affected project is executed:

/lib/python3.7/site-packages/model_mommy/__init__.py:7: DeprecationWarning: Important: model_mommy is no longer maintained. Please use model_bakery instead: https://pypi.org/project/model-bakery/

In the end, even though I didn’t update right away, everything kept working and I was constantly reminded that I needed to make the change.