Tag: Python

This week, I’m back at my tremendously irregular Django tips series, where I share small pieces of code and approaches to common themes that developers face when working on their web applications.

The topic of today’s post is how to implement a counter that isn’t vulnerable to race conditions. Counting is everywhere, when handling money, when controlling access limits, when scoring, etc.

One common rookie mistake is to do it like this:

model = MyModel.objects.get(id=id)
model.count += 1
model.save(update_fields=["count"])

An approach that is subject to race conditions, as described below:

• Process 1 gets count value (let’s say it is currently 5)
• Process 2 gets counts value
• Process 1 increments and saves
• Process 2 increments and saves

Instead of 7, you end up with 6 in your records.

On a low stakes project or in a situation where precision is not that important, this might do the trick and not become a problem. However, if you need accuracy, you will need to do it differently.

Approach 1

with transaction.atomic():
    model = (
        MyModel.objects.select_for_update()
        .get(id=id)
    )
    model.count += 1
    model.save(update_fields=["count"])

In this approach, when you first fetch the record, you ask for the database to lock it. While you are handling it, no one else can access it.

Since it locks the records, it can create a bottleneck. You will have to evaluate if fits your application’s access patterns. As a rule of thumb, it should be used when you require access to the final value.

Approach 2

from django.db.models import F

model = MyModel.objects.filter(id=id).update(
    count=F("count") + 1
)

In this approach, you don’t lock any values or need to explicitly work inside a transaction. Here, you just tell the database, that it should add 1 to the value that is currently there. The database will take care of atomically incrementing the value.

It should be faster, since multiple processes can access and modify the record “at the same time”. Ideally, you would use it when you don’t need to access the final value.

Approach 3

from django.core.cache import cache

cache.incr(f"mymodel_{id}_count", 1)

If your counter has a limited life-time, and you would rather not pay the cost of a database insertion, using your cache backend could provide you with an even faster method.

The downside, is the level of persistence and the fact that your cache backend needs to support atomic increments. As far as I can tell, you are well served with Redis and Memcached.

For today, this is it. Please let me know if I forgot anything.

Django management commands can be very helpful while developing your application or website, we are very used to runserver, makemigrations, migrate, shell and others. Third party packages often provide extra commands and you can easily add new commands to your own apps.

Today lets take a look at some less known and yet very useful commands that Django provides out of the box.

---

diffsettings

$ python manage.py diffsettings --default path.to.module --output unified

Dealing with multiple environments and debugging their differences is not as rare as we would like. In that particular scenario diffsettings can become quite handy.

Basically, it displays the differences between the current configuration and another settings file. The default settings are used if a module is not provided.

- DEBUG = False
+ DEBUG = True
- EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
+ EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
- TIME_ZONE = 'America/Chicago'
+ TIME_ZONE = 'UTC'
+ USE_SRI = True
- USE_TZ = False
+ USE_TZ = True
...

sendtestemail

$ python manage.py sendtestemail my@address.com

This one does not require an extensive explanation. It lets you test and debug your email configuration by using it to send the following message:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: Test email from host on 2022-04-28 19:08:56.968492+00:00
From: webmaster@localhost
To: my@address.com
Date: Thu, 28 Apr 2022 19:08:56 -0000
Message-ID: <165117293696.405310.3477251481753991809@host>

If you're reading this, it was successful.
-----------------------------------------------------------

inspectdb

$ python manage.py inspectdb

If you are building your project on top of an existing database (managed by other system), inspectdb can look into the schema and generate the respective models for Django’s ORM, making it very easy to start using the data right away. Here’s an example:

# This is an auto-generated Django model module.
# You'll have to do the following manually to clean this up:
#   * Rearrange models' order
#   * Make sure each model has one field with primary_key=True
#   * Make sure each ForeignKey and OneToOneField has `on_delete` set to the desired behavior
#   * Remove `managed = False` lines if you wish to allow Django to create, modify, and delete the table
# Feel free to rename the models, but don't rename db_table values or field names.

...

class AuthPermission(models.Model):
    content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING)
    codename = models.CharField(max_length=100)
    name = models.CharField(max_length=255)

    class Meta:
        managed = False
        db_table = 'auth_permission'
        unique_together = (('content_type', 'codename'),)
...

showmigrations

$ python manage.py showmigrations --verbosity 2

When you need to inspect the current state of the project’s migrations in a given environment the above command is the easiest way to get that information. It will tell you what migrations exist, which ones were applied and when.

admin
 [X] 0001_initial (applied at 2021-01-13 19:49:24)
 [X] 0002_logentry_remove_auto_add (applied at 2021-01-13 19:49:24)
 [X] 0003_logentry_add_action_flag_choices (applied at 2021-01-13 19:49:24)
auth
 [X] 0001_initial (applied at 2021-01-13 19:49:24)
 [X] 0002_alter_permission_name_max_length (applied at 2021-01-13 19:49:24)
 [X] 0003_alter_user_email_max_length (applied at 2021-01-13 19:49:24)
...

---

There are many other useful management commands that are missing in the base Django package, to fill that gap there are some external packages available such as django-extensions. But I will leave those to a future post.

While the admin is a well known and very useful app for your projects, Django also includes another admin package that isn’t as popular (at least I never seen it being heavily used) but that can also be quite handy.

I’m talking about the admindocs app. What it does is to provide documentation for the main components of your project in the Django administration itself.

It takes the existing documentation provided in the code to developers and exposes it to users that have the is_staff flag enabled.

This is what they see:

I can see this being very helpful for small websites that are operated by teams of “non-developers” or even for people providing support to customers. At least when a dedicated and more mature solution for documentation is not available.

To install you just have to:

• Install the docutils package.
• Add django.contrib.admindocs to the installed apps.
• Add path('admin/doc/', include('django.contrib.admindocs.urls')) to the URLs.
• Document your code with “docstrings” and “help_text” attributes.

More details documentation can be found here. And for today, this is it.

In this year’s first issue of my irregular Django quick tips series, lets look at the builtin tools available for managing access control.

The framework offers a comprehensive authentication and authorization system that is able to handle the common requirements of most websites without even needing any external library.

Most of the time, simple websites only make use of the “authentication” features, such as registration, login and logout. On more complex systems only authenticating the users is not enough, since different users or even groups of users will have access to distinct sets of features and data records.

This is when the “authorization” / access control features become handy. As you will see they are very simple to use as soon as you understand the implementation and concepts behind them. Today I’m gonna focus on how to use these permissions on the Admin, perhaps in a future post I can address the usage of permissions on other situations. In any case Django has excellent documentation, so a quick visit to this page will tell you what you need to know.

Under the hood

The above picture is a quick illustration of how this feature is laid out in the database. So a User can belong to multiple groups and have multiple permissions, each Group can also have multiple permissions. So a user has a given permission if it is directly associated with him or or if it is associated with a group the user belongs to.

When a new model is added 4 permissions are created for that particular model, later if we need more we can manually add them. Those permissions are <app>.add_<model>, <app>.view_<model>, <app>.update_<model> and <app>.delete_<model>.

For demonstration purposes I will start with these to show how the admin behaves and then show how to implement an action that’s only executed if the user has the right permission.

The scenario

Lets image we have a “store” with some items being sold and it also has a blog to announce new products and promotions. Here’s what the admin looks like for the “superuser”:

We have several models for the described functionality and on the right you can see that I added a test user. At the start, this test user is just marked as regular “staff” (is_staff=True), without any permissions. For him the admin looks like this:

After logging in, he can’t do anything. The store manager needs the test user to be able to view and edit articles on their blog. Since we expect in the future that multiple users will be able to do this, instead of assigning these permissions directly, lets create a group called “editors” and assign those permissions to that group.

Afterwards we also add the test user to that group (in the user details page). Then when he checks the admin he can see and edit the articles as desired, but not add or delete them.

The actions

Down the line, the test user starts doing other kinds of tasks, one of them being “reviewing the orders and then, if everything is correct, mark them as ready for shipment”. In this case, we don’t want him to be able to edit the order details or change anything else, so the existing “update” permissions cannot be used.

What we need now is to create a custom admin action and a new permission that would let specific users (or groups) execute that action. Lets start with the later:

class Order(models.Model):
    ...
    class Meta:
        ...
        permissions = [("set_order_ready", "Can mark the order as ready for shipment")]

What we are doing above, is telling Django there is one more permission that should be created for this model, a permission that we will use ourselves.

Once this is done (you need to run manage.py migrate), we can now create the action and ensure we check that the user executing it has the newly created permission:

class OrderAdmin(admin.ModelAdmin):
    ...
    actions = ["mark_as_ready"]

    def mark_as_ready(self, request, queryset):
        if request.user.has_perm("shop.set_order_ready"):
            queryset.update(ready=True)
            self.message_user(
                request, "Selected orders marked as ready", messages.SUCCESS
            )
        else:
            self.message_user(
                request, "You are not allowed to execute this action", messages.ERROR
            )

    mark_as_ready.short_description = "Mark selected orders as ready"

As you can see, we first check the user as the right permission, using has_perm and the newly defined permission name before proceeding with the changes.

And boom .. now we have this new feature that only lets certain users mark the orders as ready for shipment. If we try to execute this action with the test user (that does not have yet the required permission):

Finally we just add the permission to the user and it’s done. For today this is it, I hope you find it useful.