Category: Technology and Internet

Everything related to technology and the Internet. Tutorials, projects, news…

  • CSP headers using Cloudflare Workers

    Last January I made a small post about setting up a “Content-Security-Policy” header for this blog. On that post I described the steps I took to reach a final result, that I thought was good enough given the “threats” this website faces.

    This process usually isn’t hard If you develop the website’s software and have an high level of control over the development decisions, the end result ends up being a simple yet very strict policy. However if you do not have that degree of control over the code (and do not want to break the functionality) the policy can end up more complex and lax than you were initially hoping for. That’s what happened in my case, since I currently use a standard installation of WordPress for the blog.

    The end result was a different security policy for different routes and sections (this part was not included on the blog post), that made the web-server configuration quite messy.

    (With this intro, you might have already noticed that I’m just making excuses to redo the initial and working implementation, in order to test some sort of new technology)

    Given the blog is behind the Cloudflare CDN and they introduced their “serverless” product called “Workers” a while ago, I decided that I could try to manage the policy dynamically on their servers.

    Browser <--> CF Edge Server <--> Web Server <--> App

    The above line describes the current setup, so instead of adding the CSP header on the “App” or the “Web Server” stages, the header is now added to the response on the last stage before reaching the browser. Let me describe how I’ve done it.

    Cloudflare Workers

    First a very small introduction to Workers, later you can find more detailed information on Workers.dev.

    So, first Cloudflare added the v8 engine to all edge servers that route the traffic of their clients, then more recently they started letting these users write small programs that can run on those servers inside the v8 sandbox.

    The programs are built very similarly to how you would build a service worker (they use the same API), the main difference being where the code runs (browser vs edge server).

    These “serverless” scripts can then be called directly through a specific endpoint provided by Cloudflare. In this case they should create and return a response to the requests.

    Or you can instruct Cloudflare to execute them on specific routes of your website, this means that the worker can generate the response, execute any action before the request reaches your website or change the response that is returned.

    This service is charged based on the number of requests handled by the “workers”.

    The implementation

    Going back to the original problem and based on the above description, we can dynamically introduce or modify the “Content-Security-Policy” for each request that goes through the worker which give us an high degree of flexibility.

    So for my case a simple script like the one below, did the job just fine.

    addEventListener('fetch', event => {
      event.respondWith(handleRequest(event.request))
    })
    
    /**
     * Forward the request and swap the response's CSP header
     * @param {Request} request
     */
    async function handleRequest(request) {
      let policy = "<your-custom-policy-here>"
      let originalResponse = await fetch(request)
      response = new Response(originalResponse.body, originalResponse)
      response.headers.set('Content-Security-Policy', policy)
      return response
    }

    The script just listens for the request, passes it to a handler function (lines 1-3), forwards to the origin server (line 12), grabs the response (line 13), replaced the CSP header with the defined policy (line 14) and then returns the response.

    If I needed something more complex, like making slight changes to the policy depending on the User-Agent to make sure different browsers behave as expected given the different implementations or compatibility issues, it would also be easy. This is something that would be harder to achieve in the config file of a regular web server (nginx, apache, etc).

    Enabling the worker

    Now that the script is done and the worker deployed, in order to make it run on certain requests to my blog, I just had to go to the Cloudflare’s dashboard of my domain, click on the “workers” section and add the routes I want it to be executed:

    cloudflare workers routes modal
    Configuring the routes that will use the worker

    The settings displayed on the above picture will run the worker on all requests to this blog, but is can be made more specific and I can even have multiple workers for different routes.

    Some sort of conclusion

    Despite the use-case described in this post being very simple, there is potential in this new “serverless” offering from Cloudflare. It definitely helped me solve the problem of having different policies for different sections of the website without much trouble.

    In the future I might comeback to it, to explore other user-cases or implementation details.

  • Django Friday Tips: Testing emails

    I haven’t written one of these supposedly weekly posts with small Django tips for a while, but at least I always post them on Fridays.

    This time I gonna address how we can test emails with the tools that Django provides and more precisely how to check the attachments of those emails.

    The testing behavior of emails is very well documented (Django’s documentation is one of the best I’ve seen) and can be found here.

    Summing it up, if you want to test some business logic that sends an email, Django replaces the EMAIL_BACKEND setting with a testing backend during the execution of your test suite and makes the outbox available through django.core.mail.outbox.

    But what about attachments? Since each item on the testing outbox is an instance of the EmailMessage class, it contains an attribute named “attachments” (surprise!) that is list of tuples with all the relevant information:

    ("<filename>", "<contents>", "<mime type>")

    Here is an example:

    # utils.py
    from django.core.mail import EmailMessage
    
    
    def some_function_that_sends_emails():
        msg = EmailMessage(
            subject="Example email",
            body="This is the content of the email",
            from_email="some@email.address",
            to=["destination@email.address"],
        )
        msg.attach("sometext.txt", "The content of the file", "text/plain")
        msg.send()
    
    
    # tests.py
    from django.test import TestCase
    from django.core import mail
    
    from .utils import some_function_that_sends_emails
    
    
    class ExampleEmailTest(TestCase):
        def test_example_function(self):
            some_function_that_sends_emails()
    
            self.assertEqual(len(mail.outbox), 1)
    
            email_message = mail.outbox[0]
            self.assertEqual(email_message.subject, "Example email")
            self.assertEqual(email_message.body, "This is the content of the email")
            self.assertEqual(len(email_message.attachments), 1)
    
            file_name, content, mimetype = email_message.attachments[0]
            self.assertEqual(file_name, "sometext.txt")
            self.assertEqual(content, "The content of the file")
            self.assertEqual(mimetype, "text/plain")

    If you are using pytest-django the same can be achieved with the mailoutbox fixture:

    import pytest
    
    from .utils import some_function_that_sends_emails
    
    
    def test_example_function(mailoutbox):
        some_function_that_sends_emails()
    
        assert len(mailoutbox) == 1
    
        email_message = mailoutbox[0]
        assert email_message.subject == "Example email"
        assert email_message.body == "This is the content of the email"
        assert len(email_message.attachments) == 1
    
        file_name, content, mimetype = email_message.attachments[0]
        assert file_name == "sometext.txt"
        assert content == "The content of the file"
        assert mimetype == "text/plain"

    And this is it for today.

  • Setting up a Content-Security-Policy

    A couple of weeks ago, I gave a small talk on the Madeira Tech Meetup about a set of HTTP headers that could help website owners protect their assets and their users. The slides are available here, just in case you want to take a look.

    The content of the talk is basically a small review about what exists, what each header tries to achieve and how could you use it.

    After the talk I remembered that I didn’t review the heades of this blog for quite sometime. So a quick visit to Mozilla Observatory, a tool that lets you have a quick look of some of the security configurations of your website, gave me an idea of what I needed to improve. This was the result:

    The Content-Security-Header was missing

    So what is a Content Security Policy? On the MDN documentation we can find the following description:

    The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page.

    Mozilla Developer Network

    Summing up, in this header we describe with a certain level of detail the sources from where each type of content can be fetched in order to be allowed and included on a given page/app. The main goal of this type of policy is to mitigate Cross-Site Scripting attacks.

    In order to start building a CSP for this blog a good approach, in my humble opinion, is to start with the more basic and restrictive policy and then proceed evaluating the need for exceptions and only add them when strictly necessary. So here is my first attempt:

    default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

    Lets interpret what it says:

    • default-src: This is the default value for all non-mentioned directives. self means “only things that come from this domain”.
    • object-src: No <object>, <embed> or <applet> here.
    • report-uri: All policy violations should be reported by the browser to this URL.

    The idea was that all styles, scripts and images should be served by this domain, anything external should be blocked. This will also block inline scripts, styles and data images, which are considered unsafe. If for some reason I need to allow this on the blog I could use unsafe-inline, eval and data: on the directive’s definition but in my opinion they should be avoided.

    Now a good way to find out how this policy will affect the website and to understand how it needs to be tuned (or the website changed) we can activate it using the “report only mode:

    Content-Security-Policy-Report-Only: <policy>

    This mode will generate some reports when you (and other users) navigate through the website, they will be printed on the browser’s console and sent to the defined report-uri, but the resources will be loaded anyway.

    Here are some results:

    CSP violations logs on the browser console
    Example of the CSP violations on the browser console

    As an example below is a raw report from one of those violations:

    {
        "csp-report": {
            "blocked-uri": "inline",
            "document-uri": "https://blog.ovalerio.net/",
            "original-policy": "default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly",
            "violated-directive": "default-src"
        }
    }

    After a while I found that:

    • The theme used on this blog used some data: fonts
    • Several inline scripts were being loaded
    • Many inline styles were also being used
    • I have some demos that load content from asciinema.org
    • I often share some videos from Youtube, so I need to allow iframes from that domain
    • Some older posts also embeded from other websites (such as soundcloud)

    So for the blog to work fine with the CSP being enforced, I either had to include some exceptions or fix errors. After evaluating the attack surface and the work required to make the changes I ended up with the following policy:

    Content-Security-Policy-Report-Only: default-src 'self'; script-src 'self' https://asciinema.org 'sha256-A+5+D7+YGeNGrYcTyNB4LNGYdWr35XshEdH/tqROujM=' 'sha256-2N2eS+4Cy0nFISF8T0QGez36fUJfaY+o6QBWxTUYiHc=' 'sha256-AJyUt7CSSRW+BeuiusXDXezlE1Wv2tkQgT5pCnpoL+w=' 'sha256-n3qH1zzzTNXXbWAKXOMmrBzjKgIQZ7G7UFh/pIixNEQ='; style-src 'self' 'sha256-MyyabzyHEWp8TS5S1nthEJ4uLnqD1s3X+OTsB8jcaas=' 'sha256-OyKg6OHgnmapAcgq002yGA58wB21FOR7EcTwPWSs54E='; font-src 'self' data:; img-src 'self' https://secure.gravatar.com; frame-src 'self' https://www.youtube.com https://asciinema.org; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

    A lot more complex than I initially expected it to be, but it’s one of the drawbacks of using a “pre-built” theme on a platform that I didn’t develop. I was able (in the available time) to fix some stuff but fixing everything would take a lot more work.

    All those sha-256 hashes were added to only allow certain inline scripts and styles without allowing everything using unsafe-inline.

    Perhaps in the future I will be able to change to a saner theme/platform, but for the time being this Content-Security-Policy will do the job.

    I started enforcing it (by changing Content-Security-Policy-Report-Only to Content-Security-Policy) just before publishing this blog post, so if anything is broken please let me know.

    I hope this post has been helpful to you and if you didn’t yet implement this header you should give it a try, it might take some effort (depending on the use case) but in the long run I believe it is totally worth it.

  • Firefox’s DoH, the good, the bad and the ugly

    First of all, DoH stands for “DNS over HTTPS”.

    So last week Mozilla announced that future versions of Firefox will use DoH by default, a decision that at first sight might seem innocuous or even good but thanks to some implementation details it ended up being very controversial.

    The reaction that followed the announcement on many technology focused forums and communities was mostly negative (example 1, example 2 and example 3), pointing out many problems, mostly with the way it was implemented and with the default settings used by Mozilla.

    One of the first outcomes was OpenBSD, a well known operating system, announcing that the version of Firefox distributed through its “package manager” will have this change disabled by default.

    In this post I will try to summarize the core of whole controversy and list the pros and cons of it.

    How does a DNS query work?

    In a very brief and not 100% accurate way, when you try to visit a website such as www.some-example.com, your computer first asks a DNS server (resolver) for the IP address of that website, this server address is usually defined on your system either manually (you set it up) or automatically (when you join a given WiFi network, for example, the network will tell you a server you can use).

    That server address generally is set system wide and will be used by all apps. If the server knows the location of the website it will tell you the answer, otherwise it will try to find the location using one of 2 approaches (I will avoid any details here) and come back to you with the result.

    You browser will then use this result to fetch the contents of the website. The bellow image describes this flow:

    Diagram of a client making a DNS query to a local server.
    Source: https://commons.wikimedia.org

    This system is kind of distributed across many entities. Different people across the globe will contact different servers according to their settings and network/location.

    DNS over HTTPS

    The previously described flow already exists for decades and does not change with DoH, what changes is the way you contact the server in order to ask for the website location and the way this data is transmitted.

    While the standard implementation uses UDP and the information travels in cleartext throughout the network (everybody can see it) with DoH this is done as an HTTP request that uses TCP with an encrypted connection, protecting you from malicious actors.

    In the end this should be a good thing, but as we will see later on the post things will do south.

    Current implementation

    A great deal of the discussion this week was sparked by a blog post telling people to turn off Firefox’s DoH, the main complains resolve around not the DoH in itself but the way Mozilla decided to introduce it. Being opt-out and not opt-in, the browser ignoring system configuration and using the servers of a single company by default.

    With the current implementation we end up with:

    The good

    The good part is the obvious reason for using DNS over HTTPS, all your websites queries are encrypted and protected while in transit on the network. It is the extra protection that has been needed for “DNS traffic” for a while.

    The bad

    The first bad part is that the browser will work differently from the rest of the apps which can cause confusion (why this URL work on the browser and not on my app?), the browser no longer will connect to the same server that was defined for the whole systems

    Related to the above problem there is also special network configurations that will stop working such as internal DNS names, rules and filters that are often used on private networks and rely on the internal DNS servers. For these scenarios Mozilla described a series of checks and fallbacks (such as “canary domains”) to accommodate this situation, however they look like fragile hacks.

    The ugly

    The ugly part is that all DNS traffic from the browser will go to a single entity by default, no matter where you are or which network you are using, which raises privacy concerns and increases the centralization of the system. There is the option of manually setting up a different server however 99% of the users will rely on that single provider.

    Conclusion

    The overall the intention was good and having encrypted DNS resolution is something that has been required for a very long time but hasn’t become mainstream yet.

    The core of the problem with Mozilla’s approach is making it “opt-out”, which means all users will now tell a single “Mozilla partner” the websites they visit by default, without being aware of it.

    It will also create some problems to solutions that are deployed network wide and rely on setting certain DNS configurations, since Firefox will not respect them. We can also expect an increased centralization on a system that has been previously working the other way around.

    Lets hope that in the future DoH and other encrypted alternatives become standardized so we can continue to use DNS as we always did and don’t have to manage it on every application.

  • Rust examples and exercises

    Learning to program in Rust is as easy like other languages out there, because it ends up having different constrains and new concepts that you will have to go through, in the beginning everybody fights the compiler at least a little bit.

    I started this journey a while ago, however I’ve been progressing slowly just dedicating some time once in a while when I don’t anything else to do.

    I did what many recommendations on the internet tell you to do, start by reading the official book, that is in fact pretty good. But after reading one or two chapters, we need to practice and play with the language to have a feel of it and explore the new concepts you had just learned.

    So in this small post I just want to share two open resources that can be used while you read the book to practice what you have just learned.

    The first one is a website with examples you can modify and execute live in the browser called Rust by Example.

    The second is an official rust project that will put your knowledge up to a test called Rustlings.

    You can use it like the above video or with rustlings watch that stop and reload each exercise until you solve it.

    This is it, I hope they end being helpful to someone else as well.

  • 8 useful dev dependencies for django projects

    In this post I’m gonna list some very useful tools I often use when developing a Django project. These packages help me improve the development speed, write better code and also find/debug problems faster.

    So lets start:

    Black

    This one is to avoid useless discussions about preferences and taste related to code formatting. Now I just simply install black and let it care of these matters, it doesn’t have any configurations (with one or two exceptions) and if your code does not have any syntax errors it will be automatically formatted according to a “style” that is reasonable.

    Note: Many editors can be configured to automatically run black on every file save.

    https://github.com/python/black

    PyLint

    Using a code linter (a kind of static analysis tool) is also very easy, can be integrated with your editor and allows you to catch many issues without even running your code, such as, missing imports, unused variables, missing parenthesis and other programming errors, etc. There are a few other In this case pylint does the job well and I never bothered to switch.

    https://www.pylint.org/

    Pytest

    Python has a unit testing framework included in its standard library (unittest) that works great, however I found out that there is an external package that makes me more productive and my tests much more clear.

    That package is pytest and once you learn the concepts it is a joy to work with. A nice extra is that it recognizes your older unittest tests and is able to execute them anyway, so no need to refactor the test suite to start using it.

    https://docs.pytest.org/en/latest/

    Pytest-django

    This package, as the name indicates, adds the required support and some useful utilities to test your Django projects using pytest. With it instead of python manage.py test, you will execute just pytest like any other python project.

    https://pytest-django.readthedocs.io

    Django-debug-toolbar

    Debug toolbar is a web panel added to your pages that lets you inspect your requests content, database queries, template generation, etc. It provides lots of useful information in order for the viewer to understand how the whole page rendering is behaving.

    It can also be extended with other plugin that provide more specific information such as flamegraphs, HTML validators and other profilers.

    https://django-debug-toolbar.readthedocs.io

    Django-silk

    If you are developing an API without any HTML pages rendered by Django, django-debug-toobar won’t provide much help, this is where django-silk shines in my humble opinion, it provides many of the same metrics and information on a separate page that can be inspected to debug problems and find performance bottlenecks.

    https://github.com/jazzband/django-silk

    Django-extensions

    This package is kind of a collection of small scripts that provide common functionality that is frequently needed. It contains a set of management commands, such as shell_plus and runserver_plus that are improved versions of the default ones, database visualization tools, debugger tags for the templates, abstract model classes, etc.

    https://django-extensions.readthedocs.io

    Django-mail-panel

    Finally, this one is an email panel for the django-debug-toolbar, that lets you inspect the sent emails while developing your website/webapp, this way you don’t have to configure another service to catch the emails or even read the messages on terminal with django.core.mail.backends.console.EmailBackend, which is not very useful if you are working with HTML templates.

    https://github.com/scuml/django-mail-panel

  • Pixels Camp v3

    Like I did in previous years/versions, this year I participated again on Pixels.camp, a kind of conference plus hackathon. For those who aren’t aware, it is one of the biggest (if not the biggest) technology event in Portugal (from a technical perspective not counting with the Web Summit).

    So, as I did in previous editions, I’m gonna leave here a small list with the nicest talks I was able to attend.

    Lockpicking versus IT security

    This one was super interesting, Walter Belgers showed the audience a set of problems in make locks and compared those mistakes with the ones regularly done by software developers.

    Al least for me the more impressive parts of the whole presentation were the demonstrations of the flaws on regular (and high security) locks.

    Talk description here.


    Containers 101

    “Everybody” uses containers nowadays, on this talk the speaker took a step back and went through the history and the major details behind this technology. Then he shows how you could implement a part of it yourself using common Linux features and tools.

    Talk description here.


    Static and dynamic analysis of events for threat detection

    This one was a nice overview about Siemens infrastructure for threat detection, their approaches and used tools. It was also possible to understand some of the obstacles and challenges a company must address to protect a global infrastructure.

    Talk description here.


    Protecting Crypto exchanges from a new wave of man-in-the-browser attacks

    This presentation used the theme of protecting crypto-currency exchanges but gave lots of good hints on how to improve security of any website or web application. The second half of the talk focused on a kind of attack called man-in-the-browser and focused on a demonstration of it. In my opinion, this last part was weaker and I left with the impression it lacked details about the most crucial part of the attack while spending a lot of time on less important stuff.

    Talk description here.

  • Easy backups with Borg

    One of the oldest and most frequent advises to people working with computers is “create backups of your stuff”. People know about it, they are sick of hearing it, they even advice other people about it, but a large percentage of them don’t do it.

    There are many tools out there to help you fulfill this task, but throughout the years the one I end up relying the most is definitely “Borg“. It is really easy to use, has good documentation and runs very well on Linux machines.

    Here how they describe it:

    BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

    The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

    Borg’s Website

    The built-in encryption and de-duplication features are some of its more important selling points.

    Until recently I’ve had a hard time recommending it to less technical people, since Borg is mostly available through the command line and can take some work to implement the desired backup “policy”. There is a web based graphical user interface but I generally don’t like them as a replacement for native desktop applications.

    However in the last few months I’ve been testing this GUI frontend for Borg, called Vorta, that I think will do the trick for family and friends that ask me what can they use to backup their data.

    The tool is straight forward to use and supports the majority of Borg’s functionality, once you setup the repository you can instruct it to regularly perform your backups and forget about it.

    I’m not gonna describe how to use it, because with a small search on the internet you can quickly find lots of articles with that information.

    The only advise that I would like to leave here about Vorta, is related to the the encryption and the settings chosen when creating your repository. At least on the version I used, the recommend repokey option will store your passphrase on a local SQLite database in clear-text, which is kind of problematic.

    This seems to be viewed as a feature:

    Fallback to save repo passwords. Only used if no Keyring available.

    Github Repository

    But I could not find the documentation about how to avoid this “fallback”.

  • Channels and Webhooks

    Django is an awesome web framework for python and does a really good job, either for building websites or web APIs using Rest Framework. One area where it usually fell short was dealing asynchronous functionality, it wasn’t its original purpose and wasn’t even a thing on the web at the time of its creation.

    The world moved on, web-sockets became a thing and suddenly there was a need to handle persistent connections and to deal with other flows “instead of” (or along with) the traditional request-response scheme.

    In the last few years there has been several cumbersome solutions to integrate web-sockets with Django, some people even moved to other python solutions (losing many of the goodies) in order to be able to support this real-time functionality. It is not just web-sockets, it can be any other kind of persistent connection and/or asynchronous protocol in a microservice architecture for example.

    Of all alternatives the most developer friendly seems to be django-channels, since it lets you keep using familiar django design patterns and integrates in a way that seems it really is part of the framework itself. Last year django-channels saw the release of it second iteration, with a completely different internal design and seems to be stable enough to start building cool things with it, so that is what we will do in this post.

    Webhook logger

    In this blog post I’m gonna explore the version 2 of the package and evaluate how difficult it can be to implement a simple flow using websockets.

    Most of the tutorials I find on the web about this subject try to demonstrate the capabilities of “channels” by implementing a simple real-time chat solution. For this blog post I will try something different and perhaps more useful, at least for developers.

    I will build a simple service to test and debug webhooks (in reality any type of HTTP request). The functionality is minimal and can be described like this:

    • The user visits the website and is given a unique callback URL
    • All requests sent to that callback URL are displayed on the user browser in real-time, with all the information about that request.
    • The user can use that URL in any service that sends requests/webhooks as asynchronous notifications.
    • Many people can have the page open and receive at the same time the information about the incoming requests.
    • No data is stored, if the user reloads the page it can only see new requests.

    In the end the implementation will not differ much from those chat versions, but at least we will end up with something that can be quite handy.

    Note: The final result can be checked on Github, if you prefer to explore while reading the rest of the article.

    Setting up the Django project

    The basic setup is identical to any other Django project, we just create a new one using django_admin startproject webhook_logger and then create a new app using python manage.py startapp callbacks (in this case I just named the app callbacks).

    Since we will not store any information we can remove all database related stuff and even any other extra functionality that will not be used, such as authentication related middleware. I did this on my repository, but it is completely optional and not in the scope of this small post.

    Installing “django-channels”

    After the project is set up we can add the missing piece, the django-channels package, running pip install channels==2.1.6. Then we need to add it to the installed apps:

    INSTALLED_APPS = [
        "django.contrib.staticfiles", 
        "channels", 
    ]

    For this project we will use Redis as a backend for the channel layer, so we need to also install the channels-redis package and add the required configuration:

    CHANNEL_LAYERS = {
        "default": {
            "BACKEND": "channels_redis.core.RedisChannelLayer",
            "CONFIG": {"hosts": [(os.environ.get("REDIS_URL", "127.0.0.1"), 6379)]},
        }
    }

    The above snippet assumes you are running a Redis server instance on your machine, but you can configure it using a environment variable.

    Add websocket’s functionality

    When using “django channels” our code will not differ much from a standard django app, we will still have our views, our models, our templates, etc. For the asynchronous interactions and protocols outside the standard HTTP request-response style, we will use a new concept that is the Consumer with its own routing file outside of default urls.py file.

    So lets add these new files and configurations to our app. First inside our app lets create a consumer.py with the following contents:

    # callbacks/consumers.py
    from channels.generic.websocket import WebsocketConsumer
    from asgiref.sync import async_to_sync
    import json
    
    
    class WebhookConsumer(WebsocketConsumer):
        def connect(self):
            self.callback = self.scope["url_route"]["kwargs"]["uuid"]
            async_to_sync(self.channel_layer.group_add)(self.callback, self.channel_name)
            self.accept()
    
        def disconnect(self, close_code):
            async_to_sync(self.channel_layer.group_discard)(
                self.callback, self.channel_name
            )
    
        def receive(self, text_data):
            # Discard all received data
            pass
    
        def new_request(self, event):
            self.send(text_data=json.dumps(event["data"]))

    Basically we extend the standard WebsocketConsumer and override the standard methods. A consumer instance will be created for each websocket connection that is made to the server. Let me explain a little bit what is going on the above snippet:

    • connect – When a new websocket connection is made, we check which callback it desires to receive information and attach the consumer to the related group ( a group is a way to broadcast a message to several consumers)
    • disconnect – As the name suggests, when we lose a connection we remove the “consumer” from the group.
    • receive – This is a standard method for receiving any data sent by the other end of the connection (in this case the browser). Since we do not want to receive any data, lets just discard it.
    • new_request – This is a custom method for handling data about a given request/webhook received by the system. These messages are submitted to the group with the type new_request.

    You might also be a little confused with that async_to_sync function that is imported and used to call channel_layer methods, but the explanation is simple, since those methods are asynchronous and our consumer is standard synchronous code we have to execute them synchronously. That function and sync_to_async are two very helpful utilities to deal with these scenarios, for details about how they work please check this blog post.

    Now that we have a working consumer, we need to take care of the routing so it is accessible to the outside world. Lets add an app level routing.py file:

    # callbacks/routing.py
    from django.conf.urls import url
    
    from .consumers import WebhookConsumer
    
    websocket_urlpatterns = [url(r"^ws/callback/(?P<uuid>[^/]+)/$", WebhookConsumer)]

    Here we use a very similar pattern (like the well known url_patterns) to link our consumer class to connections of certain url. In this case our users could connect to an URL that contains the id (uuid) of the callback that they want to be notified about new events/requests.

    Finally for our consumer to be available to the public we will need to create a root routing file for our project. It looks like this:

    # <project_name>/routing.py
    from channels.routing import ProtocolTypeRouter, URLRouter
    from callbacks.routing import websocket_urlpatterns
    
    application = ProtocolTypeRouter({"websocket": URLRouter(websocket_urlpatterns)})

    Here we use the ProtocolTypeRouter as the main entry point, so what is does is:

    It lets you dispatch to one of a number of other ASGI applications based on the type value present in the scope. Protocols will define a fixed type value that their scope contains, so you can use this to distinguish between incoming connection types.

    Django Channels Documentation

    We just defined the websocket protocol and used the URLRouter to point to our previous defined websocket urls.

    The rest of the app

    At this moment we are able to receive new websocket connections and send to those clients live data using the new_request method on the client. However at the moment we do not have information to send, since we haven’t yet created the endpoints that will receive the requests and forward their data to our consumer.

    For this purpose lets create a simple class based view, it will receive any type of HTTP request (including the webhooks we want to inspect) and forward them to the consumers that are listening of that specific uuid:

    # callbacks/views.py
    
    class CallbackView(View):
        def dispatch(self, request, *args, **kwargs):
            channel_layer = get_channel_layer()
            async_to_sync(channel_layer.group_send)(
                kwargs["uuid"], {"type": "new_request", "data": self._request_data(request)}
            )
            return HttpResponse()

    In the above snippet, we get the channel layer, send the request data to the group and return a successful response to calling entity (lets ignore what the self._request_data(request) call does and assume it returns all the relevant information we need).

    One important piece of information is that the value of the type key on the data that is used for the group_send call, is the method that will be called on the websocket’s consumer we defined earlier.

    Now we just need to expose this on our urls.py file and the core of our system is done.

    # <project_name>/urls.py
    
    from django.urls import path
    from callbacks.views import CallbackView
    
    urlpatterns = [
        path("<uuid>", CallbackView.as_view(), name="callback-submit"),
    ]

    The rest of our application is just standard Django web app development, that part I will not cover in this blog post. You will need to create a page and use JavaScript in order to connect the websocket. You can check a working example of this system in the following URL :

    http://webhook-logger.ovalerio.net

    For more details just check the code repository on Github.

    Deploying

    I not going to explore the details about the topic of deployments but someone else wrote a pretty straightforward blog post on how to do it for production projects that use Django channels. You can check it here.

    Final thoughts

    With django-channels building real-time web apps or projects that deal with other protocols other than HTTP becomes really simple. I do think it is a great addition to the current ecosystem, it certainly is an option I will consider from now on for these tasks.

    Have you ever used it? do you any strong opinion about it? let me know on the comments section.

    Final Note: It seems based on recent messages on the mailing list that the project might suspend its developments in its future if it doesn’t find new maintainers. It would definitely be a shame, since it has a lot of potential. Lets see how it goes.

  • Experiment: ChainSentinel.co

    The amount and maturity of the tools available to help developers in process of building new applications and products is often crucial to the success of any given technology, platform or ecosystem.

    Nowadays in this blockchain trend we are witnessing, the front runner and most mature contender is Ethereum, for sure. The quality and quantity of the tools and content (documentation, tutorials, etc) available to developers in order to build on top of it, is miles away from the competition.

    Recently I’ve been working and experimenting with NEO blockchain (as you can see on some of my previous posts), a team that I took part even won an award of merit in their most recent dApp competition (Github repository). During that period we felt the pain of the lack of maturity and documentation that affected this new “ecosystem”.

    Things got better, but there are a few things still missing, such as tools that help you integrate your applications and services with the blockchain, tools to make the developer’s life easier and tools to make their dApps more useful, such as the equivalent to Ethereum’s web3.js and Metamaskextension for example.

    Even though you can achieve a lot with NEO’s JSON RPC API and through running your own node, I still think things should be easier. So at the last Whitesmith hackathon we’ve tried to address a subset of these pains.

    We’ve put together, on that limited timeframe, a simple and rough service that delivers blockchain events as traditional Webhooks (websockets are planned) to make it easier for everybody to interact in real-time with any smart-contract.

    We are looking for feedback to understand if it is something more developers also need and in that case work towards improving the service. Feel free to take a look at:

    https://chainsentinel.co

  • Django Friday Tips: Links that maintain the current query params

    Basically when you are building a simple page that displays a list of items that contain a few filters you might want to maintain them while navigating, for example while browser through the pages of results.

    Nowadays many of this kind of pages are rendered client-side using libraries such as vue and react, so this doesn’t pose much of a problem since the state is easily managed and requests are generated according to that state.

    But what if you are building a simple page/website using traditional server-side rendered pages (that for many purposes is totally appropriate)? Generating the pagination this way while maintaining the current selected filters (and other query params) might give you more work and trouble than it should.

    So today I’m going to present you a quick solution in the form of a template tag that can help you easily handle that situation. With a quick search on the Internet you will almost for sure find the following answer:

    @register.simple_tag
    def url_replace(request, field, value):
        dict_ = request.GET.copy()
        dict_[field] = value
        return dict_.urlencode()
    

    Which is great and work for almost scenario that comes to mind, but I think it can be improved a little bit, so like one of lower ranked answers suggests, we can change it to handle more than one query parameter while maintaining the others:

    @register.simple_tag(takes_context=True)
    def updated_params(context, **kwargs):
        dict_ = context['request'].GET.copy()
        for k, v in kwargs.items():
            dict_[k] = v
        return dict_.urlencode()

    As you can see, with takes_context we no longer have to repeatedly pass the request object to the template tag and we can give it any number of parameters.

    The main difference for the suggestion on “Stack Overflow” it that this version allows for repeating query params, because we don’t convert the QueryDict to a dict.  Now you just need to use it in your templates like this:

    https://example.ovalerio.net?{% updated_params page=2 something='else' %}
  • Looking for security issues on your python projects

    In today’s post I will introduce a few open-source tools, that can help you improve the security of any of your python projects and detect possible vulnerabilities early on.

    These tools are quite well known in the python community and used together will provide you with great feedback about common issues and pitfalls.

    Safety and Piprot

    As I discussed some time ago on a post about managing dependencies and the importance of checking them for known issues, in python there is a tool that compares the items of your requirements.txt with a database of known vulnerable versions. It is called safety (repository)  and can be used like this:

    safety check --full-report -r requirements.txt

    If you already use pipenv safety is already incorporated and can be used by running: pipenv check (more info here).

    Since the older the dependencies are, the higher the probability of a certain package containing bugs and issues, another great tool that can help you with this is piprot (repository).

    It will check all items on your requirements.txt and tell you how outdated they are.

    Bandit

    The next tool in the line is bandit, which is a static analyzer for python built by the Open Stack Security Project, it checks your codebase for common security problems and programming  mistakes that might compromise your application.

    It will find cases of hardcoded passwords, bad SSL defaults, usage of eval, weak ciphers, different “injection” possibilities, etc.

    It doesn’t require much configuration and you can easily add it to your project. You can find more on the official repository.

    Python Taint

    This last one only applies if you are building a web application and requires a little bit more effort to integrate in your project (at its current state).

    Python Taint (pyt) is a static analyzer that tries to find spots were your code might be vulnerable to common types of problems that affect websites and web apps, such as SQL injection, cross site scripting (XSS), etc.

    The repository can be found here.

    If you are using Django, after using pyt you might also want to run the built in manage.py check command, (as discussed in a previous post) to verify some specific configurations of the framework present on your project.

     

  • Finding blockchain challenges to learn and practice

    With “Blockchain” technologies gaining a certain momentum in the last few years, we have witnessed a spread of different implementations with many variations in several aspects, even tough they are  built over the same principals.

    This field is not easy to get started with and after taking the time to learn one approach, lets say “smart contract development in ethereum”, there might be less motivation to explore other technologies and networks.

    An approach that is being used by the proponents of these networks, is to organize challenges and competitions to try to call the attention of other developers. With these competitions you can learn, explore, solve certain problems and in the end even win some reward (in the form of cryptocurrency/tokens). Most of these challenges happen online and you don’t even have to leave the comfort of your home.

    For those interested, the most difficult part is finding the currently open challenges and competitions, so last week I put together a bare-bones page (plus RSS feed and calendar) aggregating the challenges that I find on the web.

    You can access the page in the following URL: https://blockchain-challenges.ovalerio.net

    I will try to keep it updated. In case you find it useful and discover any unlisted challenge, please submit it there, so it can be added to the list.

    Note 22/04/2020: Given the website was no longer updated or maintained, I took it of the air.

  • Creating a NEP5 Token on NEO network

    In this post I will try to introduce a way of creating a digital token, that doesn’t rely on the Ethereum network, neither on their ERC20 standard.

    Other than Ethereum there are several other blockchain implementations that ofter the possibility of running smart-contracts and therefore allowing the creation of new digital tokens.

    The option that I will address today is NEO, through its NEP5 standard. I will not elaborate on the advantages and disadvantages between the 2 approaches, given that it will depend on many factors and your current situation (perhaps a future post, but there are many other publications that already did that exercise).

    A key difference for tokens based on NEO is that you have multiple choices regarding how do you write your smart-contract, instead of relying just on `Solidity`. If this is better or worse, that’s another discussion as well.

    For today’s post I will write everything in python, and use neo-python to compile the smart-contract and run the network node that will be used to deploy it on the blockchain.


    Brief Introduction to NEO

    Shortly, NEO is a blockchain project that has been around for a while, that intends build a smart economy over a distributed network. In other words and in practice, it is a platform with many similarities to Ethereum, with some key differences on some philosophical aspects and technical decisions.

    It was founded in China during the year of 2014 and has been gaining momentum recently, outside of their home country there are many people working on this platform, perhaps the most well known is an international community called City of Zion (CoZ)that develops open source tools and organizes all kinds of events and initiatives to support this project.

    As you will see in a later stage of this post, we will use one of CoZ’s tools to connect to the network and to manage “our” smart-contracts.

    The NEP5 Standard

    As it happens with Ethereum, NEO allows you to run smart contracts, therefore you can create your own tokens over this network/platform and for example run an “Initial Coin Offering” (ICO).

    The ecosystem benefits if all these tokens do have a common interface, so like the ERC20 (now EIP20) there is the NEP5 standard (the document can be found here). Complying with this common interface is highly advisable, since it will make the management of the token using most wallets easier for your users.

    As a small overview, so you can be aware of the simplicity of the proposed interface, your smart-contract should implement at least the following methods: totalSupply, name, symbol, decimals, balanceOfand transfer.

    Of course there are many other things that are required to make your smart-contract and respective token usable, such as initialization procedures and configuration of certain parameters like the total amount of tokens, how many decimals it has, which wallet should be the owner, etc. In this post we will stick with the basics.

    The smart-contract

    As it was said before, I will use python throughout the remaining of this post. Since the examples present in the proposal document are in C#, I will base the rest of the article on “NEO ICO Template” provided by the Neon Exchange, which is implemented in python, complies with NEP5 and has all the remaining utilities implemented.

    A detailed post about how to use this template already exists and can be found here. Some sections of that article are already a bit out-dated, but it remains very informative nevertheless. To avoid duplicating content, I will provide a lighter version and show how we can make use of the built-in neo-python features instead of calling the smart-contract methods directly, demonstrating how NEP5 standard can also make you users life easier.

    The Node and deployment

    So lets start!

    Assuming you already have the neo-python installed (if you don’t, you can follow the instructions here), the first think you should do is to launch the `prompt` and open your wallet:

    $ np-prompt -p
    ...
    open wallet {{wallet_path}}

    If you cloned the repository it will be something like:

    $ python neo/bin/prompt -p 
    ...
    open wallet {{wallet_path}}

    Next we will download the example smart-contract code in another terminal window:

    $ git clone git@github.com:neonexchange/neo-ico-template.git

    Before we build the smart-contract, we will need to edit a few settings that will be exclusive for our token. On the {{path_to_smartcontract}}/nex/token.py file, lets edit a few parameters (there are several others you could change, but lets stick to the basics here):

    # nex/token.py
    TOKEN_NAME = 'Example ICO Token'
    TOKEN_SYMBOL = 'EIT'
    TOKEN_OWNER = b'{{your_wallet_script_hash}}'
    

    To get the {{your_wallet_script_hash}} just type wallet on the terminal window running neo-python and you should see something like this [I 180506 20:50:38 UserWallet:538] Script hash b'SCRIPT HASH HERE' <class 'bytes'> printed on the terminal. Just copy it to your contract code and you’re done.

    The other options include changing the amount of tokens, how many will be initially “minted” and added to the owner’s wallet, etc.

    Now it is time to “compile” the smart-contract from python to the NEO’s virtual machine “format”. To do that, run the following command on the “prompt”:

    build {{path_to_smart-contract}}/ico_template.py 0710 05 True False

    The extra arguments are:

    • 0710 – Parameter types ( 07 is string, 10is a array).
    • 05 – Is the return type, in this case it means bytearray.
    • True – It requires storage.
    • False – It doesn’t use dynamic invocation.

    Now an ico_template.avm should had been created, we will use this file to deploy our smart-contract to the blockchain. To do so, you will need `GAS` (400 of it, check here the values) and since this is just a test a better approach is to use the test network (or even a private network) in order to avoid wasting funds. To deploy the smart-contract you should run:

    import contract {{path_to}}/ico_template.avm 0710 05 True False

    and follow the interactive instructions. After this final step the smart-contract should be ready to use.

    Using the newly created token

    Now that everything is deployed and we are ready to start using our new token, the first thing that we need to do is to “run” the deploy instruction in order to setup the initial amount of tokens. To deploy we need to find get the hash of the imported contract and invoke it with the deploy parameter.

    contract search {{query for your contract}}
    # grab the "hash" value
    testinvoke {{hash}} deploy []
    

    Then we can add this token to our wallet and interact with it using a friendlier interface than having to invoke manually the methods of the contract like we did with the deploy action. We achieve this with the command: import token {{hash}}.

    At this point you will be able to see your new token balance when you check your wallet, something similar to the following snippet:

        "synced_balances": [
            "[NEO]: 1000.0 ",
            "[NEOGas]: 25099.999 ",
            "[EIT]: 2499980 "
        ],
    

    From now on, to send tokens to someone else, instead of doing something like this:

    testinvoke 0xfd941304d9cf36f31cd141c7c7029d81b1efa4f3 transfer ["AUiPgh9684vjScBDJ5FFsYzBWyJjf6GQ6K","ASfh5fCf6jZ2RxKNoDfN6dN817B9kaNRgY", "10"]

    you have this friendlier interface:

    wallet tkn_send EIP AUiPgh9684vjScBDJ5FFsYzBWyJjf6GQ6K ASfh5fCf6jZ2RxKNoDfN6dN817B9kaNRgY 10

    If you check the help command, you will see that you have a few helper methods to easily interact with your NEP5 token:

    wallet tkn_send {token symbol} {address_from} {address to} {amount} 
    wallet tkn_send_from {token symbol} {address_from} {address to} {amount}
    wallet tkn_approve {token symbol} {address_from} {address to} {amount}
    wallet tkn_allowance {token symbol} {address_from} {address to}
    wallet tkn_mint {token symbol} {mint_to_addr} (--attach-neo={amount}, --attach-gas={amount})
    wallet tkn_register {addr} ({addr}...) (--from-addr={addr})
    

    And we just finished this small tutorial. To sum it up, I’ve made a small video going through the whole process:


    Given the popularity of the “blockchain” movement nowadays, we are starting to have several alternative networks that are able to run smart-contracts, some of them more mature than the others, but many of them very capable.

    Playing with several of the competing alternatives before jumping to the implementation phase of our solution is important, so we can understand which one will be a better fit for our particular situation.

    If you have been following this field for the last few years, you know it is moving rapidly and many breakthroughs are still happening. Nevertheless at this moment we already have solid foundations for building decentralized applications on top of the blockchain, for this purpose NEO is positioning itself as solid solution to take into account.

  • Django Friday Tips: Adding RSS feeds

    Following my previous posts about RSS and its importance for an open web, this week I will try to show how can we add syndication to our websites and other apps built with Django.

    This post will be divided in two parts. The first one covers the basics:

    • Build an RSS feed based on a given model.
    • Publish the feed.
    • Attach that RSS feed to a given webpage.

    The second part will contain more advanced concepts, that will allow subscribers of our page/feed to receive real-time updates without the need to continuously check our feed. It will cover:

    • Adding a Websub / Pubsubhubbub hub to our feed
    • Publishing the new changes/additions to the hub, so they can be sent to subscribers

    So lets go.

    Part one: Creating the Feed

    The framework already includes tools to handle this stuff, all of them well documented here. Nevertheless I will do a quick recap and leave here a base example, that can be reused for the second part of this post.

    So lets supose we have the following models:

    class Author(models.Model):
    
        name = models.CharField(max_length=150)
        created_at = models.DateTimeField(auto_now_add=True)
    
        class Meta:
            verbose_name = "Author"
            verbose_name_plural = "Authors"
    
        def __str__(self):
            return self.name
    
    
    class Article(models.Model):
    
        title = models.CharField(max_length=150)
        author = models.ForeignKey(Author, on_delete=models.CASCADE)
    
        created_at = models.DateTimeField(auto_now_add=True)
        updated_at = models.DateTimeField(auto_now=True)
    
        short_description = models.CharField(max_length=250)
        content = models.TextField()
    
        class Meta:
            verbose_name = "Article"
            verbose_name_plural = "Articles"
    
        def __str__(self):
            return self.title
    

    As you can see, this is for a simple “news” page where certain authors publish articles.

    According to the Django documentation about feeds, generating a RSS feed for that page would require adding the following Feedclass to the views.py (even tough it can be placed anywhere, this file sounds appropriate):

    from django.urls import reverse_lazy
    from django.contrib.syndication.views import Feed
    from django.utils.feedgenerator import Atom1Feed
    
    from .models import Article
    
    
    class ArticlesFeed(Feed):
        title = "All articles feed"
        link = reverse_lazy("articles-list")
        description = "Feed of the last articles published on site X."
    
        def items(self):
            return Article.objects.select_related().order_by("-created_at")[:25]
    
        def item_title(self, item):
            return item.title
    
        def item_author_name(self, item):
            return item.author.name
    
        def item_description(self, item):
            return item.short_description
    
        def item_link(self, item):
            return reverse_lazy('article-details', kwargs={"id": item.pk})
    
    
    class ArticlesAtomFeed(ArticlesFeed):
        feed_type = Atom1Feed
        subtitle = ArticlesFeed.description
    

    On the above snippet, we set some of the feed’s global properties (title, link, description), we define on the items() method which entries will be placed on the feed and finally we add the methods to retrieve the contents of each entry.

    So far so good, so what is the other class? Other than standard RSS feed, with Django we can also generate an equivalent Atom feed, since many people like to provide both that is what we do there.

    Next step is to add these feeds to our URLs, which is also straight forward:

    urlpatterns = [
        ...
        path('articles/rss', ArticlesFeed(), name="articles-rss"),
        path('articles/atom', ArticlesAtomFeed(), name="articles-atom"),
        ...
    ]
    

    At this moment, if you try to visit one of those URLs, an XML response will be returned containing the feed contents.

    So, how can the users find out that we have these feeds, that they can use to get the new contents of our website/app using their reader software?

    That is the final step of this first part. Either we provide the link to the user or we include them in the respective HTML page, using specific tags in the head element, like this:

    <link rel="alternate" type="application/rss+xml" title="{{ rss_feed_title }}" href="{% url 'articles-rss' %}" />
    <link rel="alternate" type="application/atom+xml" title="{{ atom_feed_title }}" href="{% url 'articles-atom' %}" />
    

    And that’s it, this first part is over. We currently have a feed and a mechanism for auto-discovery, things that other programs can use to fetch information about the data that was published.

    Part Two: Real-time Updates

    The feed works great, however the readers need continuously check it for new updates and this isn’t the ideal scenario. Neither for them, because if they forget to regularly check they will not be aware of the new content, neither for your server, since it will have to handle all of this extra workload.

    Fortunately there is the WebSub protocol (previously known as Pubsubhubbub), that is a “standard” that has been used to deliver a notification to subscribers when there is new content.

    It works by your server notifying an external hub (that handles the subscriptions) of the new content, the hub will then notify all of your subscribers.

    Since this is a common standard, as you might expect there are already some Django packages that might help you with this task. Today we are going to use django-push with https://pubsubhubbub.appspot.com/ as the hub, to keep things simple (but you could/should use another one).

    The first step, as always, is to install the new package:

    $ pip install django-push
    

    And then add the package’s Feed class to our views.py (and use it on our Atom feed):

    from django_push.publisher.feeds import Feed as HubFeed
    
    ...
    
    class ArticlesAtomFeed(ArticlesFeed, HubFeed):
        subtitle = ArticlesFeed.description
    

    The reason I’m only applying this change to the Atom feed, is because this package only works with this type of feed as it is explained in the documentation:

    … however its type is forced to be an Atom feed. While some hubs may be compatible with RSS and Atom feeds, the PubSubHubbub specifications encourages the use of Atom feeds.

    This no longer seems to be true for the more recent protocol specifications, however for this post I will continue only with this type of feed.

    The next step is to setup which hub we will use. On the  settings.py file lets add the following line:

    PUSH_HUB = 'https://pubsubhubbub.appspot.com'
    

    With this done, if you make a request for your Atom feed, you will notice the following root element was added to the XML response:

    <link href="https://pubsubhubbub.appspot.com" rel="hub"></link>

    Subscribers will use that information to subscribe for notifications on the hub. The last thing we need to do is to tell the hub when new entries/changes are available.

    For that purpose we can use the ping_hub function. On this example the easiest way to accomplish this task is to override the Article  model save() method on the models.py file:

    from django_push.publisher import ping_hub
    
    ...
    
    class Article(models.Model):
        ...
        def save(self, *args, **kwargs):
            super().save(*args, **kwargs)
            ping_hub(f"https://{settings.DOMAIN}{reverse_lazy('articles-atom')}")
    

    And that’s it. Our subscribers can now be notified in real-time when there is new content on our website.