Tag: Django

  • Django Friday Tips: Secret Key

    One thing that is always generated for you when you start a new django project is the SECRET_KEY string. This value is described in the documentation as:

    A secret key for a particular Django installation. This is used to provide cryptographic signing, and should be set to a unique, unpredictable value.

    The rule book mandates that this value should not be shared or made public, since this will defeat its purpose and many securing features used by the framework. Given that on any modern web development process we have multiple environments such as production and staging, or in the cases where we might deploy the same codebase different times for different purposes, we will need to generate and have distinct versions of this variable so we can’t rely solely on the one that was generated when the project was started.

    There is no official way to generate new values for the secret key, but with a basic search on the Internet, you can find several sources and code snippets for this task. So which one to use? The django implementation has a length of 50 characters, chosen randomly from an alphabet with size 50 as well, so we might start with this as a requirement. Better yet, why not call the same function that django-admin.py uses itself?

    So for a new project, the first thing to do is to replace this:

    SECRET_KEY = "uN-pR3d_IcT4~ble!_Str1Ng..."

    With this:

    SECRET_KEY = os.environ.get("SECRET_KEY", None)

    Then for each deployment we can generate a distinct value for it using a simple script like this one:

    from django.utils.crypto import get_random_string
    
    chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
    print("SECRET_KEY={}".format(get_random_string(50, chars)))

    Usage:

    $ python script_name.py >> .env

    Some people think the default function is not random enough and proposed a different alternative (that also works), if you feel the same way check this script.

  • Browsing folders of markdown files

    If you are like me, you have a bunch of notes and documents written in markdown spread across many folders. Even the documentation of some projects involving many people is done this way and stored, for example, in a git repository. While it is easy to open the text editor to read these files, it is not the most pleasant experience, since the markup language was made to later generate readable documents in other formats (eg. HTML).

    For many purposes setting up the required configuration of tools to generate documentation (like mkdocs) is not practical, neither it was the initial intent when it was written. So last weekend I took a couple of hours and built a rough (and dirty) tool to help me navigate and read the markdown documents with a more pleasant experience, using the browser (applying style as github).

    I called it mdvis and it is available for download through “pip”. Here’s how working with it looks like:

    It does not provide many features and is somewhat “green”, but it serves my current purposes. The program is open-source so you can check it here, in case you want to help improving it.

  • Django Friday Tips: Security Checklist

    Security is one of those areas where it is very hard to know if everything is taken care of. So you have been working on this project for a while and you want to deploy it into a production server, there are several settings on this new environment that should differ from your development one.

    Since this is very common situation and there are many examples of misconfigurations that later turned to security issues, django has a security checklist (since version 1.8) to remind you of some basic aspects (mostly on/off switches) that you should make sure that are set correctly.

    To run it on your project you simply have to execute the following command:

    $python manage.py check --deploy

    After the verification you will be presented with warnings like this one:

    (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE_CLASSES, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.

    More information can be found in the documentation, since it uses the check framework, that has several interesting use cases.

    Interested in more information about security in django? Check this video from the last edition of “Django Under the Hood“.

  • Django Friday Tips: Managing Dependencies

    This one is not specific of django but it is very common during the development of any python project. Managing the contents of the requirements.txt file, that sometimes grows uncontrollably can be a mess. One of the root causes is the common work-flow of using virtualenv, install with pip all the required libraries and then do something like:

    $pip freeze > requirements.txt

    At the beginning this might work great, however soon you will need to change things and remove libraries. At this point, things start to get a little trickier, since you do not know which lines are a direct dependency of your project or if they were installed because a library you already removed needed them. This leads to some tedious work in order to maintain the dependency list clean.

    To solve this problem we might use pip-tools, which will help you declare the dependencies in a simple way and automatically generate the final requirements.txt. As it is shown in the project readme, we can declare the following requirements.in file:

    django
    requests
    pillow
    celery

    Then we generate our “official” requirements.txt with the pip-compile command, that will product the following output:

    #
    # This file is autogenerated by pip-compile
    # Make changes in requirements.in, then run this to update:
    #
    #    pip-compile requirements.in
    #
    amqp==1.4.8               # via kombu
    anyjson==0.3.3            # via kombu
    billiard==3.3.0.22        # via celery
    celery==3.1.19
    django==1.9
    kombu==3.0.30             # via celery
    pillow==3.0.0
    pytz==2015.7              # via celery
    requests==2.8.1
    

    Now you can keep track of where all those libraries came from. Need to add or remove packages? Just run pip-compile again.

  • Django friday tips: Switch the user model

    In the most recent versions of django, you’re no longer attached to the default user model. So unlike what happened some time ago, when you had two models (User and Profile) “linked” together through an one-to-one relationship, nowadays you can extend or substitute the base user model.

    It is as simples as adding the following line to your settings.py:

    AUTH_USER_MODEL = 'djangoapp.UserModel'

    If you only want to extend it and avoid having to implement some boilerplate, you should sub class the AbstractUserModel like this:

    from django.contrib.auth.models import AbstractUser
    
    
    class User(AbstractUser):
        ...
    

    This way you will be able to use all th predefined features, and even use the admin settings of the default user model by doing the following in your admin.py file:

    from django.contrib.auth.admin import UserAdmin as DefaultUserAdmin
    
    @admin.register(User)
    class UserAdmin(DefaultUserAdmin):
        ...
    

    If you are starting out I hope this information was useful.

  • Newsletters for Python web developers

    The amount of new information that is added each day to the web is overwhelming, trying to keep up daily with everything about a given topic can be a time consuming process. One good way I found to tackle this problem and to avoid wasting a good chunk of my day searching and filtering through lots of new content in order to know what’s going on, was to subscribe to good resources that curate this material and send to my email box at the end of each week/month.

    Over time I found that the following 4 sources have continuously provided me with selection of good and up to date content summing up what I might have missed in the previous week/month related to Python and web development in general.

    Pycoders weekly

    This weekly newsletter is not focused on the web but address what’s going on on the python community, suggests good articles so you can level up your python skills and showcases interesting projects or libraries.

    Url: http://pycoders.com/

    Django Round-Up

    This one is comes less frequently but I found the quality of the content to be high. As its name shows, Django round-up focus exclusively on contents related to the web framework.

    Url: https://lincolnloop.com/django-round-up/

    HTML5 Weekley

    The first two were about the server side, with this one we move to the browser. HTML5 Weekly focuses on what can be done in the browser and were these technologies are heading to.

    Url: http://html5weekly.com/

    Javascript Weekly

    Being a web development post we can’t leave JavaScript behind, at least for now. This newsletter gives you the latest news and tools related to this programming language.

    Url: http://javascriptweekly.com/

    I hope you like it. If you find them useful you might also want to follow my Django Collection bundle (which I described in this old post), where I collect useful material related with the Django web framework.

  • Moving to Python 3

    A week ago the support period for the last version of Python 2 was extended by 5 years (from 2015 to 2020) and this event just ignited once again the discussion about the fragmentation in the python ecosystem. Some have the opinion that version 2 should have a 2.8 release while others keep saying that the future is python 3 and this event will delay even more the adoption of the new version.

    The truth is, version 3 already has almost 5 and half years (released in December 2008) and it seem it didn’t have yet conquered enough number of users to dethrone the old version. While the first iterations of the new major version met many critics (3.0 until 3.2),  the last 2 releases seems to have conquered very good reviews and after many years the majority of the most important libraries and modules seems to have support for python 3 (can be checked here ).

    This way and after some thought, i decided that it is time (maybe a little late) to change my default development version for new projects to python3, since it really is the future of this programming language and it is time to move on. There will be exceptions of course, like old projects that need to be maintained,  new ones where the requirements do not allow the new version or where the needed packages do not yet support python 3.

    So lets check what this “new” version has to offer.

  • Django Resources

    As I said in earlier posts in this blog, when i build websites or webapps where there are no technology impositions, i usually choose to do it in Python and in most of the cases, that’s the equivalent to say i choose to do it in Django.

    Over the last year, since i started using Bundlr,  I’ve been aggregating some resources like blog entries, tutorials and videos that i found useful and that could become handy in the future.

    Today I’m sharing the collection here, since it might helpful to someone else. I hope you like it and if you know more references that should be included in the list, please share it in the comments or send me an email.

    The list can be found here.

    Edit July 2016: Since I removed my account, the list is not longer available on Bundlr. Check recent posts, it will be published again soon.

  • First experience with MOOC

    The year of 2012 for the Internet was definitely the year of the “massive open online courses” with some startups of online education stepping up to the big stage (Coursera, Udacity, etc) and some well know names coming up with their own initiatives (MIT, Harvard and Berkeley at Edx). So in the beginning of this year there were many opportunities to learn something new or update your knowledge with college level quality, where the only prerequisite is your motivation.

    So i decide to give it a try, in January I picked up a topic that i wanted to learn/improve and signed up for it. The course wasn’t taken in any of that major sites that i previously mentioned but the system was based on Edx. At the end of the month, i started the 10gen‘s 7 week course on “MongoDB for Developers” (Given in Python) and followed the weekly classes flawlessly till the final exam in the middle of March.

    In the next few paragraphs i will describe my experience based on some notes that i took during that period, basically, i will tell what i liked and what i think that should be improved.

    On the first week in a course for developers, the rythm was kinda slow with the instructors wasting too much time with the basics of python and how to install some libraries. At first i thought everyone would think the same, but in the discussions i noticed that many of the fellow students didn’t even knew how to install python on their machines. Even though it was a nice thing to do, in my opinion for this kind of course previous python experience should be a prerequisite.

    In the next few weeks things started to get interesting when we focused more on mongodb and talked about its operations, design, performance, the aggregation framework, etc. Every week a new batch of 3 to 10 minute videos (with few exceptions), covering each one a new concept or use case about the week’s topic was released, plus some questions to make sure you understood the what was being explained in each video. Personally i like this approach, i didn’t move to the next video until i completely understood the previous one, and if i had doubts it was as simple as watch the video again and use the discussions in the case the doubts persists. The responses to your questions were posted generally pretty fast, many times by the instructor but most of the times by fellow students.

    To complete the week you had to complete some kind of homework that weighed 50% of your final grade. Some people complained that it was relatively easy to complete these tasks, but in my opinion the purpose of this homework is to certify that you, at the end of each week, understood the key concepts lectured and not to test the capacity and expertise of the participants.

    In the last week of the course, you only had to complete the exam, the content posted by the instructor were optional and consisted in interviews with professionals talking about mongodb implementations in production right now on codecademy and foursquare.

    One improvement that i would like to see in future courses is a discussion box per video where you didn’t have to leave the video page to ask questions or to answer the ones you know.

    In conclusion, i really liked the experience and i will certainly put my new “mongodb” skills in action on a future project. Right now I’m already aiming to a new course for the summer (when my weekly schedule is lighter). If you already took one of these online courses, I would like to listen what you have to say about them. Feel free to use the comments.

  • Recovering your bookmarks

    Some time ago, while cleaning stuff in my computer, I decided to switch my browser to Opera and delete the version of Firefox that I was using at the time. While doing that and removing all the Firefox folders that are left behind, I accidentally erased all my bookmarks and I didn’t had them synced with some on-line service. Well that wasn’t good, I had references stored there that I wanted to keep.

    When trying to recover the file ‘places.sqlite’ I found an bookmark backup generated by Firefox. When I opened the file I found that it was a mess, basically it was bunch of big JSON objects stored in one line containing lots of garbage (I only needed the urls).

    I kept that file until today, when I finally decided that I would put those bookmarks again in my browser. As Opera doesn’t import this kind of files, I made a little python script that extracts the names and urls of the backup and generates a single file that opera can import, while keeping the folder structure.

    Well, it worked, so I tought it might be usefull to someone else and pushed it to github. If any of you ever have the same problem give it a shoot and use this “quick fix”. You can find it here with some instructions on how to use it. If you find any problem, use the comments and github issues.

  • Generators, Decorators and Metaclasses

    For some time now, I’ve been trying to improve my python skills and learn a little bit more deeply the how the language works. The objective of this quest is to write more efficient and structured code, because it seems to me that I’m not using the full potential of this programing language.

    Yesterday i found at stackoverflow 3 comments from the same person answering 3 different questions, one about the yield statement in python, other about decorators and another explaining metaclasses. The posts are long but the explanation very good and with several examples, I thought that they were so good that I must share them with those who are trying to learn more advanced python.So here they are, in chronological order:

  • Simple JSON Parser for Python

    Some time ago i started to follow a Blog that weekly proposes some programming exercices and i solved one of their problems (an old one, from 2009) . So today i’m posting here my solution for the problem of this week. Basically they ask us to write a JSON parser in our favorite computer language, so i chose “Python” and tried to complete the task.

    For those who don’t know what JSON is:

    JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate.

    My implementation is quite simple and it can contain some bugs (and is not optimized), so if you discover any error just leave a reply (we are always learning). Bellow is my code and a link to Github where you can also comment the code. In the following weeks i’ll try to solve more of their problems.

    class json_parser:
    
        def __init__(self, string):
            self.json_data = self.__remove_blanks(string)
            self.pointer = 0
    
        def __remove_blanks(self, string):
            new_list = []
            inside_string = False
            for i in list(string):
                if inside_string or i != ' ':
                    new_list.append(i)
                if i == '"':
                    inside_string = not inside_string
    
            return "".join(n for n in new_list)
    
        def __parse_obj(self):
            new_dic = {}
            self.pointer += 1
            while self.json_data[self.pointer] != '}':
                if self.json_data[self.pointer] == '"':
                    key = self.__parse_string()
                else:
                    raise Exception  # The only possible type of value for a key is String
    
                if self.json_data[self.pointer] == ':':
                    self.pointer += 1
                else:
                    raise Exception  # invalid object
    
                value = self.__parse_value()
                if value == -1:
                    return -1
    
                new_dic[key] = value
                if self.json_data[self.pointer] == ',':
                    self.pointer += 1
    
            self.pointer += 1
            return new_dic
    
        def __parse_array(self):
            new_array = []
            self.pointer += 1
            while self.json_data[self.pointer] != ']':
                value = self.__parse_value()
                if value == -1:
                    return -1
                else:
                    new_array.append(value)
    
                if self.json_data[self.pointer] == ',':
                    self.pointer += 1
            self.pointer += 1
            return new_array
    
        def __parse_string(self):
            self.pointer += 1
            start = self.pointer
            while self.json_data[self.pointer] != '"':
                self.pointer += 1
                if self.pointer == len(self.json_data):
                    raise Exception  # the string isn't closed
            self.pointer += 1
            return self.json_data[start:self.pointer - 1]
    
        def __parse_other(self):
            if self.json_data[self.pointer:self.pointer + 4] == 'true':
                self.pointer += 4
                return True
    
            if self.json_data[self.pointer:self.pointer + 4] == 'null':
                self.pointer += 4
                return None
    
            if self.json_data[self.pointer:self.pointer + 5] == 'false':
                self.pointer += 5
                return False
    
            start = self.pointer
            while (self.json_data[self.pointer].isdigit()) or (self.json_data[self.pointer] in (['-', '.', 'e', 'E'])):
                self.pointer += 1
    
            if '.' in self.json_data[start:self.pointer]:
                return float(self.json_data[start:self.pointer])
            else:
                return int(self.json_data[start:self.pointer])
    
        def __parse_value(self):
            try:
                if self.json_data[self.pointer] == '{':
                    new_value = self.__parse_obj()
                elif self.json_data[self.pointer] == '[':
                    new_value = self.__parse_array()
                elif self.json_data[self.pointer] == '"':
                    new_value = self.__parse_string()
                else:
                    new_value = self.__parse_other()
            except Exception:
                    print 'Error:: Invalid Data Format, unknown character at position', self.pointer
                    return -1
            return new_value
    
        def parse(self):
            if self.json_data[self.pointer] == '{' or self.json_data[self.pointer] == '[':
                final_object = self.__parse_value()
            else:
                print 'Error:: Invalid inicial Data Format'
                final_object = None
    
            return final_object

    [EDIT: The previous code has several issues, so please do not use it. Python has many great packages to handle JSON documents the right way, like simplejson.]

  • E-mails: hours to seconds

    I remember those old times when i didn’t know nothing about computer programming and i was a member of the organization of the National Meeting of students of physiotherapy and after a conference about physiotherapy and spine related problems. At that time when we had to send some hundreds or even thousands of emails for other students and professional on that field, we spent hours or even days.

    One of the problems was that our email provider (Gmail) didn’t allow to send a single email for more than ‘x’ people or send more than ‘y’ emails a day,if you had exceeded the limit they would block your account for the next 24 hours. For that problem the solution is don’t use Gmail for what it isn’t intended for. There are plenty other services to do that.

    The other problem was the use of large files, full of emails adresses that we had to split through many emails. For this one, here is a possible solution, a script that connects to a SMTP server and sends the message to every contact in the “.txt” file.

    import smtplib
    from email.MIMEText import MIMEText
    
    def send_to_all(server,login,passwd,subject,msg, cfile):
    	contacts=open(cfile, 'r')
    	email=MIMEText(msg)
    	email['Subject']=subject
    	email['From']=login
    	
    	smtp=smtplib.SMTP(server, 465) #change for 587 if doesnt work
    	smtp.ehlo()
    	smtp.starttls() #comment this line if the server doesnt support TLS
    	smtp.login(login,passwd)
    	
    	for item in contacts:
    		smtp.sendmail(login,item,email.as_string())
    		
    	smtp.close()
    
    def main():
    	#Asks the info to the user
    	print "Server: "
    	server=raw_input()
    	print "Login: "
    	login=raw_input()
    	print "Password: "
    	passwd=raw_input()
    	print "Subject: "
    	subject=raw_input()
    	print "Message: "
    	msg=raw_input()
    	print "Contacts file: "
    	cfile=raw_input()
    	#sends the message to all contacts in the file
    	send_to_all(server, login, passwd, subject, msg, cfile)
    	return 0
    
    if __name__ == '__main__':
    	main()
    

    With this script you can only send simple text messages, but it might be usefull for someone anyway.