Categories
Python Software Development

Why you shouldn’t remove your package from PyPI

Nowadays most software developed using the Python language relies on external packages (dependencies) to get the job done. Correctly managing this “supply-chain” ends up being very important and having a big impact on the end product.

As a developer you should be cautious about the dependencies you include on your project, as I explained in a previous post, but you are always dependent on the job done by the maintainers of those packages.

As a public package owner/maintainer, you also have to be aware that the code you write, your decisions and your actions will have an impact on the projects that depend directly or indirectly on your package.

With this small introduction we arrive to the topic of this post, which is “What to do as a maintainer when you no longer want to support a given package?” or ” How to properly rename my package?”.

In both of these situations you might think “I will start by removing the package from PyPI”, I hope the next lines will convince you that this is the worst you can do, for two reasons:

  • You will break the code or the build systems of all projects that depend on the current or past versions of your package.
  • You will free the namespace for others to use and if your package is popular enough this might become a juicy target for any malicious actor.

TLDR: your will screw your “users”.

The left-pad incident, while it didn’t happen in the python ecosystem, is a well known example of the first point and shows what happens when a popular package gets removed from the public index.

Malicious actors usually register packages using names that are similar to other popular packages with the hope that a user will end up installing them by mistake, something that already has been found multiple times on PyPI. Now imagine if that package name suddenly becomes available and is already trusted by other projects.

What should you do it then?

Just don’t delete the package.

I admit that in some rare occasions it might be required, but most of the time the best thing to do is to leave it there (specially for open-source ones).

Adding a warning to the code and informing the users in the README file that the package is no longer maintained or safe to use is also a nice thing to do.

A good example of this process being done properly was the renaming of model-mommy to model-bakery, as a user it was painless. Here’s an overview of the steps they took:

  1. A new source code repository was created with the same contents. (This step is optional)
  2. After doing the required changes a new package was uploaded to PyPI.
  3. Deprecation warnings were added to the old code, mentioning the new package.
  4. The documentation was updated mentioning the new package and making it clear the old package will no longer be maintained.
  5. A new release of the old package was created, so the user could see the deprecation warnings.
  6. All further development was done on the new package.
  7. The old code repository was archived.

So here is what is shown every time the test suite of an affected project is executed:

/lib/python3.7/site-packages/model_mommy/__init__.py:7: DeprecationWarning: Important: model_mommy is no longer maintained. Please use model_bakery instead: https://pypi.org/project/model-bakery/

In the end, even though I didn’t update right away, everything kept working and I was constantly reminded that I needed to make the change.

Categories
Security

Security.txt

Some days ago while scrolling my mastodon‘s feed (for those who don’t know it is like Tweeter but instead of being a single website, the whole network is composed by many different entities that interact with each other), I found the following message:

To server admins:

It is a good practice to provide contact details, so others can contact you in case of security vulnerabilities or questions regarding your privacy policy.

One upcoming but already widespread format is the security.txt file at https://your-server/.well-known/security.txt.

See https://securitytxt.org/ and https://infosec-handbook.eu/.well-known/security.txt.

@infosechandbook@chaos.social

It caught my attention because my personal domain didn’t had one at the time. I’ve added it to other projects in the past, but do I need one for a personal domain?

After some thought, I couldn’t find any reason why I shouldn’t add one in this particular case. So as you might already have guessed, this post is about the steps I took to add it to my domain.

What is it?

A small text file, just like robots.txt, placed in a well known location, containing details about procedures, contacts and other key information required for security professionals to properly disclose their findings.

Or in other words: Contact details in a text file.

security.txt isn’t yet an official standard (still a draft) but it addresses a common issue that security researches encounter during their day to day activity: sometimes it’s harder to report a problem than it is to find it. I always remember the case of a Portuguese citizen, who spent ~5 months trying to contact someone that could fix some serious vulnerabilities in a governmental website.

Even though it isn’t an accepted standard yet, it’s already being used in the wild:

Need more examples? A small search finds it for you very quickly or you can also read here a small analysis of the current status on Alexa’s top 1000 websites.

Implementation

So to help the cause I added one for this domain. It can be found at https://ovalerio.net/.well-known/security.txt

Below are the steps I took:

  1. Go to https://securitytxt.org/ and fill the required fields of the form present on that website.
  2. Fill the extra fields if they apply.
  3. Generate the text document.
  4. Sign the content using your PGP key
    gpg --clear-sign security.txt
  5. Publish the signed file on your domain under https://<domain>/.well-known/security.txt

As you can see, this is a very low effort task and it can generate very high returns, if it leads to a disclosure of a serious vulnerability that otherwise would have gone unreported.

Categories
Software Development Technology and Internet

CSP headers using Cloudflare Workers

Last January I made a small post about setting up a “Content-Security-Policy” header for this blog. On that post I described the steps I took to reach a final result, that I thought was good enough given the “threats” this website faces.

This process usually isn’t hard If you develop the website’s software and have an high level of control over the development decisions, the end result ends up being a simple yet very strict policy. However if you do not have that degree of control over the code (and do not want to break the functionality) the policy can end up more complex and lax than you were initially hoping for. That’s what happened in my case, since I currently use a standard installation of WordPress for the blog.

The end result was a different security policy for different routes and sections (this part was not included on the blog post), that made the web-server configuration quite messy.

(With this intro, you might have already noticed that I’m just making excuses to redo the initial and working implementation, in order to test some sort of new technology)

Given the blog is behind the Cloudflare CDN and they introduced their “serverless” product called “Workers” a while ago, I decided that I could try to manage the policy dynamically on their servers.

Browser <--> CF Edge Server <--> Web Server <--> App

The above line describes the current setup, so instead of adding the CSP header on the “App” or the “Web Server” stages, the header is now added to the response on the last stage before reaching the browser. Let me describe how I’ve done it.

Cloudflare Workers

First a very small introduction to Workers, later you can find more detailed information on Workers.dev.

So, first Cloudflare added the v8 engine to all edge servers that route the traffic of their clients, then more recently they started letting these users write small programs that can run on those servers inside the v8 sandbox.

The programs are built very similarly to how you would build a service worker (they use the same API), the main difference being where the code runs (browser vs edge server).

These “serverless” scripts can then be called directly through a specific endpoint provided by Cloudflare. In this case they should create and return a response to the requests.

Or you can instruct Cloudflare to execute them on specific routes of your website, this means that the worker can generate the response, execute any action before the request reaches your website or change the response that is returned.

This service is charged based on the number of requests handled by the “workers”.

The implementation

Going back to the original problem and based on the above description, we can dynamically introduce or modify the “Content-Security-Policy” for each request that goes through the worker which give us an high degree of flexibility.

So for my case a simple script like the one below, did the job just fine.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Forward the request and swap the response's CSP header
 * @param {Request} request
 */
async function handleRequest(request) {
  let policy = "<your-custom-policy-here>"
  let originalResponse = await fetch(request)
  response = new Response(originalResponse.body, originalResponse)
  response.headers.set('Content-Security-Policy', policy)
  return response
}

The script just listens for the request, passes it to a handler function (lines 1-3), forwards to the origin server (line 12), grabs the response (line 13), replaced the CSP header with the defined policy (line 14) and then returns the response.

If I needed something more complex, like making slight changes to the policy depending on the User-Agent to make sure different browsers behave as expected given the different implementations or compatibility issues, it would also be easy. This is something that would be harder to achieve in the config file of a regular web server (nginx, apache, etc).

Enabling the worker

Now that the script is done and the worker deployed, in order to make it run on certain requests to my blog, I just had to go to the Cloudflare’s dashboard of my domain, click on the “workers” section and add the routes I want it to be executed:

cloudflare workers routes modal
Configuring the routes that will use the worker

The settings displayed on the above picture will run the worker on all requests to this blog, but is can be made more specific and I can even have multiple workers for different routes.

Some sort of conclusion

Despite the use-case described in this post being very simple, there is potential in this new “serverless” offering from Cloudflare. It definitely helped me solve the problem of having different policies for different sections of the website without much trouble.

In the future I might comeback to it, to explore other user-cases or implementation details.

Categories
Technology and Internet

Setting up a Content-Security-Policy

A couple of weeks ago, I gave a small talk on the Madeira Tech Meetup about a set of HTTP headers that could help website owners protect their assets and their users. The slides are available here, just in case you want to take a look.

The content of the talk is basically a small review about what exists, what each header tries to achieve and how could you use it.

After the talk I remembered that I didn’t review the heades of this blog for quite sometime. So a quick visit to Mozilla Observatory, a tool that lets you have a quick look of some of the security configurations of your website, gave me an idea of what I needed to improve. This was the result:

The Content-Security-Header was missing

So what is a Content Security Policy? On the MDN documentation we can find the following description:

The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page.

Mozilla Developer Network

Summing up, in this header we describe with a certain level of detail the sources from where each type of content can be fetched in order to be allowed and included on a given page/app. The main goal of this type of policy is to mitigate Cross-Site Scripting attacks.

In order to start building a CSP for this blog a good approach, in my humble opinion, is to start with the more basic and restrictive policy and then proceed evaluating the need for exceptions and only add them when strictly necessary. So here is my first attempt:

default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

Lets interpret what it says:

  • default-src: This is the default value for all non-mentioned directives. self means “only things that come from this domain”.
  • object-src: No <object>, <embed> or <applet> here.
  • report-uri: All policy violations should be reported by the browser to this URL.

The idea was that all styles, scripts and images should be served by this domain, anything external should be blocked. This will also block inline scripts, styles and data images, which are considered unsafe. If for some reason I need to allow this on the blog I could use unsafe-inline, eval and data: on the directive’s definition but in my opinion they should be avoided.

Now a good way to find out how this policy will affect the website and to understand how it needs to be tuned (or the website changed) we can activate it using the “report only mode:

Content-Security-Policy-Report-Only: <policy>

This mode will generate some reports when you (and other users) navigate through the website, they will be printed on the browser’s console and sent to the defined report-uri, but the resources will be loaded anyway.

Here are some results:

CSP violations logs on the browser console
Example of the CSP violations on the browser console

As an example below is a raw report from one of those violations:

{
    "csp-report": {
        "blocked-uri": "inline",
        "document-uri": "https://blog.ovalerio.net/",
        "original-policy": "default-src 'self'; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly",
        "violated-directive": "default-src"
    }
}

After a while I found that:

  • The theme used on this blog used some data: fonts
  • Several inline scripts were being loaded
  • Many inline styles were also being used
  • I have some demos that load content from asciinema.org
  • I often share some videos from Youtube, so I need to allow iframes from that domain
  • Some older posts also embeded from other websites (such as soundcloud)

So for the blog to work fine with the CSP being enforced, I either had to include some exceptions or fix errors. After evaluating the attack surface and the work required to make the changes I ended up with the following policy:

Content-Security-Policy-Report-Only: default-src 'self'; script-src 'self' https://asciinema.org 'sha256-A+5+D7+YGeNGrYcTyNB4LNGYdWr35XshEdH/tqROujM=' 'sha256-2N2eS+4Cy0nFISF8T0QGez36fUJfaY+o6QBWxTUYiHc=' 'sha256-AJyUt7CSSRW+BeuiusXDXezlE1Wv2tkQgT5pCnpoL+w=' 'sha256-n3qH1zzzTNXXbWAKXOMmrBzjKgIQZ7G7UFh/pIixNEQ='; style-src 'self' 'sha256-MyyabzyHEWp8TS5S1nthEJ4uLnqD1s3X+OTsB8jcaas=' 'sha256-OyKg6OHgnmapAcgq002yGA58wB21FOR7EcTwPWSs54E='; font-src 'self' data:; img-src 'self' https://secure.gravatar.com; frame-src 'self' https://www.youtube.com https://asciinema.org; object-src 'none'; report-uri https://ovalerio.report-uri.com/r/d/csp/reportOnly

A lot more complex than I initially expected it to be, but it’s one of the drawbacks of using a “pre-built” theme on a platform that I didn’t develop. I was able (in the available time) to fix some stuff but fixing everything would take a lot more work.

All those sha-256 hashes were added to only allow certain inline scripts and styles without allowing everything using unsafe-inline.

Perhaps in the future I will be able to change to a saner theme/platform, but for the time being this Content-Security-Policy will do the job.

I started enforcing it (by changing Content-Security-Policy-Report-Only to Content-Security-Policy) just before publishing this blog post, so if anything is broken please let me know.

I hope this post has been helpful to you and if you didn’t yet implement this header you should give it a try, it might take some effort (depending on the use case) but in the long run I believe it is totally worth it.

Categories
Random Bits

Staying on an AirBnB? Look for the cameras

When going on a trip it is now common practice to consider staying on an rented apartment or house instead of an hotel or hostel, mostly thanks to AirBnB which made it really easy and convenient for both side of the deal. Most of the time the price is super competitive and I would say a great fit for many situations.

However as it happens with almost anything, it has its own set of problems and challenges. One example of these new challenges are the reports (and confirmations) that some, lets call them malicious hosts, have been putting in place cameras to monitor the guests during their stay.

With a small search on the internet you can find

Someone equipped with the right knowledge and a computer can try to check if a camera is connected to the WiFi network, like this person did:

Toot describing that a camera that was hidden inside a box

If this is your case, the following post provides a few guidelines to look for the cameras:

Finally, try to figure out the public IP address of the network you are on ( https://dshield.org/api/myip ) and either run a port scan from the outside to see if you find any odd open ports, or look it up in Shodan to see if Shodan found cameras on this IP in the past (but you likely will have a dynamic IP address).

InfoSec Handlers Diary Blog

This page even provides a script that you can execute to automatically do most steps explained on the above article.

However, sometimes you don’t bring your computer with you, which means you would have to rely on your smartphone to do this search. I’m still trying to find a good, trustworthy and intuitive app to recommend, since using nmap on Android will not help the less tech-savvy people.

Meanwhile, I hope the above links provide you with some ideas and useful tools to look for hidden cameras while you stay on a rented place.

Categories
Python

Looking for security issues on your python projects

In today’s post I will introduce a few open-source tools, that can help you improve the security of any of your python projects and detect possible vulnerabilities early on.

These tools are quite well known in the python community and used together will provide you with great feedback about common issues and pitfalls.

Safety and Piprot

As I discussed some time ago on a post about managing dependencies and the importance of checking them for known issues, in python there is a tool that compares the items of your requirements.txt with a database of known vulnerable versions. It is called safety (repository)  and can be used like this:

safety check --full-report -r requirements.txt

If you already use pipenv safety is already incorporated and can be used by running: pipenv check (more info here).

Since the older the dependencies are, the higher the probability of a certain package containing bugs and issues, another great tool that can help you with this is piprot (repository).

It will check all items on your requirements.txt and tell you how outdated they are.

Bandit

The next tool in the line is bandit, which is a static analyzer for python built by the Open Stack Security Project, it checks your codebase for common security problems and programming  mistakes that might compromise your application.

It will find cases of hardcoded passwords, bad SSL defaults, usage of eval, weak ciphers, different “injection” possibilities, etc.

It doesn’t require much configuration and you can easily add it to your project. You can find more on the official repository.

Python Taint

This last one only applies if you are building a web application and requires a little bit more effort to integrate in your project (at its current state).

Python Taint (pyt) is a static analyzer that tries to find spots were your code might be vulnerable to common types of problems that affect websites and web apps, such as SQL injection, cross site scripting (XSS), etc.

The repository can be found here.

If you are using Django, after using pyt you might also want to run the built in manage.py check command, (as discussed in a previous post) to verify some specific configurations of the framework present on your project.

 

Categories
Blockchain

Upgrade your “neo-python” wallets

Several weeks ago I started to explore the NEO ecosystem, for those who are not aware NEO is a blockchain project that just like Ethereum pretends to create the tools and the platform to execute smart-contracts and create new types of decentralized applications. It has its pros and cons just like any other system, but that is outside of the scope of this blog post.

One of the defining characteristics of this “cryptocurrency” is the ability develop those smart-contracts in programming languages the user already is familiar with (however only a small subset of the language is available).

So I searched for the available SDKs and found the neo-python project, which is a wallet software and also a set of tools to develop using the Python programming language. The project is developed by a community of supporters of the NEO ecosystem called City of Zion.

And now the real topic of the post begins, while learning the features and exploring the codebase I found an urgent security issue with the way the wallets were being encrypted by neo-python.

Long story short, the method used to protect the wallets wasn’t correctly implemented and allowed an attacker with access to the wallet file to decrypt it without the need for the password/pass-phrase (more details here) .

Fortunately it is an actively developed project and the team responsible for it was quick to acknowledge the problem and merge the fix I proposed in a pull request. The fix is now present in the newer versions of the project, and it now forces the users to reset the security features of their wallets (check this video for more details, starting on minute 8 up to 10) .

Now in this post I would like to leave my recommendation about how to proceed after re-encrypting the wallet, because even though the issue is fixed your private keys might have been compromised before you applied the patch. If you are a user and didn’t noticed nothing yet the most probable scenario is that you weren’t compromised, since most immediate thing an attacker could/would do is to steal your funds.

Nevertheless, there is always the possibility and to avoid any bad surprises you definitely should:

  1. Properly encrypt your wallet using the reencrypt_wallet.py  script.
  2. Check the new generated wallet is working properly.
  3. Then delete the old wallet.
  4. Create a new wallet.
  5. Transfer your funds to the new wallet.

The steps 4 and 5 are necessary because the fix protects your master key but it doesn’t change it and as I previously said if a copy of your vulnerable wallet exists (created by you or by an attacker) your funds are still accessible. So don’t forget to go through them.

Other than this, the project is very interesting and while still immature it has been fun the work with it, so I will keep contributing some improvements in the near future.

 

Categories
Random Bits Technology and Internet

Managing Secrets With Vault

I’ve been looking into this area, of how to handle and manage a large quantity of secrets and users, for quite a while (old post), because when an organization or infrastructure grow, the number of “secrets” required for authentication and authorization increase as well. Is at this stage that bad practices (that are no more than shortcuts) as reusing credentials, storing them in less appropriate ways or no longer invalidating those who are no longer in required, start becoming problematic.

Yesterday at “Madeira Tech Meetup” I gave a brief introduction to this issue and explored ways to overcome it, which included a quick and basic explanation of Vault and demo about a common use case.

You can find the slides of the presentation here and if you have any suggestion or something you would like to discuss about it, feel free to comment or reach through any of the contact mediums I provided.

Categories
Software Development

Keep your dependencies under check

Nowadays most software projects with a “decent size” rely on many software dependencies, or in other words: libraries and tools, developed by other people. That usually are under constant change.

The reasons for these are clear and can go from implementing common patterns and avoid repeating ourselves, to accelerate the development, to use mature implementations and avoid some pitfalls, etc. Sometimes many projects rely on way too many dependencies for simples things (Remember the left-pad fiasco?).

Once these dependencies are loaded, integrated and working as expected, people often forget they are there and many times they stay untouched for long periods of time. Even when newer versions are released, unless something starts breaking, nobody remembers to keep them up to date, a situation that might lead to security vulnerabilities, not in your code but on the code your project depends on.

Of course I’m not telling you anything new, what I pretend to achieve with this post, is to show that there are many tools available to help you fight this problem. When you integrate them on your CI or on another step of your development process, they will keep you informed about what dependencies have known security vulnerabilities and what you should upgrade as soon as possible.

The majority of the programming languages have this sort of tools, so a little search should help you find the one that better suits you stack. Below are some examples:

As an example here is what I needed to do in order to check the dependencies of Hawkpost (an open-source project that I’m deeply involved with at the moment):

$ safety check --full-report -r requirements/requirements.txt
safety report
---
No known security vulnerabilities found

For most of these tools the basic check is this simple to do and in the long run it might save you from some headaches.

Update (26-06-2018): Added cargo-audit to the list

Categories
Python Technology and Internet

Receive PGP encrypted emails, without the sender needing to know how to do it

One common trouble of people trying to secure their email communications with PGP, is that more often that not the other end doesn’t know how to use these kind of tools. I’ll be honest, at the current state the learning curve is too steep for the common user. This causes a huge deal of trouble when you desire to receive/sent sensitive information in a secure manner.

I will give you an example, a software development team helping a customer building his web business or application, may want to receive a wide variety of access keys to external services and APIs, that are in possession of the customer and are required (or useful) to be integrated in the project.

Lets assume that the customer is not familiarized with encryption tools, the probability of that sensitive material to be shared in an insecure way is too high, he might send it through a clear text email or post it on some shared document (or file). Both the previous situations are red flags, either by the communication channel not secure enough or the possibility of existing multiple copies of the information in different places with doubtful security, all of them in clear text.

In our recent “Whitesmith Hackathon”, one of the projects tried to address this issue. We though on a more direct approach to this situation based on the assumption that you will not be able to convince the customer into learning this kind of things. We called it Hawkpost, essentially it’s a website that makes use of OpenPGP.js, where you create unique links containing a form, that the user uses to submit any information, that will then be encrypted on his browser with your public key (without the need to install any extra software) and forwarded to your email address.

You can test and used it on https://hawkpost.co, but the project is open-source, so you can change it and deploy it on your own server if you prefer. It’s still in a green state at the moment, but we will continue improving the concept according with the received feedback. Check it out and tell us what you think.