Nowadays most software developed using the Python language relies on external packages (dependencies) to get the job done. Correctly managing this “supply-chain” ends up being very important and having a big impact on the end product.
As a developer you should be cautious about the dependencies you include on your project, as I explained in a previous post, but you are always dependent on the job done by the maintainers of those packages.
As a public package owner/maintainer, you also have to be aware that the code you write, your decisions and your actions will have an impact on the projects that depend directly or indirectly on your package.
With this small introduction we arrive to the topic of this post, which is “What to do as a maintainer when you no longer want to support a given package?” or ” How to properly rename my package?”.
In both of these situations you might think “I will start by removing the package from PyPI”, I hope the next lines will convince you that this is the worst you can do, for two reasons:
- You will break the code or the build systems of all projects that depend on the current or past versions of your package.
- You will free the namespace for others to use and if your package is popular enough this might become a juicy target for any malicious actor.
TLDR: your will screw your “users”.
The left-pad incident, while it didn’t happen in the python ecosystem, is a well known example of the first point and shows what happens when a popular package gets removed from the public index.
Malicious actors usually register packages using names that are similar to other popular packages with the hope that a user will end up installing them by mistake, something that already has been found multiple times on PyPI. Now imagine if that package name suddenly becomes available and is already trusted by other projects.
What should you do it then?
Just don’t delete the package.
I admit that in some rare occasions it might be required, but most of the time the best thing to do is to leave it there (specially for open-source ones).
Adding a warning to the code and informing the users in the README file that the package is no longer maintained or safe to use is also a nice thing to do.
A good example of this process being done properly was the renaming of model-mommy to model-bakery, as a user it was painless. Here’s an overview of the steps they took:
- A new source code repository was created with the same contents. (This step is optional)
- After doing the required changes a new package was uploaded to PyPI.
- Deprecation warnings were added to the old code, mentioning the new package.
- The documentation was updated mentioning the new package and making it clear the old package will no longer be maintained.
- A new release of the old package was created, so the user could see the deprecation warnings.
- All further development was done on the new package.
- The old code repository was archived.
So here is what is shown every time the test suite of an affected project is executed:
/lib/python3.7/site-packages/model_mommy/__init__.py:7: DeprecationWarning: Important: model_mommy is no longer maintained. Please use model_bakery instead: https://pypi.org/project/model-bakery/In the end, even though I didn’t update right away, everything kept working and I was constantly reminded that I needed to make the change.
9 replies on “Why you shouldn’t remove your package from PyPI”
My problem was on npm and was because packages were not deleted. In particular it was with cryptojs or Crypto-js (and many case sensitive variations) – all turned out to be broken in some way. Some had additional suffixes and I eventually tracked down one that responded to bug reports. Since the Python version worked I was able to compare and port the changes to that particular JS package. There is now a way to make packages as deprecated for npm but delete no longer works.
The problem I see is really that broken packages can occupy the namespace and never get updated, security-fixed or removed. I think this applies to any package system and single level names make it worse.
The alternative to reuse after deletion is to lock that package name as unusable, but only if the package was detected to have security vulnerabilities or was malware. Although given the number of dependencies in typical programs these days and the regularity of security vulnerability detection this will likely cause chaos.
This is simply an important variant on the “Once it’s on the internet, LEAVE IT IN PLACE” meme. Right down to mimicking the “This page is out of date please look ☞ over there ☞” thing.
Good to think about!
I’m not a package maintainer, but this raises an important problem that I’ve seen using packages.
If I maintain package “X”, and it has a hard dependency on packages “a”, “b”, and “c” – what do I do when one or more packages disappear and/or fail to be maintained?
Case #1:
Both I, and the users of my package are just plain effed. As a consequence, my package is as useful as teats on a boar hog and anyone who depends on it now comes after ME because my package is “broken”.
Case #2 is essentially a restatement of case #1 except things fail for different reasons and the users go after the other maintainer with guns, knives and pitchforks.
How do we solve this?
AFAIK, the only way to guarantee this is to either hope someone else takes over the project, or take it over myself – which I may not have the time or skill to do – especially if it’s a security-sensitive package.
So what’s a poor user/maintainer to do?
Indeed, an unpleasant situation for the user/maintainer to be in. It can be seen as a variant of the first point I mentioned.
You can also “yank” a release on PyPi. This will remove the version of the package from a generic install of the package but allow those who pip install a package with the version number to install it. Pip will warn you that a package has been yanked.
Good point, it is also an alternative solution (as long as the package doesn’t get deleted).
Why you shouldn’t remove your package from PyPI
View Reddit by speckz – View Source
ICYMI (In case you missed it) – Yesterday’s Python on Microcontrollers Newsletter from AdafruitDaily.com went out.If you missed it, subscribe now! – You’ll get one terrific newsletter each Tuesday (before this post). Over 7,990 subscribers worldwide!The next newsletter goes out in a week and being subscribed the best way to keep up with all things Python for hardware. No spam, no selling lists, leave any time. Hi everyone! It’s the latest Python for Microcontrollers newsletter, brought you by the community! We’re on Discord, Twitter, and for past newsletters – view them all here. If you’re reading this on the web, subscribe here. Let’s get started!Black Lives Matter Microcontroller Board Coming to CircuitPythonLadyada is making final tests on the Adafruit Black Lives Matter Board. Shown on Ask an Engineer & Top Secret, this board shows up as a USB flash drive when connected to a computer to hold files, pictures, and CircuitPython code. It has 4 edge-lit LEDs and 4 capacitive touchpads and the iconic symbol on the front. Common alkaline batteries power this board and it has a lanyard ring for wearing – Top Secret.Feather takes flight with single board computersSingle Board Computers – GHI Electronics… GHI has expanded their STM32H7 series to a Feather, ‘bit, breakout, and TFT-screen sporting Portal. We recently added STM32x7 support to CircuitPython, it would be great to get these boards added! Adafruit Blog.RedMonk’s Language Popularity SurveyYet another programming survey (YAPS): RedMonk’s language popularity rankings – Programming language popularity: Python overtakes Java as Rust reaches the top 20. For the first time since 2012, Java is not in one of the two top spots – ZDNet.A FeatherWing to Raspberry Pi Adapter BoardAn adapter board for attaching a FeatherWing to a Raspberry Pi – GitHub.Python Releases This WeekPython 3.8.5 was released as a security hotfix. Python 3.9.0b5, the last beta before 3.9.0, is also available – Python.Org.An Smart LED Octahedron with CircuitPythonA self contained edge-lit octahedron. It’s controlled by a Serpente M0 running CircuitPython with a wire broken out for use as a capacitive touch switch. The halves snap together with homemade magnetic connectors – Twitter.Thermal Camera Python SupportUTi165 Thermal Camera Python example now works with Windows – thanks to whomever fixed the bug in opencv for Python – you can now grab frames from these low-cost thermal cameras on Mac, Linux and Windows – Adafruit Learning System.CircuitPython Deep Dive Stream with Scott ShawcroftThis week, Scott streams his work on FPGA APIs and ESP32-S2 WiFi scanning.You can see the latest video and past videos on the Adafruit YouTube channel under the Deep Dive playlist – YouTube.CircuitPython Day is 9-9-2020Adafruit has chosen September 9, 2020 (9/9/2020) as the snakiest day of this year for CircuitPython Day! Much more to come on events and happenings to include a CircuitPython team livestream, collaboration with hardware and software folks, and highlighting all things Python and Python on Hardware. More information will be forthcoming. Ideas? Send them via email to circuitpythonday@adafruit.com.Adafruit UpdateAdafruit is shipping orders!Production is up and now is the best time to get the Adafruit items you’ve been waiting for – Shop Adafruit.News from around the web!Animal Crossing: Nova Light time helper – inspired by Animal Crossing New Horizons. Powered by an Adafruit Feather M4, RTC, and NeoPixels using CircuitPython – charlyn.codes.An LED message scroller faceshield – add a real-time clock for face time – Twitter.A PyPortal Pynt clock/weather display using CircuitPython. An NTP controlled clock and a weather station display on the screen – Reddit and GitHub.Electronic Cats has announced the Bast WiFi board based on the new ESP32-S2 processor. CircuitPython support is planned via version 6.0 – Electronic Cats (Spanish).Trinket M0 with CircuitPython and Arduino – kingernorth.LoRaWAN Node with MicroPython and CircuitPython AHT10 Temp & Humid Sensor. Maxiiot DL7612-AS923-TH LoRa Module – Twitter.Why you shouldn’t remove your package from PyPI – Ovalerio.net.Understanding and Decoding a JPEG Image using Python – yasoob.me.Continuous Documentation: Hosting Read the Docs on GitHub Pages – Michael Altfield and Twitter.Documenting Python Code: A Complete Guide – Real Python and Twitter.Python Keywords: An Introduction – Real Python and Twitter.4 powerful features Python is still missing – Infoworld.Unravelling attribute access in Python – snarky.ca.How many years of experience do you have as a developer, a breakdown – Women Who Code.Work vs. automation, the meme – Twitter.PyDev of the Week: Carlton Gibson on Mouse vs PythonCircuitPython Weekly for August 3rd, 2020 notes on YouTube#ICYDNCI What was the most popular, most clicked link, in last week’s newsletter? IEEE Ranks Python #1 Programming Language.New Boards Supported by CircuitPythonThe number of supported microcontrollers and Single Board Computers (SBC) grows every week. This section outlines which boards have been included in CircuitPython or added to CircuitPython.org.This week we had 4 new boards added!MakerDiary M60 Keyboard
MakerDiary nRF52840 M.2 Developer Kit
MDBT50Q-DB-40
ScoutMakes Azul
Looking for adding a new board to CircuitPython? It’s highly encouraged! Adafruit has four guides to help you do so:How to Add a New Board to CircuitPython
How to add a New Board to the circuitpython.org website
Adding a Single Board Computer to PlatformDetect for Blinka
Adding a Single Board Computer to Blinka
New Learn Guides!CLUE Altimeter from Carter NelsonCircuitPython Sin Complicaciones para la Circuit Playground Express y la Bluefruit from Álvaro FigueroaMini VOTE Keyboard from Collin CunninghamNeoPixel Run LED Arcade Game from Noe and PedroAdafruit MicroSD SPI or SDIO Card Breakout Board from Jeff EplerCircuitPython Libraries!CircuitPython support for hardware continues to grow. We are adding support for new sensors and breakouts all the time, as well as improving on the drivers we already have. As we add more libraries and update current ones, you can keep up with all the changes right here!For the latest libraries, download the Adafruit CircuitPython Library Bundle. For the latest community contributed libraries, download the CircuitPython Community Bundle.If you’d like to contribute, CircuitPython libraries are a great place to start. Have an idea for a new driver? File an issue on CircuitPython! Have you written a library you’d like to make available? Submit it to the CircuitPython Community Bundle. Interested in helping with current libraries? Check out the CircuitPython.org Contributing page. We’ve included open pull requests and issues from the libraries, and details about repo-level issues that need to be addressed. We have a guide on contributing to CircuitPython with Git and Github if you need help getting started. You can also find us in the #circuitpython channel on the Adafruit Discord.You can check out this list of all the Adafruit CircuitPython libraries and drivers available.The current number of CircuitPython libraries is 263!Updated Libraries!Here’s this week’s updated CircuitPython libraries:Adafruit_CircuitPython_Display_Text
Adafruit_CircuitPython_SGP30
Adafruit_CircuitPython_BusDevice
Adafruit_CircuitPython_RGB_Display
Adafruit_CircuitPython_MiniMQTT
PyPI Download Stats!We’ve written a special library called Adafruit Blinka that makes it possible to use CircuitPython Libraries on Raspberry Pi and other compatible single-board computers. Adafruit Blinka and all the CircuitPython libraries have been deployed to PyPI for super simple installation on Linux! Here are the top 10 CircuitPython libraries downloaded from PyPI in the last week, including the total downloads for those libraries:Keep checking back for updated download stats coming soon!What’s the team up to this week?What is the team up to this week? Let’s check in!BryanThis week I made some updates to the CAN Bus FeatherWing and sent it off to the fine folks at OSH Park and OSH Stencils. I’m looking forward to working on this as there are lots of interesting applications for an interface for talking with CAN Bus. One of the most common places you’ll find a CAN bus is in your car.My car for example, a 2009 VW Rabbit, has a pretty interesting network that connects all the different control modules (ECUs) that allow it to have all the features that make it an old but still relatively modern car. Keyless locks, power windows, radio, indicator lights, and even the engine all talk with each other through a CAN Bus powered network, allowing you to do fun things like rolling down the windows using the key fob. I’m excited to listen to this bus to see how the different parts are talking to each other. Keep in mind, that if I had one I would with near certainty be voiding my car’s warranty, and there is always the chance of irreparable damage to the car.My immediate goal is to be able to read OBD2 diagnostic information that is mandated to be available on modern cars, at least in the US. Being an older and “aggressively loved” car that is long in the tooth, my Rabbit is no stranger to throwing codes (AKA Check Engine Lights) that would prevent me from passing a smog check. Fortunately, if you can read it, there is a good amount of information available from the car that can help troubleshoot issues. Unfortunately you need a dedicated piece of hardware to be able to read the diagnostic codes from the car.DanI’m continuing work on the ATT (attribute) protocol for HCI _bleio. Currently I’m working on the code to populate the attribute table. The information in the attribute table is passed to a peer who wants to find out all the details of the services provided by a BLE device. This process is called “discovery”. Once I have the attribute table set up, I can then implement the code that passes back the detail to the peer.I just changed our CircuitPython builds so they upload the mpy-cross executables they build to a central place. We build executables for x64 Linux, Raspbian, Windows, and MacOS. These executables will now be easier to find.With a logic analyzer, I got some timing data for the NeoPixel write operations for Adafruit’s Arduino library and CircuitPython, on both SAMD21 and SAMD51 processors. Some non-standard NeoPixel-like RGB LED’s were not working properly on CircuitPython but did work on Arduino, and this data will help us understand why.JeffThe bulk of my time has gone into a new guide which will cover both the SD card improvements in CircuitPython 6, as well as the new SD card breakout that supports SDIO. As a part of this, I have continued to prepare support for the SD card slot on the STM32F405 Feather, which requires SDIO. Other work includes pulling in updates to microlab and investigating adding the “LSB first” mode to busio.SPI.KattniThis week, I created a many more fritzing objects for upcoming Adafruit products. This will continue as we work through a huge backlog of products, since production was put on hold during the initial stages of COVID.One of the updated breakouts was the STEMMA QT version of the VL53L0X. I updated the guide to reflect the new version with a new Overview page, updated Pinouts page, updated Fritzing diagrams, and new schematic and fab prints on the Downloads page.Next up is updating the BME680 guide with the STEMMA QT version, the same way I updated the VL53L0X. As well, I have a fairly long list of miscellaneous to attend to between other major things. The busy continues!LucianThis past week I worked on implementing the NeoPixel and the PulseIn/PulseOut module combo on ESP32-S2, diving into the RMT (remote control) peripheral. Espressif has a funny habit of naming their peripherals in a way that makes them sound kind of limited, when the truth is just the opposite – just as the “LED Controller” is actually an extremely flexible and general purpose PWM controller that is also suitable for servos and tones, the “Remote Control” peripheral is capable of far more than just IR pulse trains (such as emulating the NeoPixel protocol). I’m currently considering whether I should implement a port-level mediator system for the RMT, since it’s possible we may also want to use it for things like RotaryIO, in which case it’d be good to have a similar channel-negotiator system like the one I implemented on the STM32.Over the weekend, I also took some time to play with the Open Book Featherwing from Oddly Specific Objects. I’ve always enjoyed playing with language learning software/hardware and saw an opportunity to do something along those lines on the Open Book, which is a compact e-paper e-reader that runs on CircuitPython via the Feather M4 Express (or presumably other Feathers, I haven’t checked yet). The really cool thing about the Open Book is that it has a NOR flash chip called Babel that is entirely dedicated to unicode character support – this is useful since good international embedded fonts are kind of hard to find and have a hard time fitting on the Feather 2MB NOR flash alongside CircuitPython.Shown below is my flash card app, which assembles random Japanese sentences from a json file full of “cards” and “words”, allowing you to make many variations of a particular sentence type. For instance, if you’d like to study conjugated verbs, you can design a card that will create many different sentences that focus on this topic but use different words and structures for variation. I’m hoping to expand on it to include tags and SRS features – it’s been really cool how easy it was to move a Python app designed on a PC over to CircuitPython!MelissaThis past week, I worked on the Matrix Portal library and wrote a couple of examples that will hopefully be used in some future projects. One of the examples displayed a scrolling message of YouTube subscriber and view counts for a given channel. This was based on a similar project done on the PyPortal with some changes for the lower resolution.Another example displayed the current bitcoin price. This one was pretty simple in that the library initialized the display and WiFi connections and went out to the internet to get the price and display it. This was similar to another PyPortal example that did the same thing.I’m still working on changing up the library so only certain components of it can be used and thus it will use less memory.ScottLast week I spent some time testing a PyPortal project with the refined requests library and following up on PRs from the week before. I also switched over to a desktop with Arch linux for development.On Friday during my stream, I verified ESP pinouts with my Feather adapter and ordered v3 for testing. More confident with choosing pinouts for Adafruit designs.This week I’m focussed on native WiFi on the -S2. I’ve got the scanning and connection API done in shared-bindings and am starting to hook it to the IDF. Running into some linking issues due to bringing in the WiFi stack. Good to sort out now. WIP branch is here. Check out my streams for the latest progress!Outside of CircuitPython, in my evenings, my interest in system on a chip (SoC) design has been renewed by the OrangeCrab, a Feather form factor FPGA board, and the open PDK announcement from Google. I’ve been experimenting with nMigen, a Python powered hardware description library, to create “gateware” for the OrangeCrab.Upcoming events!The Linux Plumbers Conference for 2020 will be an online event taking place 24-28 August. They are working hard to recreate as much of the feel of an in-person LPC as possible – linuxplumbersconference.org.PyCon AU has announced they are holding PyConline AU, an online event, from 4–6 September 2020 – pycon.org.au.PyGotham is a New York City based, eclectic, Py-centric conference covering many topics. PyGotham TV taking place October 2-3, 2020 with a single track of talks presented online – Event Website and Call for Proposals.PyCon India 2020: the premier conference in India on using and developing the Python programming language. Held online October 3-5, 2020. A call for proposals is now open through August 14, 2020. – Website and TwitterMeetup GroupsData Science using Python is a consortium of 970 Meetup groups with over 700K members. There are meetings happening worldwide, see their main page for details – Meetup.PyData is a consortium of 175 groups with about 170K members, and provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other – Meetup.PyLadies has 28 groups with over 28K members. Find out what’s happening in PyLadies Meetup groups around the world and start meeting up with the ones near you – Meetup.Send Your Events InAs for other events, with the COVID pandemic, most in-person events are postponed or cancelled. If you know of virtual events or events that may occur in the future, please let us know on Discord or on Twitter with hashtag #CircuitPython.Latest releasesCircuitPython’s stable release is 5.3.1 and its unstable release is 6.0.0-alpha.2. New to CircuitPython? Start with our Welcome to CircuitPython Guide.20200801 is the latest CircuitPython library bundle.1.12 is the latest MicroPython release. Documentation for it is here.3.8.5 is the latest Python release. The latest pre-release version is 3.9.0b5.1787 Stars Like CircuitPython? Star it on GitHub!Call for help – Translating CircuitPython is now easier than ever!One important feature of CircuitPython is translated control and error messages.With the help of fellow open source project Weblate, we’re making it even easier to add or improve translations.Sign in with an existing account such as Github, Google or Facebook and start contributing through a simple web interface. No forks or pull requests needed!As always, if you run into trouble join us on Discord, we’re here to help.jobs.adafruit.com – Find a dream job, find great candidates!jobs.adafruit.com has returned and folks are posting their skills (including CircuitPython) and companies are looking for talented makers to join their companies – from Digi-Key, to Hackaday, Microcenter, Raspberry Pi and more.23,742 thanks!The Adafruit Discord community, where we do all our CircuitPython development in the open, reached over 23,742 humans, thank you! Adafruit believes Discord offers a unique way for CircuitPython folks to connect. Join today at https://adafru.it/discord.ICYMI – In case you missed itThe wonderful world of Python on hardware! This is our first video-newsletter-podcast that we’ve started! The news comes from the Python community, Discord, Adafruit communities and more. It’s part of the weekly newsletter, then we have a segment on ASK an ENGINEER and this is the video slice from that! The complete Python on Hardware weekly videocast playlist is here.This video podcast is on iTunes, YouTube, IGTV (Instagram TV), and XML.Weekly community chat on Adafruit Discord server CircuitPython channel – Audio / Podcast edition – Audio from the Discord chat space for CircuitPython, meetings are usually Mondays at 2pm ET, this is the audio version on iTunes, Pocket Casts, Spotify, and XML feed.And lastly, we are working up a one-spot destination for all things podcast-able here – podcasts.adafruit.comCodecademy “Learn Hardware Programming with CircuitPython”Codecademy, an online interactive learning platform used by more than 45 million people, has teamed up with the leading manufacturer in STEAM electronics, Adafruit Industries, to create a coding course, “Learn Hardware Programming with CircuitPython”. The course is now available in the Codecademy catalog.Python is a highly versatile, easy to learn programming language that a wide range of people, from visual effects artists in Hollywood to mission control at NASA, use to quickly solve problems. But you don’t need to be a rocket scientist to accomplish amazing things with it. This new course introduces programmers to Python by way of a microcontroller — CircuitPython — which is a Python-based programming language optimized for use on hardware.CircuitPython’s hardware-ready design makes it easier than ever to program a variety of single-board computers, and this course gets you from no experience to working prototype faster than ever before. Codecademy’s interactive learning environment, combined with Adafruit’s highly rated Circuit Playground Express, present aspiring hardware hackers with a never-before-seen opportunity to learn hardware programming seamlessly online.Whether for those who are new to programming, or for those who want to expand their skill set to include physical computing, this course will have students getting familiar with Python and creating incredible projects along the way. By the end, students will have built their own bike lights, drum machine, and even a moisture detector that can tell when it’s time to water a plant.Visit Codecademy to access the Learn Hardware Programming with CircuitPython course and Adafruit to purchase a Circuit Playground Express.Codecademy has helped more than 45 million people around the world upgrade their careers with technology skills. The company’s online interactive learning platform is widely recognized for providing an accessible, flexible, and engaging experience for beginners and experienced programmers alike. Codecademy has raised a total of $43 million from investors including Union Square Ventures, Kleiner Perkins, Index Ventures, Thrive Capital, Naspers, Yuri Milner and Richard Branson, most recently raising its $30 million Series C in July 2016.Contribute!The CircuitPython Weekly Newsletter is a CircuitPython community-run newsletter emailed every Tuesday. The complete archives are here. It highlights the latest CircuitPython related news from around the web including Python and MicroPython developments. To contribute, edit next week’s draft on GitHub and submit a pull request with the changes.Join our Discord or post to the forum for any further questions.