25 February 2014
There comes a time in every engineer's career when they are faced with the
question: "How the f#@% am I going to deploy this application?". Don't worry,
that's normal. You might find yourself using coping mechanisms like
scp, but sooner or later the problem must be addressed.
The canonical solution for this problem is to use packages. Typically rpm's or deb's; something linked to the operating system. For those of you deploying python code, there is a third1 way: pip. Pip has a few distinct advantages over OS packaging:
If you use pip, you'll need to host your own Python Package Index (PyPI). There are several options to choose from: pypiserver, pyshop, mypypi, and a host of others. Most of these options are easy to set up and simply store the packages on disk. I initially used one of these and had no complaints…until our servers went down. Then I had to rebuild and re-upload a bunch of packages by hand in the midst of an outage. After that experience, I decided that we had to use something more stable for the backend. S3 came to mind.
I initially built PyPICloud to store packages in S3 and cache the package list using SQLAlchemy, but decided to make it more generic. It now supports storing packages in S3 or as files on disk, and it caches the package list using SQLAlchemy or Redis. It has a full ACL, including groups and read/write permissions on a per-package basis. Permissions and user credentials can be specified in a config file, stored in a SQLAlchemy database, or delegated to your own HTTP API. All of these components are pluggable, and it should be very easy to write your own if you want to store packages in Cassandra, MongoDB, or anything else.
I set up a live demo at pypi.stevearc.com. You can
browse the web interface and install packages from there. If you want to try
running it yourself, there are quick-start instructions on
github. If you have docker
installed you can start it with
docker run -p 8080:8080 stevearc/pypicloud.
If people like it I might add more features, like LDAP support or something. If you have any feature requests, file an issue on github!