Packaging and publishing your package is a great way to get other people interested in your project.
Here is a simple step by step guide to build and publish your first python package.
We’ll describe all the steps we’ve been through to create Scitime, a package that gives an estimate of algorithm training times (you can learn more about Scitime here).
Note: We will assume that you already have a project in GitHub that you would like to package and publish.
Step 0: Have your project licensedBefore doing anything else, your project should have a license since it’s going to be open source.
Depending on how you’d like your package to be used, many licenses are available.
Some usual licenses for open source projects are MIT or BSD.
To add a license to your project, simply add a LICENSE file to the repository root by following the steps described here.
Step 1: make your code publish-readyThere are a few preliminary things you need to do to have your project package-able:Your project structure needs to be in place.
Usually, the root of the repository contains a folder with the name of the repository — this is where the core project code should be.
What’s outside of this folder is the extra code that’s necessary to run and build the package (tests, documentation, etc.
)The core folder should include one (or more) module(s) and an __init__.
py file which imports the classes/functions you’ll want the end user to have access to.
Ideally, a proper logging system (instead of prints) should be set by using the logging package.
Ideally, your core code should be organized in one or more classes.
Example of __init__.
py if Estimator is the class that will be accessed by the end user (this class being defined in the estimate.
py file)Example of log system: the LogMixin class can then be used inside any other classStep 2: Create a setup.
py with setuptoolsOnce your project has a set structure, you should then add a setup.
py file at the root of the repository.
This mostly helps to automate all the publishing and version maintaining processes.
Here’s an example of what a setup.
py should look like (source here).
Example of a setup.
py fileA few notes:If your package has dependencies, a clean way to handle these is to add them in the setup file, through the install_requires argument (if the list is long you can always point to a requirement.
txt file as done above)If you want metadata (from the repository) to be downloaded when anyone installs the package, you should add these through the package_data argumentMore information about the setup() function can be found hereNote: steps 3 to 6 are optional (but highly recommended), however you can jump straight to step 7 if you want to publish your package right now.
Step 3: Set local tests and test coverage checksIf it is not already the case, at this point, your project should definitely have unit tests.
Although there are many frameworks that can help you do that, one simple way of doing it is using pytest.
All tests should be in a dedicated folder (named tests/ or testing/ for example).
In that folder, put all the test files you need in order to cover as much of your core code as possible.
Here’s an example of how to write a unit test.
Here’s also one of our Scitime’s test files.
Once in place, you can run your tests locally by running python -m pytest from the repository root.
Once your tests are created, you should also be able to estimate the coverage.
This is important as you want to maximize the amount of code that’s tested in your project (to reduce the amount of unexpected bugs).
A lot of frameworks are available for this as well, for Scitime we used codecov.
You can decide on a threshold of minimum coverage allowed by creating a .
codecov.
yml file and also decide on what files to include in the coverage analysis by creating a .
coveragerc file.
Example of a .
codecov.
yml fileExample of a .
coveragerc fileStep 4: Standardize syntax and code styleYou’ll also need to make sure that your code follows the PEP8 guidelines (i.
e has a standard style and that the syntax is correct).
Again, a lot of tools can help here.
We used flake8.
Step 5: Create a proper documentationNow that your project has tests and a good to structure, it should be a good time to add a proper documentation.
First thing is to have a nice readme file that will show up on your Github repo root.
Once this is done, here are a few additional “nice to have”:Pull Request & issue templates: when a new PR or issue is created, these files permit you to template the descriptions as you wish.
Follow these steps for PRs and these steps for issues to create them.
Here are Scitime’s PR template and issue template.
A contribution guide.
This should be a short guide on how you want external users to contribute to your package.
Here’s Scitime’s contribution guide.
A code of conduct (here’s Scitime’s).
Tags and a description (see screenshot below).
Badges in the readme file (here’s a nice article on how to set these up).
As the readme file should be quite synthesized, it’s also pretty standard to have a more in-depth documentation.
You can use sphinx to do so, and then host the documentation on readthedocs (here’s Scitime’s example).
The documentation related files are usually in a docs/ folder (as done for Scitime).
Here’s a nice sphinx & readthedocs tutorial.
Example of a repo that has tags and a descriptionStep 6: Set up continuous integrationAt this point, your project is not far away from being ready to be published.
However, it seems a little overwhelming to have to deal with updating the docs, running the tests and checking the style and coverage after each commit.
Fortunately, Continuous Integration (CI) helps you with that.
You can use webhooks to GitHub to automate all these things after each commit.
Here’s the set of CI tools we used for Scitime:For running tests, we used travis-ci and appveyor (for tests on windows platforms).
For travis-ci, besides setting up the webhooks on the repository, you also have to create a .
travis.
yml file in which you can not only run tests but also upload updated coverage outputs and check style and format.
Same goes for appveyor by creating a appveyor.
yml fileCodecov and readthdocs also have dedicated webhooksExample of a .
travis.
yml file: note that here, for each commit, the tests are run along with the test coverage checks.
But there’s also a flake8 check (the logic being defined in the flake_diff.
sh file)Example of a appveyor.
yml file: here, we only run the testsThis should make the whole process of updating the repository a lot easier.
Example of commit history with integrated webhooksStep 7: Create your first release and publicationAt this point, your soon-to-be package should look similar to this:your_package/ __init__.
py your_module.
pydocs/tests/setup.
pytravis.
ymlappveyor.
yml.
coveragerc.
codecov.
ymlREADME.
mdLICENSE.
github/ CODE_OF_CONDUCT.
md CONTRIBUTING.
md PULL_REQUEST_TEMPLATE.
md ISSUE_TEMPLATE/It’s now ready to be published!.The first thing to do is to create your first release on GitHub — this is to keep track of the state of your project at a given point in time, a release should be created each time the version changes.
You can follow this to create the release.
Once done, the only thing left to do is to publish your package.
The most common platforms to publish a python package are PyPI and Conda.
We will describe how to publish in both:For PyPI, you first need to create an account, and then follow these steps using twine.
This should be fairly straightforward, and PyPI also provides a test environment that can be used right before the actual deploy.
The latter basically consists of creating a source distribution (python setup.
py sdist) and uploading it with twine (twine upload dist/*).
Once done, there should be a PyPI page corresponding to your package (here’s Scitime’s) and anyone should be able to install your package running the pip command.
For conda, we recommend publishing your package through conda-forge which is a community that helps you publish and maintain your package through their conda channel.
You can follow these steps to add your package to the community, you will then be added to the conda-forge GitHub organization and be able to maintain your package quite easily (here’s Scitime’s conda-forge page) — anyone should then be able to install your package running the conda command.
All done!You should now have your package finally published and available to anyone!.Although most of the work is done, you’ll still have to maintain your project as you need to make some updates: this basically means changing the versions each time you make significant changes, creating new releases and going through step 7 again.
.. More details