What’s __init__ for me?Designing for Python package import patternsJacob DeppenBlockedUnblockFollowFollowingJan 8I have had a few conversations lately about Python packaging, particularly around structuring the import statements to access the various modules of a package.
This was something I had to do a lot of investigation of and experimentation with when I was organizing the leiap package.
Still, I have not seen a good guide to best practices in various scenarios, so I thought I would share my thoughts here.
Import patternsThe key to designing how the user will interact with the modules is the package’s __init__.
py file.
This will define what gets brought into the namespace with the import statement.
ModulesIt is usually a good idea to split code into smaller modules for a couple of reasons.
Primarily, modules can contain all of the code related to a particular coherent topic (e.
g.
, all of the I/O functionality) without being cluttered by code related to something completely different (e.
g.
, plotting).
For this reason, it is common to see large classes get a dedicated module (e.
g.
, geodataframe.
py within geopandas).
Secondarily, dividing code into appropriate logical units makes it easier to read and easier to understand.
However, a good module structure for the developer may or may not be a good module structure for the user.
In some cases, the user might not need to know that there are various modules underlying the package.
In other cases, there might be good reasons that the user should explicitly ask only for the modules they need.
That is what I want to explore here: what are the different use cases and what approach do they call for from the package developer.
An example packagePython packages come in a variety of structures, but let’s create a simple demo one here that we can use in all the examples.
/src /example_pkg __init__.
py foo.
py bar.
py baz.
py setup.
py README.
md LICENSEIt is composed of three modules: foo.
py, bar.
py, and baz.
py, each of which has a single function that prints the name of the module where the function resides.
foo.
pydef foo_func(): print(‘this is a foo function’)bar.
pydef bar_func(): print(‘this is a bar function’)baz.
pydef baz_func(): print(‘this is a baz function’)Your code as a grocery storeNow is a good time to acknowledge that talking about import statements and package structures can be pretty hard to follow, especially in text.
To help make things a bit clearer, let’s think about a Python package as a grocery store and your users as the shoppers.
As the developer, you are the store owner and manager.
Your job is to figure out how to set up your store so that you serve your customers best.
The structure of your __init__.
py file will determine that setup.
Below, I’ll walk through three alternative ways to set up that file: the general store, the convenience store, and the online store.
The General StoreIn this scenario, the user gets access to everything right away on import example_pkg.
In their code, they only need to type the package name and the class, function, or other object they want, regardless of what module of the source code it lives in.
This scenario is like an old-timey general store.
Once the customer walks in the door, they can see all the goods placed with minimal fuss in bins and shelves around the store.
Behind the scenes# __init__.
pyfrom .
foo import *from .
bar import *from .
baz import *User implementationimport example_pkgexample_pkg.
foo_func()example_pkg.
bar_func()example_pkg.
baz_func()AdvantagesUsers do not need to know module names or remember, for instance, which function is in which module.
They only need the package name and the function name.
In the general store, all the products are on display with minimal signage.
The customer doesn’t need to know which aisle to go down.
Users can access any functionality once the top-level package is imported.
Everything is on display.
Tab-completion gives you everything with just example_pkg.
<TAB>.
Tab-completion is like the general store grocer who knows exactly where everything is and is happy to help.
When new features are added to modules, you do not need to update any import statements; they will automatically be included.
In the general store, there is no fancy signage to change.
Just put a new item on the shelf.
DisadvantagesRequires that all functions and classes must be uniquely named (i.
e.
, there are not functions called save() in both the foo and bar modules).
You don’t want to confuse your customers by putting apples in two different bins.
If the package is large, it can add a lot to the namespace and (depending on a lot of factors) can slow things down.
A general store can have a lot of little odds and ends that any individual customer might not want.
That can might be overwhelming for your customers.
Requires a bit more effort and vigilance to keep some elements away from the user.
For example, you might need to use underscores to keep functions from importing (e.
g.
, _function_name()).
Most general stores don’t have a big storage area where things like brooms and mops are kept; those items are visible to the customer.
Even if it is unlikely that they would pick up a broom and start sweeping your floors, you might not want them to.
In that case, you have to take extra steps to hide those supplies from view.
RecommendationsUse when it is hard to predict the workflow of a typical user (e.
g.
, general packages like pandas or numpy).
This is the “general” part of general store.
Use when the user might frequently bounce around between different modules (e.
g.
, the leiap package)Use when function and class names are very descriptive and easy to remember and specifying the module names will not improve readability.
If your products are familiar things like fruits and vegetables, you don’t need a lot of signage; customers will figure things out quite easily.
Use with just a few modules.
If there are many modules, it can be more difficult for a new user to find the functionality they want in the docs.
If your general store gets too big, customers won’t be able to find the things they want.
Use when objects might be added or removed frequently.
It’s easy to add and remove products in the general store without disrupting the customer.
Well-known examplespandasnumpy (with additional complexity)seabornThe Convenience StorePhoto by Caio Resende from PexelsBy far the easiest to read and understand is a variation on the general store scenario that I call the convenience store.
Instead of from .
module import *, you can specify exactly what to import with from .
module import func within __init__.
py.
The convenience store shares a lot of traits with the general store.
It has a relatively limited selection of goods which can be replaced at any time with minimal hassle.
The customer doesn’t need a lot of signage to find what they need because most of the goods are easily in view.
The biggest difference is that a convenience store has a bit more order.
The empty boxes, brooms, and mops are all kept out of view of the customer and only the products for sale are on the shelves.
Behind the scenes# __init__.
pyfrom .
foo import foo_funcfrom .
bar import bar_funcfrom .
baz import baz_funcUser implementationimport example_pkgexample_pkg.
foo_func()example_pkg.
bar_func()example_pkg.
baz_func()AdvantagesShares all of the advantages of the general store, and adds:Somewhat easier to control what objects are made available to the userDisadvantages__init__.
py can end up very cluttered if there are many modules with many functions.
Like the general store, a convenience store that is too cluttered will be harder for customers to navigate.
When new features are added to a module (i.
e.
, new class or functions), they have to be explicitly added to the __init__.
py file too.
Modern IDEs can help detect missed imports, but it is still easy to forget.
Your convenience store has some minimal signage and price tags.
You have to remember to update these when you change what is on the shelf.
RecommendationsI would add the following to the recommendations from the general store:Especially useful when your modules more or less consist of a single Class (e.
g.
, from geopandas.
geodataframe import GeoDataFrame)Use when you have a small number of objects to importUse when your objects have clear namesUse when you know exactly which objects your users will need and which they will notUse when you do not expect to frequently add a lot of new modules and objects that will need to be imported.
Well-known examplegeopandasOnline grocery shoppingAnyone who has bought groceries online knows that ordering the right product can take some effort on the part of the customer.
You have to search for the product, choose a brand, choose the desired size, etc.
All of these steps, however, allow you to buy exactly what you want from a nearly limitless stockroom.
In the case of Python packages, in some cases, it might be more prudent to eschew the convenience of simply importing the entire package and instead force the user to be more clear about what pieces are being imported.
This allows you as the developer to include a lot more pieces to the package without overwhelming the user.
Behind the scenes# __init__.
pyimport example_pkg.
fooimport example_pkg.
barimport example_pkg.
bazUser implementationThere are (at least) three different methods that a user could adopt in this case.
import example_pkgexample_pkg.
foo.
foo_func()example_pkg.
bar.
bar_func()example_pkg.
bar.
baz_func()orfrom example_pkg import foo, bar, bazfoo.
foo_func()bar.
bar_func()baz.
baz_func()orimport example_pkg.
foo as ex_fooimport example_pkg.
bar as ex_barimport example_pkg.
baz as ex_bazex_foo.
foo_func()ex_bar.
bar_func()ex_baz.
baz_func()AdvantagesSimplifies the __init__.
py file.
Only needs to be updated when a new module is added.
Updating your online store is relatively painless.
You only need to change a setting in your product database.
It is flexible.
It can be used to import only what the user needs or to import everything.
The customers in your online store can search for only what they want or need.
No need to bother looking through a “fruit” bin when all you need is an apple.
But if they do want everything in the “fruit” bin, they can get that too.
Aliasing can clean up long package.
module specifications (e.
g.
, import matplotlib.
pyplot as plt).
While online grocery shopping can be a big pain at first, if you save your shopping list for the future, your shopping can be done a lot quicker.
Can have multiple objects with the same name (e.
g.
, functions called save() in both the foo and bar modules)DisadvantagesSome of the import methods can make code more complicated to read.
For example, foo.
foo_func() does not indicate which package foo comes from.
The most readable method (import example_pkg, with no alias) can lead to long code chunks (e.
g.
, example_pkg.
foo.
foo_func()) that clutter things up.
Can be hard for users to track down all of the possible functionality.
In your online grocery store, it would be hard for the shopper to see all of the possible goods.
RecommendationsUse when you have a complex series of modules, most of which any one user will never need.
Use when import example_pkg imports a LOT of objects and might be slow.
Use when you can define pretty clear workflows for different kinds of users.
Use when you can expect the user to be able to navigate your documentation well.
Examplesmatplotlib *scikit-learn *bokeh *scipy ** These packages actually use combinations of different approaches in their __init__.
py files.
I include them here because to users, they are generally used à la carte (e.
g.
, import matplotlib.
pyplot as plt or import scipy.
stats.
kde).
ConclusionThe three scenarios I outlined are certainly not the only possible structures for a Python package, but I hope they cover most of the cases that anyone reading learning this from a blog might be considering.
In conclusion, I’ll return to a point I made earlier: a good module structure for the developer may or may not be a good module structure for the user.
Whatever your decision, don’t forget to put yourself in the user’s shoes even, or especially, because that user is most likely to be you.
.