Starting A Python Project

Organization and Structure

When starting a new Python project, it can be tempting to open up a code editor and start writing some code. This is all well and good if you're just testing something out. But ideally, if you want your code to solve some specific need, then you'll want to structure your project accordingly.

Who Is This For?

This post is, in part, inspired by Guilherme Latrova's article, Organize Python code like a PRO. If you're just getting started with Python, or if you've been working with it for a while and want to get a better sense of how to organize your projects, do yourself a favor and read it!

This post is mostly being written to remind myself what steps to take when starting a project, as I will inevitably forget something down the road.

In case you're here, I'll also try to add some contextualization in case that's helpful.

Keep It Organized

Regardless of what you're coding, you will have a better time if you keep things organized. For me, this usually means having a file structure that compartmentalizes different aspects of my code.

This usually means that I have a "main" file, that, for all intents and purposes, functions as a table of contents for the rest of my code.

I then try to organize my code in separate modules.

Python modules can be imported by other .py files, and that is where I keep distinct, related pieces of code.

(For a good, concise explanation on what makes a Python script/module, see Trey Hunter's article, Module vs Script in Python).

Structure

As a jumping off point, here is what my typical project structure might look like as I get things going.

project_root/
┣ .venv/                        # virtual environment
┣ app/
┃ ┣ core/                       # base app logic
┃ ┃ ┣ config.py
┃ ┃ ┣ exceptions.py
┃ ┃ ┣ log_config.py
┃ ┃ ┗ __init__.py
┃ ┣ util/                       # app specific tasks
┃ ┃ ┣ do_thing.py
┃ ┃ ┣ do_other_thing.py
┃ ┃ ┗ __init__.py
┃ ┣ main.py                     # entry point to application
┃ ┗ __init__.py
┣ tests/                        # tests live outside application
┣ .gitignore
┗ README.md

At times, an additional nest inside a src directory may make sense, depending on what you're trying to do.

project_root/
┣ src/
┃ ┣ app/
┃ ┃ ┣ core/
┃ ┃ ┣ util/
┃ ┃ ┣ main.py
┃ ┃ ┗ __init__.py
┃ ┗ tests/
┣ .gitignore
┗ README.md

So how does structuring your project help you in the long run?

Project Root

You can name your project_root folder whatever you want. But this is where everything related to your project will live.

It can be tempting to just start creating a bunch of .py files, and then link them all together in some spaghetti-noodle type of way.

And if that works for you, who am I to stand in your way?

For me, I like to streamline the amount of time it takes me to open my project, get acquainted with what is what, and start doing some actual coding.

This has been almost impossible when I don't take time with these other pieces.

README

By looking at this project file structure, you should more or less know what to expect when you open up any of the .py files.

But chances are, if you've spent any considerable amount of time away from your code, it may take you a moment to remember what is what.

That is why I would highly recommend creating a README.md file. Even if you don't intend for anyone other than yourself to see your little package, it will save you a lot of time down the road.

The contents should more or less remind you what it is that you're trying to accomplish, and how to go about accomplishing that very thing.

Even if it's the reminder of a command like:

python -m app.main

.gitignore

If you're new to coding, it's possible that Git intimidates you. I'm not here to demystify the whole thing, but I'm definitely going to encourage you to embrace it.

Even if you're not coding with others, a major advantage of using git will be the ability to roll things back if something goes wrong.

In addition, if you get something working (say, in "production"), it's great to be able to have a "development" version, knowing full well that you're not messing up the work you've already done.

This will make your life easier.

For that reason, one of the first things I do in my project is create a .gitignore file. All this does is it tells Git which files/folders to ignore.

I use the .gitignore Generator plugin for VSCode, but there are a lot of tools to help with this.

I know it's kind of a pain, but spending a little time familiarizing yourself with Git will pay off in the long term.

Virtual Environment

I briefly wrote about this before, but I want to reiterate here. In almost every case, you want to ensure you have a virtual environment created for your project.

Using python -m venv .venv (or python3 -m venv .venv if you're on a Mac) should do the trick.

Remember, if you're in the terminal, this will create the .venv directory in your current working directory, so make sure you are in your project_root when doing so.

It's one thing to create a virtual environment, but it won't do you any good if you don't activate it.

I could spend a lot more time talking about why this matters, but that may have to wait for a different post.

Once you have activated your virtual environment, then you can begin installing any packages you may need in your project.

I do want to talk about dependencies some day, but for now, I'll link to this great resource by James Bennett, Boring Python: dependency management.

It may be heavy if you're just starting out, but still worth bookmarking, if nothing else.

main.py

In your app directory, you should have a main.py and __init__.py, the latter which makes app a "package."

As I mentioned above, the main.py should more or less map out what your application is doing.

I would try to make the code as easy to read as possible. Leave all the logic and heavy lifting to the other modules in your application.

For example, here is a slightly condensed version of the main.py I use for this site.

from fastapi import FastAPI

from pbn.api.api_v1.api import api_router
from pbn.core import log_rich
from pbn.core.config import get_app_settings
from pbn.db.db import disconnect_mongo, init_db
from pbn.lib.util import incl_static


def get_app() -> FastAPI:

    settings = get_app_settings()

    # Creap app instance with parameters defined in settings
    app = FastAPI(**settings.fastapi_kwargs)

    log_rich.setup_rich_logger()  # Use Rich logging/tracebacks
    incl_static(app)  # mount static assets (css, images, etc...)

    app.add_event_handler("startup", init_db)  # connect to db
    app.add_event_handler("shutdown", disconnect_mongo)

    app.include_router(api_router)  # All routes registered here

    return app

app = get_app()

What you'll note is that a lot of the app's heavy lifting is happening elsewhere. But this gives you a sense of what's happening when the app is starting up.

As you can imagine, the app structure is further broken down by concern. For example, if something goes wrong with my database connection, I may start looking in my db directory instinctively.

Core

So what's the deal with this core directory?

For me, it provides a location for modules that act as a skeleton for my application. This often means configuration, logging, and custom exceptions.

Each one of these things is extremely useful for any application, but you may find that you don't need any of them.

I would argue that having even a simple configuration file might be worth it, though, if only to give yourself flexibility to scale your little program.

Sure, it might just be some pet project, but it might also grow into something you weren't expecting, so why limit yourself?

I would also argue that logging is a wonderful tool that shouldn't be ignore when developing.

Util

At this point, you probably get where I'm going with this. You may find that util is too broad and not specific enough. This is probably a good instinct.

At this point, you can start thinking about what you're trying to accomplish with your little program.

You can find a plethora of articles on different design paradigms, and they probably all have something of value.

But if you find yourself spending too much time thinking about a paradigm and not actually coding, then you're probably doing it wrong.

The idea is to reduce the complexity you see when opening your project, and reducing your time to code.

Tests

This is an all-too-short section telling you that you should always plan to test your code. All I'll say is that, contrary to what you may think, testing will actually speed up your development time.

Unless you are able to write code that works perfectly the first time around, testing will provide you with immediate feedback when you make changes to your code.

This, in turn, will reduce the frustration you may feel when one piece of code breaks something else.

Conclusion

This post started with a slightly different intent. I wanted to touch on a lot of different things, but this post is already a bit too long.

Ultimately, I hope it helps capture a process that is easy to take for granted. Experienced developers likely have more complex and streamlined processes to get started with their projects.

But for me, even now, every time I start a new project I need to remind myself of my own "best practices" that minimize the amount of time I spend "thinking" about what I'm trying to accomplish.

Hopefully this helps some of you also think about your own process.