Python Concurrency API Documentation
Python concurrency provides a number of different APIs.
Although the standard library supports concurrency via coroutines, threads, and processes, there is no central location to access all of the relevant API documentation.
Additionally, other Python documentation is very useful when using Python concurrency APIs, such as HowTos, Glossary, and more.
In this guide, you will discover all of the Python concurrency API documentation.
Like me, I'm sure you will dive into this documentation often. As such you may want to bookmark this page.
Let's get started.
Python Documentation
Python provides all documentation in one place on docs.python.org.
This includes a range of different types of documentation.
Not all of this documentation is useful if we want to dig into Python concurrency.
The most useful documentation is the Python Standard Library documentation. This provides a page or pages for each Python module and lists all (or most) of the public functions and classes we can use.
Other Python API documentation helpful for concurrency includes:
Additionally, it can be helpful to read the source code for the stdlib when trying to understand the behavior of a class or function.
The source code for the standard library is accessible and readable on GitHub.
We will take a closer look at each of these areas and highlight all of the specific API documentation pages you should consider when developing concurrent Python applications.
The Python Standard Library
The most Python standard library, or stdlib, represents the bulk of modules, functions, and classes that come with Python.
Python’s standard library is very extensive, offering a wide range of facilities as indicated by the long table of contents listed below. The library contains built-in modules (written in C) that provide access to system functionality such as file I/O that would otherwise be inaccessible to Python programmers, as well as modules written in Python that provide standardized solutions for many problems that occur in everyday programming.
-- The Python Standard Library
Importantly, Python provides concurrency out of the box, e.g. in the stdlib. Nothing extra needs to be installed.
The Python Standard Library documentation page provides an excellent overview of all modules in the library organized into logical groups.
The two groups that are most relevant for Python concurrency development are:
- Concurrent Execution
- Networking and Interprocess Communication
Each of these areas has a dedicated page that details the relevant modules.
Let's take a closer look at each of these in turn.
Python Concurrent Execution
The "Concurrent Execution" group contains a suite of modules relevant for classical concurrency with threads and processes.
We can access the dedicated page for the group here:
It is described as modules that support CPU-bound and I/O bound tasks.
... support for concurrent execution of code. The appropriate choice of tool will depend on the task to be executed (CPU bound vs IO bound) and preferred style of development (event driven cooperative multitasking vs preemptive multitasking).
-- Concurrent Execution
This involves 10 top-level modules, each with a dedicated webpage detailing the specific classes and module functions available.
They are:
- threading — Thread-based parallelism
- multiprocessing — Process-based parallelism
- multiprocessing.shared_memory — Shared memory for direct access across processes
- concurrent — The concurrent package
- concurrent.futures — Launching parallel tasks
- subprocess — Subprocess management
- sched — Event scheduler
- queue — A synchronized queue class
- contextvars — Context Variables
- _thread — Low-level threading API
The two most important modules are the "threading" and the "multiprocessing" modules that provide thread-based concurrency and process-based concurrency in Python.
You can learn more about these modules in the guides:
The next most important module is the "concurrent.futures" module that provides the ThreadPoolExecutor and ProcessPoolExecutor classes for modern "executor style" thread and process pools in Python.
You can learn more about these classes in the guides:
The multiprocessing module has its own submodules.
Some of these submodules are described on the multiprocessing page itself, and some are not.
The full list of multiprocessing submodules is as follows:
- multiprocessing.connection API for dealing with sockets.
- multiprocessing.dummy Dumb wrapper around threading.
- multiprocessing.managers Share data between process with shared objects.
- multiprocessing.pool Create pools of processes.
- multiprocessing.shared_memory Provides shared memory for direct access across processes.
- multiprocessing.sharedctypes Allocate
Perhaps one of the more important submodules of the multiprocessing module is the "pool" submodule.
It provides the Pool class and ThreadPool class that implement a more classical thread and process pool support in Python.
You can learn more about these classes in the guides:
Python Networking and Interprocess Communication
The "Networking and Interprocess Communication" provides modules that add support for socket I/O and inter-process communication or IPC.
We can access the dedicated page for the group here:
The IPC support is capable of communication both on the same machine and across systems on a network.
The modules described in this chapter provide mechanisms for networking and inter-processes communication. Some modules only work for two processes that are on the same machine, e.g. signal and mmap. Other modules support networking protocols that two or more processes can use to communicate across machines.
-- Networking and Interprocess Communication
This involves 7 top-level modules, each with a dedicated webpage detailing the specific classes and module functions available.
They are:
- asyncio — Asynchronous I/O
- socket — Low-level networking interface
- ssl — TLS/SSL wrapper for socket objects
- select — Waiting for I/O completion
- selectors — High-level I/O multiplexing
- signal — Set handlers for asynchronous events
- mmap — Memory-mapped file support
Perhaps only one is most relevant to Python concurrency and that is the "asyncio" module.
This module provides an ecosystem of dedicated pages describing the high-level and low-level APIs.
They are:
High-level Asyncio APIs
Low-level Asyncio APIs
Asyncio Guides and Tutorials
The Python Language Reference
The Python Language Reference provides a summary of all the Python language constructs.
This includes the syntax and semantics of the language itself.
This is good stuff to know generally.
There are some aspects of the language we need to know in greater detail relevant to Python concurrency. This mostly relates to constructs that were introduced for coroutine-based concurrency in asyncio.
The most relevant parts of the language reference for concurrency are as follows:
Data model
The Python data model reference describes the standard objects and their relationships.
The key part of the data model to understand for concurrency are coroutines.
This includes subsections on:
Expressions
The Python expressions reference describes Python expressions.
There are a number of new expressions introduced for working with coroutines.
Including "async def", "async for", "async with", and "await", and "await for".
The relevant parts of the API to learn more about these expressions are as follows:
- Displays for lists, sets and dictionaries
- Generator expressions
- Yield expressions
- Asynchronous generator functions
- Asynchronous generator-iterator methods
- Await expression
Compound statements
The Python compound statements reference describes Python expressions.
This section provides more detail on the expressions relevant to coroutines, although in this case with a focus on compound statements.
The most important section is on "Coroutines"
It covers the following:
Python HOWTOs
The Python HOWTOs provides tutorials on how to implement specific solutions to common technical problems in Python programs.
The majority of the HOWTOs are not relevant to Python concurrency.
However, there are a few that are relevant and important, specifically related to logging and socket I/O.
They are:
The Logging cookbook provides examples of logging from multiple threads and multiple processes.
For example:
- Logging from multiple threads
- Logging to a single file from multiple processes
- A more elaborate multiprocessing example
The socket programming provides some information about how threads behave when sockets die.
Python Frequently Asked Questions
The Python FAQs provide answers to many commonly asked questions, organized by page into topics.
The most relevant FAQ topic pages for Python concurrency is "Library and Extension FAQ".
This page has a section on threads with answers to the following questions:
- How do I program using threads?
- None of my threads seem to run: why?
- How do I parcel out work among a bunch of worker threads?
- What kinds of global value mutation are thread-safe?
- Can’t we get rid of the Global Interpreter Lock?
Python Glossary
The Python Glossary provides a list of terms and their definitions.
There are a lot of terms here with precise definitions.
I prefer to rely on these definitions wherever possible, to remain consistent with the Python community.
There are many terms in the glossary relevant to concurrency, a select few important terms you may want to review include the following list:
- asynchronous context manager
- asynchronous generator
- asynchronous generator iterator
- asynchronous iterable
- asynchronous iterator
- awaitable
- context variable
- coroutine
- coroutine function
- GIL
- global interpreter lock
Standard Library Source Code
The source code for the Python standard library is part of the CPython project on GitHub.
This is the project for the default C-based Python interpreter.
Reviewing the code directly will always review the most recent version of the code, which might be for a version of Python different from the one you are using.
This often does not matter a lot if you are using a very recent version of the interpreter.
There are many relevant Python files directories to consider when reviewing code related to concurrency.
A select sample of files and directories to consider are listed below.
- Python Threading: threading.py
- Python Threading Queues: queue.py
- Python Multiprocessing: multiprocessing
- Python Multiprocessing: multiprocessing/dummy
- Python Asyncio: asyncio
- Python Concurrent Futures concurrent/futures
Python Concurrency PEPs
PEP stands for Python Enhancement Proposal.
It is a procedure used in the Python community for managing extensions to the language and standard library.
PEP: Python Enhancement Proposal. A PEP is a design document providing information to the Python community, or describing a new feature for Python or its processes or environment. PEPs should provide a concise technical specification and a rationale for proposed features.
-- Python Glossary
All PEPs are stored and managed on a dedicated subdomain peps.python.org
A number of Python concurrency capabilities were originally proposed and managed as PEPs before being included into the language and standard library.
They can be helpful to read in order to understand the motivation for specific language and library features. They also provide concise examples that can help to understand specific classes and methods.
Not all features have a PEP. For example, the _thread and threading modules were added to Python early in its development life, before the PEP process.
The list below contains those most relevant to Python concurrency.
Threading
Multiprocessing
Concurrent Futures
Coroutines and Asyncio
- PEP 342 – Coroutines via Enhanced Generators
- PEP 334 – Simple Coroutines via SuspendIteration
- PEP 380 – Syntax for Delegating to a Subgenerator
- PEP 492 – Coroutines with async and await syntax
- PEP 525 – Asynchronous Generators
- PEP 530 – Asynchronous Comprehensions
- PEP 3145 – Asynchronous I/O For subprocess.Popen
- PEP 3153 – Asynchronous IO support
- PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module
Python Wiki
Python provides a public wiki that allows the community to collect resources on Python programming.
Welcome to the Python Wiki, a user-editable compendium of knowledge based around the Python programming language.
-- The Python Wiki
The wiki is hosted on the subdomain wiki.python.org.
The Wiki does provide many helpful links to PyCon videos related to Python concurrency and information on the GIL.
Some of the most useful Python wiki pages related to Python concurrency are as follows:
Takeaways
You now know how to navigate the Python API documentation relevant for concurrency.