Python Concurrency Learning Paths
Python concurrency is a big topic and can be challenging for beginners.
This page provides you with step-by-step learning paths that you can use to get started and get good at Python concurrency.
You might want to bookmark this page and come back to it again and again.
Table of Contents
New to Concurrency in Python
The first step is to get familiar with the basics of concurrency in Python.
- Step 1: Get familiar with the Python concurrency landscape
- Step 2: Get familiar with the Python concurrency APIs that are available
- Step 3: Get familiar with the API docs, books, and glossary of terms.
You can then dive into one of the specific concurrency APIs below.
Threading Learning Path
Python provides thread-based concurrency in the “threading” module.
This section provides a learning path that you can use to get started and get effective with the threading module, super fast.
- Step 1: Discover what a thread is in Python.
- Step 2: Discover how to run code in a new thread.
- Step 3: Discover how to get the most out of threads.
Next, you can browse all tutorials or dive into a topic below to learn more.
Configure Threads
- How to Configure Threads
- How to Change the Thread Name
- How to Use Daemon Threads
- Handle Unexpected Exceptions
Basic Threading
- Thread Blocking Call
- What is the Main Thread
- How to Query Thread Attributes
- Thread Utility Functions
- How to Use Thread-Local Data
- How to Join a Thread
- How to Restart a Thread
- Wait for a Result from a Thread
Intermediate
- Share Variables Between Threads
- Threading Return Values
- How to Stop a Thread
- How to Close a Thread
- How to Kill a Thread
- Auto-Start Threads
- How to Sleep a Thread
Advanced Threading
- 5 Threading Anti-Patterns
- Interrupt the Main Thread
- Thread Context Variables
- Thread Stack Size
- Context Switch Interval
Thread-Safe Queues
- Thread-Safe Queue
- Thread-Safe SimpleQueue
- Thread-Safe LifoQueue
- Thread-Safe Priority Queue
- Queue task_done() and join()
Background Threads
Thread Safety
- Thread Atomic Operations
- Volatile Variables
- Thread-Safe Counter
- Thread-Safe Print
- Thread-Safe Logging
- Thread-Safe Dictionary
- Thread-Safe List
- Thread-Safe Write to File
- Thread-Safety Context Managers
- Thread-Safe Random Numbers
Failure Modes
- How to Identify a Deadlock
- Thread Livelocks in Python
- Race Condition With Timing
- Race Condition Shared Variable
- Thread Starvation in Python
Sync Primitives
- Threading Mutex Lock
- Threading RLock
- Threading Condition Variable
- Threading Semaphore
- Threading Event Object
- Threading Timer Thread
- Threading Barrier
- Lock vs Semaphore
- Advanced Semaphore Examples
Intermediate Locking
Advanced Locking
- Lock Contention
- Lock Striping in Python
- Lock Splitting in Python
- Lock Retry with Back-Off
- Thread Lock Overhead
- How to Use a Spinlock
- How to Use Busy Waiting
Thread Patterns
Multiprocessing Learning Path
Python provides process-based concurrency in the “multiprocessing” module.
This section provides a learning path that you can use to get started and get effective with the multiprocessing module, super fast.
- Step 1: Discover what a process is in Python.
- Step 2: Discover how to run code in a new child process.
- Step 3: Discover how to get the most out of processes.
Next, you can browse all tutorials or dive into a topic below to learn more.
Configure Processes
- Change the Process Name
- Daemon Process in Python
- Multiprocessing Start Methods
- Configure Child Process
Basic Processes
- Main Process in Python
- Main Thread of Process
- Get Process By Name
- How to Get the Process PID
- Get the Number of CPUs
- Process Utility Functions
- How to Query Process Status
- How to Restart a Process
- How to Join a Process
- Child stop main process
- print() from a Child Process
Intermediate
- Multiprocessing for-loops
- Nested for-loops
- Why Not Always Use Processes
- Multiprocessing Best Practices
- Multiprocessing Logging
- Threads in Processes
- Automatically Start Processes
- Return Value From Process
Advanced Processes
- Multiprocessing Context
- Multiprocessing Freeze Support
- Multiprocessing vs GIL
- Multiprocessing vs Threading
- Inherit Global Variables
- Shared ctypes in Python
Failure Modes
Parents and Children
Sync Primitives
- Process-Safe in Python
- Multiprocessing Lock
- Multiprocessing RLock
- Multiprocessing Event Object
- Multiprocessing Semaphore
- Multiprocessing Condition Var
- Multiprocessing Barrier
Queues and Pipes
- Multiprocessing Queue
- Multiprocessing SimpleQueue
- Multiprocessing JoinableQueue
- Multiprocessing Pipe
Killing Processes
- How to Safely Stop a Process
- Kill a Process in Python
- Kill Process By PID
- Kill All Child Processes
- Orphaned Processes
- Exit a Process with sys.exit()
- Process Exit Codes
Managers
Asyncio Learning Path
Python provides coroutine-based concurrency in the “asyncio” module.
This section provides a learning path that you can use to get started and get effective with the asyncio module, super fast.
- Step 1: Discover asyncio programs in Python.
- Step 2: Discover how to create and run tasks in asyncio.
- Step 3: Discover how to get the most out of asyncio.
Next, you can browse all tutorials on the topic.
Asyncio Language
- What is async/await
- How to use “async def”
- How to use “async for”
- How to use “async with”
- How to use “await”
- What are awaitables
Async Tasks
- What is a Task
- How to create a task
- How to check task status
- How to get task results
- How to cancel a task
- How to handle task exceptions
- How to use task names
- How to use done callbacks
- How to get coroutine from a task
- How to get the current task
- How to get all tasks
- What is a “pending” task
Iterators and Generators
Async Primitives
- Coroutine safety
- Async mutex locks
- Async events
- Async condition variables
- Async semaphores
- Async bounded semaphores
- Using thread locks in asyncio
- Coroutine race conditions
- Coroutine deadlocks
Advanced Operations
Async Streams
Basics
- What is asyncio
- Why use asyncio
- What is the event loop
- Asyncio vs threading
- Asyncio common errors
- Asyncio runtime warning
- Coroutines faster than threads
- Coroutines use less memory
- Two asyncio APIs
Coroutines
- What is a Coroutine
- How to run a coroutine
- Get return value from coroutine
- Run coroutine in the background
- Run coroutine after a delay
- How to sleep() a coroutine
- How to get a task for a coroutine
- How to run coroutine from thread
- Coroutine function types
Advanced Tasks
- How to wait_for() tasks
- How to shield() tasks
- How to run task in thread
- Run blocking tasks in asyncio
Groups of Tasks
- How to use gather()
- How to use gather() callbacks
- How to cancel all gather() tasks
- How to wait() for tasks
- How to as_completed() tasks
Async Queues
Async Subprocesses
ThreadPoolExecutor Learning Path
Python provides thread pool-based concurrency with the “ThreadPoolExecutor” class.
This section provides a learning path that you can use to get started and get effective with the ThreadPoolExecutor class, super fast.
- Step 1: Discover the ThreadPoolExecutor class in Python.
- Step 2: Discover how to run code in the ThreadPoolExecutor.
- Step 3: Discover how to get the most out of ThreadPoolExecutor.
Next, you can browse all tutorials or dive into a topic below to learn more.
Configure
- Configure Max Workers
- How to Configure Thread Names
- Worker Initializer Functions
- How to Use the Context Manager
Basics
- map() vs. submit()
- wait() vs. as_completed()
- Common Usage Patterns
- Wait For The First Task To Finish
- Wait For All Tasks to Finish
- How To Get Task Results
- How to Check Task Status
- How to Add a Callback to a Task
Intermediate
Comparisons
- ThreadPoolExecutor vs. Thread
- ThreadPoolExecutor vs. GIL
- ThreadPoolExecutor vs. AsyncIO
- ThreadPoolExecutor vs. PPE
Advanced
- How to Limit Pending Tasks
- How to Retry Failed Tasks
- How to Stop All Tasks if Task Fails
- How to Submit a Follow-up Task
- ThreadPoolExecutor Thread-Safe
- How To Identify Deadlocks
Failure Modes
Example Applications
ProcessPoolExecutor Learning Path
Python provides process pool-based concurrency with the “ProcessPoolExecutor” class.
This section provides a learning path that you can use to get started and get effective with the ProcessPoolExecutor class, super fast.
- Step 1: Discover the ProcessPoolExecutor class in Python.
- Step 2: Discover how to run code in the ProcessPoolExecutor.
- Step 3: Discover how to get the most out of ProcessPoolExecutor.
Next, you can browse all tutorials or dive into a topic below to learn more.
Configure
- Configure Max Workers
- Config Worker Initialize Function
- How to Use The Context Manager
- Config Multiprocessing Context
Basics
- map() vs submit()
- wait() vs as_completed()
- Common Usage Patterns
- How to Wait For The First Result
- How to Wait For a Task to Finish
- How to Get Results
- How to Check the Status of a Task
- How to Add a Callback to a Task
Intermediate
ThreadPool Learning Path
Python provides thread pool-based concurrency with the “multiprocessing.pool.ThreadPool” class.
This section provides a learning path that you can use to get started and get effective with the ThreadPool class, super fast.
- Step 1: Discover the ThreadPool class in Python.
- Step 2: Discover how to run code in the ThreadPool.
- Step 3: Discover how to get the most out of ThreadPool.
Next, you can browse all tutorials on this topic.
Basics
- The dummy module
- ThreadPool class
- ThreadPool life-cycle
- Should we use ThreadPool
- Concurrent for-loops
Configure
Intermediate
Advanced
- When are workers started
- Does ThreadPool stop main
- How to get results from async
- Tasks fail silently
- Is ThreadPool slower
- Get num remaining tasks
- ThreadPool exception handling
Callback Functions
Issue Tasks
- Issue tasks with apply
- Issue tasks with apply_async
- Issue tasks with map
- Issue tasks with map_async
- Issue tasks with imap
- Issue tasks with imap_unordered
- Issue tasks with starmap
- Issue tasks with starmap_async
- How to choose a method
- Use map with multiple args
- How to use AsyncResult
- How to configure chunksize
Common Tasks
- ThreadPool best practices
- Common usage patterns
- Common errors
- How to log from tasks
- How to stop all tasks
- Stop all if one task fails
- Issue follow-up tasks
- Wait for all tasks to finish
- How to show progress
- How to get first result
Comparisons
Multiprocessing Pool Learning Path
Python provides process pool-based concurrency with the “multiprocessing.Pool” class.
This section provides a learning path that you can use to get started and get effective with the multiprocessing.Pool class, super fast.
- Step 1: Discover the multiprocessing.Pool class in Python.
- Step 2: Discover how to run code in the multiprocessing.Pool.
- Step 3: Discover how to get the most out of multiprocessing.Pool.
Next, you can browse all tutorials on this topic.
Configure
- Config num workers
- Use worker initalizer
- Config max tasks per child
- Config pool context
- How to configure
Basics
- Pool PEP
- Context manager interface
- How to shutdown
- How to join
- Exception Handling
- Pool stop main exiting
- When workers are started
- Workers restarted if killed
Intermediate
- Share global vars
- Stop all tasks
- Kill all tasks
- Get all worker PIDs
- Get all worker names
- Get all thread details
- Tasks fail silently
- Get first result
- Get result from async
Advanced
Common Usage
Use with Primitives
Issue Tasks
- Issue tasks with apply
- Issue tasks with apply_async
- Issue tasks with map
- Issue tasks with map_async
- Issue tasks with imap
- Issue tasks with imap_unordered
- Issue tasks with starmap
- Issue tasks with starmap_asyc
- How to choose method
- How to use AsyncResult
- map() multiple arguments
- map() chunksize
Callback Functions
Comparisons
Concurrent File I/O Learning Path
Working with files can be slow.
This section provides a learning path that you can use to get started and get effective at using Python concurrency to speed-up file I/O operations, super fast.
- Step 1: Discover Python concurrency techniques.
- Step 2: Discover Python file I/O operations.
- Step 3: Discover how to perform file I/O concurrently.
- Multithreaded File Saving in Python
- Multithreaded File Loading in Python
- Multithreaded File Deletion in Python
- Multithreaded File Copying in Python
- Multithreaded File Moving in Python
- Multithreaded File Renaming in Python
- Multithreaded File Appending in Python
- Multithreaded File Unzipping in Python
- Multithreaded File Zipping in Python
Next, you can browse all tutorials on this topic.
Need More Help?
Do you need more help with Python concurrency?
Perhaps you need help with a topic not covered above?
- Contact me directly and I will do my best to help.