Develop Concurrent Python Programs, Once You Learn Threading
Imagine developing Python programs that scaled to as many as 1,000+ concurrent tasks (and much more).
Your python code does not have to be slow! (or limited by the GIL)
If you can invest as little as one hour per day, then in one week you will know how to confidently bring concurrent programming to your current and future projects.
Convinced? Jump to the book, otherwise read on…
How much faster could your code run
(if it used 10s, 100s or even 1,000s of threads)?
How many times has this happened to you…
- …you write a program to download many files
- …you write a program to execute many database queries
- …you write a program to read many files
You use a loop, one iteration per file, per query, etc.
And it is SO SLOW.
You run the script and are frustrated at how long it takes.
Yet, you have 2, 4, 8 or more CPU cores sitting idle.
Waiting for work.
What a waste!
- What if you could develop Python programs that were concurrent from the start?
- What if your concurrent Programs performed each task simultaneously in separate threads?
This is possible right now with a rarely used Python module that offers thread-based concurrency (and is already installed on your system).
Python concurrency is the path to faster code
Python is a joy to use, but getting Python code to run fast is challenging.
Concurrency is a standard approach to running multiple functions simultaneously.
Python concurrency has a bad reputation. So bad, that many developers believe Python does not support true concurrency.
I’m happy to say that these misconceptions are dead wrong.
Python supports real concurrency with first-class native support for threads and processes.
It always has.
- …on all recent Python versions, like Python v2.6+ and v3.0+.
- …on all major platforms, like Windows, macOS, and Linux.
- …with all major hardware, like Intel, AMD, ARM, and Apple Silicon.
And most importantly, Python concurrency is easy and fun to use.
The trick is to use the threading module and the right types of tasks.
“but threads don’t work because of the GIL…”
Have you heard this before:
“Python doesn’t support threads because of the Global Interpreter Lock (GIL).“
Yes, it’s true. The threading module is limited by the infamous Global Interpreter Lock (GIL).
Critically, the GIL is released allowing multiple threads to run in parallel.
When is the GIL released?
- …the GIL is released when performing blocking I/O
- …the GIL is released when calling some third-party libraries
- …the GIL does not even exist in some third-party Python interpreters
The first point is key.
The GIL is released when performing blocking IO, such as:
- Reading or writing a file from the hard drive.
- Reading or writing to standard output, input, or error (stdin, stdout, stderr).
- Printing a document.
- Downloading or uploading a file.
- Querying a server.
- Querying a database.
- Taking a photo or recording a video.
- And so much more.
“but multiprocessing offers full parallelism…”
Yes, the multiprocessing module offers full parallelism via process-based concurrency.
It’s amazing at one thing:
There are three massive downsides of using processes as the unit of concurrency for IO-bound tasks:
- …they are heavyweight and slow to start
(and we need to start many threads when doing IO)
- …they are slow to share data
(and we need to read or write lots of data when doing IO)
- …operating systems may limit the number that can be created
(and we need 100s or 1,000s for IO)
You need threads!
Threads are the tool to use for IO-bound tasks, and the threading module is the way to use them in Python.
Threading: Python Concurrency for IO-bound Tasks
The threading module provides easy-to-use thread-based concurrency.
This is not some random third-party library that is hard to install, this is the Python standard library (already installed on your system).
This is the module you need to use to make your code run faster.
It is specifically designed for you to develop concurrent Python programs.
If you work with the Global Interpreter Lock, you can easily run 10s to 1,000s of threads simultaneously.
There’s just one problem.
Few developers know about it (or how to use it well).
- …the API documentation is thin at best, providing very little guidance on how to use its features
- …the API docs are buried deep within the Python documentation website, making it impossible to find
- …the few examples out there on the web are terse and 10 years out of date
And worst of all:
- …most Python developers think Python does not support threads
This is madness!
The perfect module for faster Python code for IO-bound tasks is right there in the standard library, and very few Python developers know it exists or can locate it, let alone believe it will work.
“Python Threading Jump-Start”
“Python Threading Jump-Start” is my new book that will teach you how to develop threading programs in Python, from scratch.
This book distills only what you need to know to get started and be effective with threading, super fast.
It’s exactly how I would teach you threading if we were sitting together, pair programming.
- 7 lessons taught by example with full working code listings.
- 26 .py code files included that you can use as templates
- 106 pages for on-screen reading open next to your editor.
- 2 formats (PDF and EPUB) for screen, tablet, and kindle reading.
- 2 megabyte .zip download that contains the ebook and all source code.
“Python Threading Jump-Start” will lead you on the path from a Python developer interested in concurrency to a developer that can confidently develop programs using threads.
- No fluff.
- Just concepts and code.
- With full working examples.
The book is divided into 7 lessons.
The idea is you can read and work through one lesson per day, and become capable in one week.
It is the shortest and most effective path that I know of transforming you into a Python threading developer.
Choose your package:
- Just the book
“Python Threading Jump-Start”
- All 7 books
Boxed set of the full
- All materials
Get it all with the
7 jump-start books
4 interview q&a books
7 printable cheat sheets
4 API mindmaps
Warning: Only for serious developers!
All prices are in USD.
(also get the book from Amazon and GooglePlay stores)
You Get 7 Laser-Focused Lessons
The structure of the book is a course with 7 lessons.
- Lesson 01: Thread-Based Concurrency
- Python Concurrency
- What is Threading in Python
- What is the Global Interpreter Lock
- What is Multiprocessing in Python
- When to Use Threads
- Why Not Always Use Multiprocessing
- Lesson 02: Create and Start New Threads
- Main Thread, Helper Thread, and New Threads
- Life-Cycle of a Thread
- How to Run a Function in a New Thread
- How to Extend the Thread Class
- Thread-Local Storage
- Lesson 03: Configuring and Interacting with Threads
- How to Configure New Threads
- How to Query the Status of a Thread
- How to Get the Current, Main, and All Thread
- ho two Handle Unexpected Exceptions in New Threads
- Lesson 04: Synchronize and Coordinate Threads
- How to Protect Critical Section with a Mutex Lock
- How to Limit Access to a Resource with a Semaphore
- How to Signal Between Threads Using an Event
- How to Coordinate Using a Condition Variable
- How to Coordinate Thread with a Barrier
- Lesson 05: Share Data Between Threads
- Share Data Using Global Variables
- Return Values From Threads via Instance Variables
- Share Data Between Threads with Queues
- Lesson 06: Run Tasks with Reusable Workers in Thread Pools
- What Are Thread Pool
- How to Use Thread Pools in Python
- How to Configure the ThreadPool Class
- How to Execute Tasks Synchronously and Asynchronously
- How to Use Callbacks to Handle Results and Errors
- How to Interact With Asynchronous Tasks
- Lesson 07: Close, Stop, and Kill Threads
- Alternatives to Stopping a Thread
- How a Thread Can Close Itself
- How to Gracefully Stop a Thread
- How to Forcefully Kill a Thread
Next, let’s look at the structure of each lesson.
Highly-structured lessons on how you can get results
Each lesson has two main parts, they are:
- 1) The lesson body that teaches the concepts and working code examples.
- 2) The lesson review that summarizes what was learned with exercises.
The body of the lesson will introduce a topic with code examples, whereas the lesson overview will review what was learned with exercises and links for further information.
Each lesson has a specific learning outcome and is designed to be completed in about one hour.
Each lesson is also designed to be self-contained so that you can read the lessons out of order if you choose, such as dipping into topics in the future to solve specific programming problems.
We Python developers learn best from real and working code examples. So each lesson has multiple large worked examples with sample output.
Next, let’s look at what you will know after finishing the book.
Your Learning Outcomes
Transform from “python developer” into “Python developer that can create scalable and concurrent programs using multithreaded programming“
After working through all of the lessons in this book, you will know:
- The difference between thread-based and process-based concurrency and the types of tasks that are well suited to the capabilities of the threading module.
- How to execute your own ad hoc functions concurrently using the Thread class.
- How to identify the main thread and the life-cycle of new threads.
- How to configure a new thread and get access to all running threads and query their status.
- How to coordinate and synchronize threads using mutex locks, semaphores, condition variables and the full suite of concurrency primitives.
- How to share data between threads using shared global variables and how to return values from new threads using instance variables.
- How to send and receive data between threads using thread-safe queues.
- How to create and configure thread pools to execute ad hoc tasks using reusable worker threads.
- How to handle results and errors and query the status of asynchronous tasks executed in thread pools.
- How to close threads, and how to trigger a graceful shutdown or forcefully kill new threads.
You will learn from code examples, not pages and pages of fluff.
Get your copy now:
100% Money-Back Guarantee
(no questions asked)
I want you to actually learn Python threading so well that you can confidently use it on current and future projects.
I designed my book to read just like I’m sitting beside you, showing you how.
I want you to be happy. I want you to win!
I stand behind all of my materials. I know they get results and I’m proud of them.
Nevertheless, if you decide that my books are not a good fit for you, I’ll understand.
I offer a 100% money-back guarantee, no questions asked.
To get a refund, contact me with your purchase name and email address.
Frequently Asked Questions
This section covers some frequently asked questions.
If you have any questions. Contact me directly. Any time about anything. I will do my best to help.
What are your prerequisites?
This book is designed for Python developers who want to discover how to use and get the most out of Python threading.
Specifically, this book is for:
- Developers that can write simple Python programs.
- Developers that need better performance from current or future Python programs.
- Developers that want to learn concurrent programming in Python.
This book does not require that you are an expert in the Python programming language or concurrency.
- You do not need to be an expert Python developer.
- You do not need to be an expert in concurrency.
What version of python do you need?
All code examples use Python 3.
Python 3.9+ to be exact.
Python 2.7 is not supported because it reached its “end of life” in 2020.
Are there code examples?
There are 26 .py code files.
Each lesson has one or more complete, standalone, and fully-working code examples.
The book is provided in a .zip file that includes a src/ directory containing all source code files used in the book.
How long will the book take you to finish?
About 1 hour per lesson.
Work at your own pace.
I recommend 1 lesson per day, over 7 days (1 week).
There’s no rush and I recommend that you take your time.
The book is designed to be read linearly from start to finish, guiding you from being a Python developer at the start of the book to being a Python developer that can confidently use Python threading in your projects by the end of the book.
In order to avoid overload, I recommend completing one lesson per day, such as in the evening or during your lunch break. This will allow you to complete the transformation in about one week.
I recommend you maintain a directory with all of the code you type from the lessons in the book. This will allow you to use the directory as your own private code library, allowing you to copy-paste code into your projects in the future.
I recommend trying to adapt and extend the examples in the lessons. Play with them. Break them. This will help you learn more about how the API works and why we follow specific usage patterns.
What format is the book?
You can read the book on your screen, next to your editor.
You can also read the book on your tablet, away from your workstation.
The ebook is provided in 2 formats:
- PDF (.pdf): perfect for reading on the screen or tablet.
- EPUB (.epub): perfect for reading on a tablet with a Kindle or iBooks app.
Many developers like to read technical books on their tablet or iPad.
How can you get more help?
The lessons in this book were designed to be easy to read and follow.
Nevertheless, sometimes we need a little extra help.
A list of further reading resources is provided at the end of each lesson. These can be helpful if you are interested in learning more about the topic covered, such as fine-grained details of the standard library and API functions used.
The conclusions at the end of the book provide a complete list of websites and books that can help if you want to learn more about Python concurrency and the relevant parts of the Python standard library. It also lists places where you can go online and ask questions about Python concurrency.
Finally, if you ever have questions about the lessons or code in this book, you can contact me any time and I will do my best to help. My contact details are provided at the end of the book.
How many pages is the book?
The PDF is 106 US letter-sized pages.
Can you print the book?
Although, I think it’s better working through it on the screen.
- You can search, skip, and jump around really fast.
- You can copy-and-paste code examples.
- You can compare code output directly.
Is there digital rights management?
The ebooks have no DRM.
Do you get free updates?
Each time I release an updated version, I will send you an email with a link so that you can download the latest version for free.
You can also email me any time and I will send you the latest version.
Can you buy the book elsewhere?
You can get a kindle or paperback version from Amazon.
Many developers prefer to buy from the kindle store on amazon directly.
In fact, the book was the #1 new release in its category shortly after its release.
Can you get a paperback version of the book?
You can get a paperback version from Amazon.
Below is a photo of me holding the paperback edition of the book.
Can you read a sample of the book?
You can read a book sample via google books “preview” or via the amazon “look inside” feature:
Generally, if you like my writing style on SuperFastPython, then you will like the books.
Can you download the source code now?
The source code (.py) files are included in the .zip with the book.
Nevertheless, you can also download all of the code from the dedicated GitHub Project:
Does threading work on your operating system?
Python concurrency is built into the Python programming language and works equally well on:
Does threading work on your hardware?
Python concurrency is agnostic to the underlying CPU hardware.
If you are running Python on a modern computer, then you will have support for concurrency, e.g. Intel, AMD, ARM, and Apple Silicon CPUs are supported.
How does the “Jump-Start Book” compare to the “Complete Guides“?
The SuperFastPython Jump-Start books are laser-focused on making you productive with a Python concurrency module or class as fast as possible.
This means that many broader topics are not covered because they are not on the critical path.
The guides on SuperFastPython.com are broader in scope and compare the class or module to related modules, describing best practices, common errors, common usage questions, and common objects.
This material may be interesting but is a distraction when you are focused on getting productive as fast as possible.
Another important difference is that the jump-start books are provided in book form, whereas the guides are very long web pages.
This makes the books easy to read on a kindle, tablet, or paperback, as well as the screen, whereas the guides must be read in the browser.
About the Author
Hi, I’m Jason Brownlee, Ph.D.
I’m a Python developer, husband, and father to two boys.
I want to share something with you.
I am obsessed with Python concurrency, but I wasn’t always this way.
My background is in Artificial Intelligence and I have a few fancy degrees and past job titles to prove it.
You can see my LinkedIn profile here:
- Jason Brownlee LinkedIn Profile
(follow me if you like)
Well, AI and machine learning has been hot for the last decade. I have spent that time as a Python machine learning developer:
- Working on a range of predictive modeling projects.
- Writing more than 1,000+ tutorials.
- Authoring over 20+ books.
There’s one thing about machine learning in Python, your code must be fast.
Modeling code is already generally fast, built on top of C and Fortran code libraries.
But you know how it is on real projects…
You always have to glue bits together, wrap the fast code and run it many times, and so on.
Making code run fast requires Python concurrency and I have spent most of the last decade using all the different types of Python concurrency available.
Including threading, multiprocessing, asyncio, and the suite of popular libraries.
I know my way around Python concurrency and I am deeply frustrated at the bad wrap it has.
This is why I started SuperFastPython.com where you can find hundreds of free tutorials on Python concurrency.
And this is why I wrote this book.
Praise for Super Fast Python
Python developers write to me all the time and let me know how helpful my tutorials and books have been.
Below are some select examples posted to LinkedIn.
What Are You Waiting For?
Stop reading outdated StackOverflow answers.
Learn Python concurrency correctly, step-by-step.
Buy now and get your copy in seconds!
(also available from Amazon and GooglePlay stores)