Last Updated on November 5, 2023

The GIL has been removed from Python!

Has it though?

Okay, let me clear some things up:

1. Yes, the GIL is probably going away (eventually).
2. No, the GIL has not gone yet.
3. Yes, we still need to care a lot about the GIL today and for years to come.
4. Yes, GIL or no-GIL, we need to know how to use the threading module.

Okay, allow me to elaborate.

Let’s get started.

Python Global Interpreter Lock (GIL)

The internals of the default Python interpreter (called CPython) are not thread-safe.

This means that there can be race conditions between multiple threads within a single Python process, potentially resulting in unexpected behavior and corrupt data.

As such, the Python interpreter makes use of a Global Interpreter Lock, or GIL for short, to make instructions executed by the Python interpreter (called Python bytecodes) thread-safe.

The GIL is a programming pattern used in the reference Python interpreter, although similar locks exist in other interpreted languages, such as Ruby. It is a lock in the sense that it uses a synchronization primitive called a mutual exclusion lock or mutex lock to ensure that only one thread of execution can execute instructions at a time within a Python process.

In CPython, the global interpreter lock, or GIL, is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. The GIL prevents race conditions and ensures thread safety.

— GLOBAL INTERPRETER LOCK, PYTHON WIKI.

The effect of the GIL is that whenever a thread within a Python program wants to run, it must acquire the lock before executing. This is not a problem for most Python programs that have a single thread of execution, called the main thread.

It can become a problem in multi-threaded Python programs, such as programs that make use of the threading.Thread class, the multiprocessing.pool.ThreadPool class, or the concurrent.futures.ThreadPoolExecutor class.

The lock is explicitly released and re-acquired periodically by each Python thread, specifically after approximately every 100 bytecode instructions executed within the interpreter.

This allows other threads within the Python process to run, if present.

The lock is also released in some specific circumstances, allowing other threads to run.

Work With the GIL

Generally, Python developers hate the GIL because it stops threads from achieving full parallelism in many circumstances.

For example :

• Why Do Python Developers Hate The GIL?

A common refrain is to “just use multiprocessing“, which is terrible advice. Terrible!

For example:

• Why Not Always Use Processes in Python

I have many tutorials on how to work with the GIL.

The GIL is released when doing blocking IO. So we can run many blocked threads in parallel for IO-bound operations like reading and writing files and sockets.

• Threading vs Multiprocessing in Python

And many CPU-intensive third-party libraries explicitly release the GIL, like NumPy.

For example:

• NumPy vs the Global Interpreter Lock (GIL)

You do not want to “just use multiprocessing” when throwing large numpy arrays around.

For example:

• Performance Cost of Naive Parallelism in NumPy

But I Thought The GIL Was Removed?

Nevertheless, I get a lot of comments and questions like the following:

• But the GIL has been removed now right?

And:

• Python no longer has a GIL!

And:

• Didn’t you hear, there’s no GIL anymore!!!

Especially in the last few weeks, I’m getting these emails a lot.

No, Python Still Has a GIL

At the time of writing, Python 3.11 has a GIL.

Python 3.12 will have a GIL.

Millions of Python developers today need to know how to use threads in Python using the current and previous versions of Python.

And that’s okay.

The threading module is pretty cool, and the ThreadPool and ThreadPoolExecutors classes are invaluable when used in the right use cases.

Everything’s fine.

But, The GIL is Going Away
(probably, eventually)

Yes. Probably…

There is “PEP 703″ titled: “Making the Global Interpreter Lock Optional in CPython” that provides a pathway to first make the GIL optional, and then remove it completely.

Many attempts to remove the GIL have been tried over the years, but this one looks successful (so far).

It is driven by Sam Gross aka “colesbury” a Facebook/Meta employee tasked with the job of developing a GIL-free version of Python. I think the focus repo is:

• Multithreaded Python without the GIL

The PEP seeks to first make the GIL optional via the –disable-gil command line argument when compiling (not running) the CPython interpreter.

The global interpreter lock will remain the default for CPython builds and python.org downloads. A new build configuration flag, –disable-gil will be added to the configure script that will build CPython with support for running without the global interpreter lock.

— PEP 703″ titled: “Making the Global Interpreter Lock Optional in CPython

The PEP was hacked on/discussed for ages and looked like it was dead.

Thanks for reviewing the PEP. The PEP was posted five months ago, and it has been 20 months since an end-to-end working implementation (that works with a large number of extensions) was discussed on python-dev.
[…]
I (and my funding organization) will have to treat the current decision-in-limbo as a “no” and will be unable to pursue the PEP further.

— Comment, colesbury, PEP 703 — Making the Global Interpreter Lock Optional in CPython

Then, all of a sudden (to laymen devs like me) it was voted on and agreed to by the Python steering committee. And even then its acceptance was provisional.

Thank you, everyone, for responding to the poll on the no-GIL proposal . It’s clear that the overall sentiment is positive, both for the general idea and for PEP 703 specifically. The Steering Council is also largely positive on both. We intend to accept PEP 703, although we’re still working on the acceptance details.

— A Steering Council notice about PEP 703 (Making the Global Interpreter Lock Optional in CPython)

This broke the (Python) internet.

GIL chatter is always good clickbait for Python devs everywhere.

For example:

• HN: Intent to approve PEP 703: making the GIL optional (490+ comments)
• HN: What’s up, Python? The GIL removed, … (290+) comments

We will probably see an optional GIL in Python 3.13 in 2024.

Perhaps Python will be distributed with a GIL and no-GIL CPython interpreter. Cool.

In 2024, CPython 3.13 is released with support for a –disable-gil build time flag. There are two ABIs for CPython, one with the GIL and one without. Extension authors target both ABIs.

— PEP 703 – Making the Global Interpreter Lock Optional in CPython

Some years later, we may see the no-GIL CPython become the default version of Python. Maybe by 2028 or 2030.

After 2–3 releases, (i.e., in 2026–2027), CPython is released with with the GIL controlled by a runtime environment variable or flag. The GIL is enabled by default. There is only a single ABI.
After another 2–3 release (i.e., 2028–2030), CPython switches to the GIL being disabled by default. The GIL can still be enabled at runtime via an environment variable or command line flag.

— PEP 703 – Making the Global Interpreter Lock Optional in CPython

These are provisional plans. Plans can change.

There are other concerns too, once we start disabling the GIL in Python 3.13.

• There may be performance implications. Some of the stdlib and third-party libs may have worse performance with no GIL (e.g. because of the alternate solutions to make the interpreter thread-safe).
• There may be library implications. Some of the stdlib or third-party libs may rely on the presence of the GIL and break in strange ways we can’t think of yet or become unstable.

Also, I’m far from an expert in GIL matters, this is just what has rubbed off on me from reading things.

The Upshot?

It’s fantastic news. I love it.

But, “didn’t you know that the GIL was removed” is somewhere between “misunderstanding” and “dead wrong“.

We still have a GIL and will for years to come.

Regular Python developers everywhere need to care about the GIL. Today. Right now:

1. When saving/loading tons of files.
2. When reading/writing to many blocking sockets.
3. When doing tons of scipy/numpy ops.
4. And so on…

Python developers hacking on legacy codebases will always have to care about the GIL.

GIL or no-GIL, Python developers still need to learn how to drive the threading module well.

Hang on, What About Sub-Interpreters?

Yes, sub-interpreters are coming in Python 3.12 (actually, the Python API is not coming until Python 3.13), and it has implications for concurrency (e.g. per-interpreter GILs).

Sub-interpreters may now be created with a unique GIL per interpreter. This allows Python programs to take full advantage of multiple CPU cores.

— What’s New In Python 3.12

Separate topic. Related, but separate. We’ll still have a GIL (or GILs).

I’m holding off on expounding on this until the release is gold to really dig into them and write a series of tutorials on how to best utilize them.

Stay tuned.

Further Reading

This section provides additional resources that you may find helpful.

Python Threading Books

• Python Threading Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• Threading Module API Cheat Sheet

I also recommend specific chapters in the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Threading: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• threading - Thread-based parallelism
• queue — A synchronized queue class

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know that there are very real plans to remove the GIL, but the effects are still years away.

Did I screw up some details about the GIL or the PEP or something?
I’m a simple humble human. Correct me, please!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Yomex Owo on Unsplash