Python

Summary of Major Changes Between Python Versions

Nicholas Hairs

Feb 2, 2024 • 9 min read

This post is designed to be a quick reference for the major changes introduced with each new version of Python. This can help with taking advantages of using new features as you upgrade your code base, or ensuring that you have the correct guards for compatibility with older versions.

There are two sections to this post: the first covers the actual changes, the second useful tools, links, and utilities that can aid with upgrading code bases.

Versions

In this section I've documented the major changes to the Python syntax and standard library. Except for the typing module I've mostly excluded changes to modules. I have not included any changes to the C-API, byte-code, or other low level parts.

For each section the end-of-life date (EOL) refers to the date at which the Python Software Foundation will not longer provide security patches for a particular version.

Python 3.7 and earlier

This section has been combined as all these versions are already EOL at the time of writing, but if you've been programming in Python for a while you may have forgotten about when these features were introduced.

async and await (3.5+)
matrix operator: a @ b (3.5+)
type hints (3.5+)
Formatted String Literals (aka f-strings) f"{something}" (3.6+)
underscore in numeric literals 1_000_000 (3.6+)
dictionaries are guaranteed insertion ordered (3.7+)
contextvars (3.7+)
dataclasses (3.7+)
importlib.resources (3.7+)

Python 3.8 (EOL Oct 2024)

Assignment expressions

Also known as the Walrus operator

if (thing := get_thing()) is not None:
  do_something(thing)
else:
  raise Exception(f"Something is wrong with {thing}")

Positional only parameters

def foo(a, b, /, c, d, *, e, f):
  # a, b: positional only
  # c, d: positional or keyword
  # e, f: keyword only

Self documenting f-strings

# Before
f"user={user}"

# Now
f"{user=}"

Importlib Metadata

import importlib.metadata
importlib.metadata.version("some-library")
# "2.3.4"
importlib.metadata.requires("some-library")
# ["thing==1.2.4", "other>=5"]
importlib.metadata.files("some-library")
# [...]

Typing: `TypedDict`, `Literal`, `Final`, `Protocol`

TypedDict - PEP 589
Literal - PEP 586
Final - PEP 591
Protocol - PEP 544

Python 3.9 (EOL Oct 2025)

Typing: Builtin Generics

Can now use dict[...], list[...], set[...] etc instead of using typing.Dict, List, Set.

Remove Prefix/Suffix

Strings and similar types can now use removeprefix and removesuffix to more safely remove things from the start or end. This is safer than string slicing methods which rely on correctly counting the length of the prefix (and remembering to change the slice if the prefix changes).

if header.startswith("X-Forwarded-"):
  section = header.removeprefix("X-Forwarded-")

Dict Union Operator (PEP 584)

combined_dict = dict_one | dict_two
updated_dict |= dict_three

Annotations (PEP 593)

my_int: Annotated[int, SomeRange(0, 255)] = 0

Zoneinfo (PEP 615)

IANA Time Zone Database is now part of standard library

import zoneinfo
some_zone = zoneinfo.ZoneInfo("Europe/Berlin")

Python 3.10 (EOL Oct 2026)

Structural Pattern Matching (PEP 634, PEP 635, PEP 636)

See change log for more examples.

match command.split():
  case ["quit"]:
    print("Goodbye!")
    quit_game()
  case ["look"]:
    current_room.describe()
  case ["get", obj]:
    character.get(obj, current_room)
  case ["go", direction]:
    current_room = current_room.neighbor(direction)
  case [action]:
    ... # interpret single-verb action
  case [action, obj]:
    ... # interpret action, obj
  case _:
    ... # anything that didn't match

Typing: Union using pipe

# Before
from typing import Optional, Union
thing: Optional[Union[str, list[str]]] = None

# Now
thing: str | list[str] | None = None

Typing: `ParamSpec` (PEP 612)

Allows for much better passing of typing information when working with Callable and other similar types.

from typing import Awaitable, Callable, ParamSpec, TypeVar

P = ParamSpec("P")
R = TypeVar("R")

def add_logging(f: Callable[P, R]) -> Callable[P, Awaitable[R]]:
  async def inner(*args: P.args, **kwargs: P.kwargs) -> R:
    await log_to_database()
    return f(*args, **kwargs)
  return inner

@add_logging
def takes_int_str(x: int, y: str) -> int:
  return x + 7

await takes_int_str(1, "A") # Accepted
await takes_int_str("B", 2) # Correctly rejected by the type checker

Typing: `TypeAlias` (PEP 613)

StrCache: TypeAlias = 'Cache[str]'  # a type alias
LOG_PREFIX = 'LOG[DEBUG]'  # a module constant

Typing: `TypeGuard` (PEP 647)

_T = TypeVar("_T")

def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]:
  return len(val) == 2

def func(names: Tuple[str, ...]):
  if is_two_element_tuple(names):
    reveal_type(names)  # Tuple[str, str]
  else:
    reveal_type(names)  # Tuple[str, ...]

Parenthesized Context Managers (PEP 617)

with (CtxManager() as example):
  ...

with (
  CtxManager1(), CtxManager2()
):
  ...

with (CtxManager1() as example, CtxManager2()):
  ...

with (CtxManager1(), CtxManager2() as example):
  ...

with (
  CtxManager1() as example1,
  CtxManager2() as example2,
):
  ...

Dataclasses: `slots`, `kw_only`

Dataclass decorator now supports following:

kw_only=True all parameters in __init__ will be marked keyword only.
slots=True the generatred dataclass will use __slots__ for storing data.

Python 3.11 (EOL Oct 2027)

Tomllib

tomllib - Standard library TOML parser

Exception Groups (PEP 654)

PEP 654 introduces language features that enable a program to raise and handle multiple unrelated exceptions simultaneously. The builtin types ExceptionGroup and BaseExceptionGroup make it possible to group exceptions and raise them together, and the new except* syntax generalizes except to match subgroups of exception groups.

Enriching Exceptions with notes (PEP 678)

The add_note() method is added to BaseException. It can be used to enrich exceptions with context information that is not available at the time when the exception is raised. The added notes appear in the default traceback.

try:
  do_something()
except BaseException as e:
  e.add_note("this happened during do_something")
  raise

Typing: `Self` (PEP 673)

class MyClass:
  @classmethod
  def from_hex(cls, s: str) -> Self:  # Self means instance of cls
    return cls(int(s, 16))
        
  def frobble(self, x: int) -> Self: # Self means this instance
    self.y >> x
    return self

Typing: LiteralString (PEP 675)

The new LiteralString annotation may be used to indicate that a function parameter can be of any literal string type. This allows a function to accept arbitrary literal string types, as well as strings created from other literal strings. Type checkers can then enforce that sensitive functions, such as those that execute SQL statements or shell commands, are called only with static arguments, providing protection against injection attacks.

Typing: Marking `TypedDict` entries as [not] required (PEP 655)

# default is required
class Movie(TypedDict):
  title: str
  year: NotRequired[int]

# default is not-required
class Movie(TypedDict, total=False):
  title: Required[str]
  year: int

Typing: Variadic Generics via `TypeVarTuple` (PEP 646)

PEP 484 previously introduced TypeVar, enabling creation of generics parameterised with a single type. PEP 646 adds TypeVarTuple, enabling parameterisation with an arbitrary number of types. In other words, a TypeVarTuple is a variadic type variable, enabling variadic generics.

This enables a wide variety of use cases. In particular, it allows the type of array-like structures in numerical computing libraries such as NumPy and TensorFlow to be parameterised with the array shape. Static type checkers will now be able to catch shape-related bugs in code that uses these libraries.

Typing: `@dataclass_transform` (PEP 681)

dataclass_transform may be used to decorate a class, metaclass, or a function that is itself a decorator. The presence of @dataclass_transform() tells a static type checker that the decorated object performs runtime “magic” that transforms a class, giving it dataclass-like behaviors.

# The create_model decorator is defined by a library.
@typing.dataclass_transform()
def create_model(cls: Type[T]) -> Type[T]:
  cls.__init__ = ...
  cls.__eq__ = ...
  cls.__ne__ = ...
  return cls

# The create_model decorator can now be used to create new model classes:
@create_model
class CustomerModel:
  id: int
  name: str

Star unpacking expressions allowed in `for` statements:

This is officially supported syntax

for x in *a, *b:
  print(x)

Python 3.12 (EOL Oct 2028)

Typing: Type Parameter Syntax (PEP 695)

Compact annotion of generic classes and functions

def max[T](args: Iterable[T]) -> T:
  ...

class list[T]:
  def __getitem__(self, index: int, /) -> T:
    ...

  def append(self, element: T) -> None:
    ...

Ability to declare type aliases using type statement (generates TypeAliasType)

type Point = tuple[float, float]

# Type aliases can also be generic
type Point[T] = tuple[T, T]

F-string changes (PEP 701)

Expression components inside f-strings can now be any valid Python expression, including strings reusing the same quote as the containing f-string, multi-line expressions, comments, backslashes, and unicode escape sequences.

Can re-use quotes (including nesting f-string statements

## Can re-use quotes
f"This is the playlist: {", ".join(songs)}"

f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}" # '2'

## Multiline f-string with comments
f"This is the playlist: {", ".join([
  'Take me back to Eden',  # My, my, those eyes like fire
  'Alkaline',              # Not acid nor alkaline
  'Ascensionism'           # Take to the broken skies at last
])}"

## Backslashes / Unicode
f"This is the playlist: {"\n".join(songs)}"

f"This is the playlist: {"\N{BLACK HEART SUIT}".join(songs)}"

Buffer protocol (PEP 688)

PEP 688 introduces a way to use the buffer protocol from Python code. Classes that implement the __buffer__() method are now usable as buffer types.

The new collections.abc.Buffer ABC provides a standard way to represent buffer objects, for example in type annotations. The new inspect.BufferFlags enum represents the flags that can be used to customize buffer creation.

Typing: `Unpack` for `**kwargs` typing (PEP 692)

from typing import TypedDict, Unpack

class Movie(TypedDict):
  name: str
  year: int

def foo(**kwargs: Unpack[Movie]):
  ...

Typing: `override` decorator (PEP 698)

Ensure's that the method being overridden by a child class actually exists in a parent class.

from typing import override

class Base:
  def get_color(self) -> str:
    return "blue"

class GoodChild(Base):
  @override  # ok: overrides Base.get_color
  def get_color(self) -> str:
    return "yellow"

class BadChild(Base):
  @override  # type checker error: does not override Base.get_color
  def get_colour(self) -> str:
    return "red"

Useful Things

Postponed Annotations (PEP 563)

In newer versions of Python, typing annotations are stored as strings when they are initially parsed. This helps with preventing circular imports, needing to quote references before they are defined, and many other issues. All versions of Python from 3.7 support
from __future__ import annotations
which allows the interpreter to parse using this new format.

Note: PEP 563 has been superseded by PEP 649 which will be implemented in Python 3.13.

Typing Extensions

This library back-ports typing features so that they are available to type checkers inspecting older code bases.

import sys

if sys.version_info < (3, 10):
  from typing_extensions import TypeAlias
else:
  from typing import TypeAlias

Python Support Schedule

To keep track of which versions of Python are support I use the following website:

Python

Check end-of-life, release policy and support schedule for Python.

endoflife.date

Ruff

This is a linter and code formatter written in Rust. It's becoming very popular as it can replace a number of existing tools and is very fast. It also includes the ability to auto-fix errors.

Thus you can combine ruff with it's pyupgrade compatible linter (UP) and then use
ruff check --fix to auto upgrade the code base.

When using pyproject.toml ruff will respect the versions specified by
project.requires-python.

Pyupgrade

This tool can be used to automatically upgrade your code base.

Black

Black is a popular code formatter.

When using pyproject.toml black will respect the versions specified by
project.requires-python.