Changing CPython’s Grammar#
There’s more to changing Python’s grammar than editing
Grammar/python.gram. Here’s a checklist.
These instructions are for Python 3.9 and beyond. Earlier versions use a different parser technology. You probably shouldn’t try to change the grammar of earlier Python versions, but if you really want to, use GitHub to track down the earlier version of this file in the devguide.
For more information on how to use the new parser, check the section on how to use CPython’s parser.
Note: sometimes things mysteriously don’t work. Before giving up, try
Grammar/python.gram: The grammar, with actions that build AST nodes. After changing it, run
build.bat --regenon Windows), to regenerate
Parser/parser.c. (This runs Python’s parser generator,
Grammar/Tokensis a place for adding new token types. After changing it, run
make regen-tokento regenerate
Doc/library/token-list.inc. If you change both
make regen-pegen. On Windows,
build.bat --regenwill regenerate both at the same time.
Parser/Python.asdlmay need changes to match the grammar. Then run
make regen-astto regenerate
Parser/tokenizer.ccontains the tokenization code. This is where you would add a new type of comment or string literal, for example.
Python/ast.cwill need changes to validate AST objects involved with the grammar change.
Python/ast_unparse.cwill need changes to unparse AST objects involved with the grammar change (“unparsing” is used to turn annotations into strings per PEP 563).
The Design of CPython’s Compiler has its own page.
Lib/ast.pyfile may need changes to accommodate any modifications in the AST nodes.
Doc/library/ast.rstmay need to be updated to reflect changes to AST nodes.
Add some usage of your new syntax to
Certain changes may require tweaks to the library module
Lib/tokenize.pyneeds changes to match changes to the tokenizer.
Documentation must be written! Specifically, one or more of the pages in
Doc/reference/will need to be updated.