Getting Started Developing With PyPy

Using Mercurial

PyPy development is based on Mercurial (hg). If you are not used to version control, the cycle for a new PyPy contributor goes typically like this:

  • Make an account on bitbucket.
  • Go to https://bitbucket.org/pypy/pypy/ and click “fork” (left icons). You get a fork of the repository, e.g. in https://bitbucket.org/yourname/pypy/.
  • Clone this new repo (i.e. the fork) to your local machine with the command hg clone ssh://hg@bitbucket.org/yourname/pypy. It is a very slow operation but only ever needs to be done once. If you already cloned https://bitbucket.org/pypy/pypy before, even if some time ago, then you can reuse the same clone by editing the file .hg/hgrc in your clone to contain the line default = ssh://hg@bitbucket.org/yourname/pypy, and then do hg pull && hg up. If you already have such a clone but don’t want to change it, you can clone that copy with hg clone /path/to/other/copy, and then edit .hg/hgrc as above and do hg pull && hg up.
  • Now you have a complete copy of the PyPy repo. Make a branch with a command like hg branch name_of_your_branch.
  • Edit things. Use hg diff to see what you changed. Use hg add to make Mercurial aware of new files you added, e.g. new test files. Use hg status to see if there are such files. Run tests! (See the rest of this page.)
  • Commit regularly with hg commit. A one-line commit message is fine. We love to have tons of commits; make one as soon as you have some progress, even if it is only some new test that doesn’t pass yet, or fixing things even if not all tests pass. Step by step, you are building the history of your changes, which is the point of a version control system. (There are commands like hg log and hg up that you should read about later, to learn how to navigate this history.)
  • The commits stay on your machine until you do hg push to “push” them back to the repo named in the file .hg/hgrc. Repos are basically just collections of commits (a commit is also called a changeset): there is one repo per url, plus one for each local copy on each local machine. The commands hg push and hg pull copy commits around, with the goal that all repos in question end up with the exact same set of commits. By opposition, hg up only updates the “working copy” by reading the local repository, i.e. it makes the files that you see correspond to the latest (or any other) commit locally present.
  • You should push often; there is no real reason not to. Remember that even if they are pushed, with the setup above, the commits are (1) only in bitbucket.org/yourname/pypy, and (2) in the branch you named. Yes, they are publicly visible, but don’t worry about someone walking around the thousands of repos on bitbucket saying “hah, look at the bad coding style of that guy”. Try to get into the mindset that your work is not secret and it’s fine that way. We might not accept it as is for PyPy, asking you instead to improve some things, but we are not going to judge you.
  • The final step is to open a pull request, so that we know that you’d like to merge that branch back to the original pypy/pypy repo. This can also be done several times if you have interesting intermediate states, but if you get there, then we’re likely to proceed to the next stage, which is...
  • Get a regular account for pushing directly to bitbucket.org/pypy/pypy (just ask and you’ll get it, basically). Once you have it you can rewrite your file .hg/hgrc to contain default = ssh://hg@bitbucket.org/pypy/pypy. Your changes will then be pushed directly to the official repo, but (if you follow these rules) they are still on a branch, and we can still review the branches you want to merge.
  • If you get closer to the regular day-to-day development, you’ll notice that we generally push small changes as one or a few commits directly to the branch default. Also, we often collaborate even if we are on other branches, which do not really “belong” to anyone. At this point you’ll need hg merge and learn how to resolve conflicts that sometimes occur when two people try to push different commits in parallel on the same branch. But it is likely an issue for later :-)

Running PyPy’s unit tests

PyPy development always was and is still thoroughly test-driven. We use the flexible py.test testing tool which you can install independently and use for other projects.

The PyPy source tree comes with an inlined version of py.test which you can invoke by typing:

python pytest.py -h

This is usually equivalent to using an installed version:

py.test -h

If you encounter problems with the installed version make sure you have the correct version installed which you can find out with the --version switch.

Now on to running some tests. PyPy has many different test directories and you can use shell completion to point at directories or files:

py.test pypy/interpreter/test/test_pyframe.py

# or for running tests of a whole subdirectory
py.test pypy/interpreter/

See py.test usage and invocations for some more generic info on how you can run tests.

Beware trying to run “all” pypy tests by pointing to the root directory or even the top level subdirectory pypy. It takes hours and uses huge amounts of RAM and is not recommended.

To run CPython regression tests you can point to the lib-python directory:

py.test lib-python/2.7/test/test_datetime.py

This will usually take a long time because this will run the PyPy Python interpreter on top of CPython. On the plus side, it’s usually still faster than doing a full translation and running the regression test with the translated PyPy Python interpreter.

Special Introspection Features of the Untranslated Python Interpreter

If you are interested in the inner workings of the PyPy Python interpreter, there are some features of the untranslated Python interpreter that allow you to introspect its internals.

Interpreter-level console

To start interpreting Python with PyPy, install a C compiler that is supported by distutils and use Python 2.7 or greater to run PyPy:

cd pypy
python bin/pyinteractive.py

After a few seconds (remember: this is running on top of CPython), you should be at the PyPy prompt, which is the same as the Python prompt, but with an extra “>”.

If you press <Ctrl-C> on the console you enter the interpreter-level console, a usual CPython console. You can then access internal objects of PyPy (e.g. the object space) and any variables you have created on the PyPy prompt with the prefix w_:

>>>> a = 123
>>>> <Ctrl-C>
*** Entering interpreter-level console ***
>>> w_a
W_IntObject(123)

The mechanism works in both directions. If you define a variable with the w_ prefix on the interpreter-level, you will see it on the app-level:

>>> w_l = space.newlist([space.wrap(1), space.wrap("abc")])
>>> <Ctrl-D>
*** Leaving interpreter-level console ***

KeyboardInterrupt
>>>> l
[1, 'abc']

Note that the prompt of the interpreter-level console is only ‘>>>’ since it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under Linux) or <Ctrl-Z>, <Enter> (under Windows).

Also note that not all modules are available by default in this mode (for example: _continuation needed by greenlet) , you may need to use one of --withmod-... command line options.

You may be interested in reading more about the distinction between interpreter-level and app-level.

pyinteractive.py options

To list the PyPy interpreter command line options, type:

cd pypy
python bin/pyinteractive.py --help

pyinteractive.py supports most of the options that CPython supports too (in addition to a large amount of options that can be used to customize pyinteractive.py). As an example of using PyPy from the command line, you could type:

python pyinteractive.py --withmod-time -c "from test import pystone; pystone.main(10)"

Alternatively, as with regular Python, you can simply give a script name on the command line:

python pyinteractive.py --withmod-time ../../lib-python/2.7/test/pystone.py 10

The --withmod-xxx option enables the built-in module xxx. By default almost none of them are, because initializing them takes time. If you want anyway to enable all built-in modules, you can use --allworkingmodules.

See our configuration sections for details about what all the commandline options do.

Tracing bytecode and operations on objects

You can use a simple tracing mode to monitor the interpretation of bytecodes. To enable it, set __pytrace__ = 1 on the interactive PyPy console:

>>>> __pytrace__ = 1
Tracing enabled
>>>> x = 5
        <module>:           LOAD_CONST    0 (5)
        <module>:           STORE_NAME    0 (x)
        <module>:           LOAD_CONST    1 (None)
        <module>:           RETURN_VALUE    0
>>>> x
        <module>:           LOAD_NAME    0 (x)
        <module>:           PRINT_EXPR    0
5
        <module>:           LOAD_CONST    0 (None)
        <module>:           RETURN_VALUE    0
>>>>

Demos

The example-interpreter repository contains an example interpreter written using the RPython translation toolchain.

Additional Tools for running (and hacking) PyPy

We use some optional tools for developing PyPy. They are not required to run the basic tests or to get an interactive PyPy prompt but they help to understand and debug PyPy especially for the translation process.

py.test and the py lib

The py.test testing tool drives all our testing needs.

We use the py library for filesystem path manipulations, terminal writing, logging and some other support functionality.

You don’t necessarily need to install these two libraries because we also ship them inlined in the PyPy source tree.

Getting involved

PyPy employs an open development process. You are invited to join our pypy-dev mailing list or look at the other contact possibilities. Usually we give out commit rights fairly liberally, so if you want to do something with PyPy, you can become a committer. We also run frequent coding sprints which are separately announced and often happen around Python conferences such as EuroPython or PyCon. Upcoming events are usually announced on the blog.

Where to start reading the sources

PyPy is made from parts that are relatively independent of each other. You should start looking at the part that attracts you most (all paths are relative to the PyPy top level directory). You may look at our directory reference or start off at one of the following points: