Commit 34fe02da authored by mattip's avatar mattip

merge default into branch

parents 2c3e7fdc d3bd39b5
......@@ -89,6 +89,7 @@ syntax: regexp
......@@ -23,6 +23,7 @@ and send us feedback!
the pypy-dev team <>
......@@ -19,6 +19,4 @@ installation
``_ explains in details how to use it.
.. _``:
:source:` <ctypes_configure/doc/>` explains in details how to use it.
......@@ -255,10 +255,6 @@ pypy_optiondescription = OptionDescription("objspace", "Object Space Options", [
"special case the 'list[integer]' expressions",
"a shortcut for operations between built-in types. XXX: "
"deprecated, not really a shortcut any more.",
"track types that override __getattribute__",
......@@ -270,9 +266,6 @@ pypy_optiondescription = OptionDescription("objspace", "Object Space Options", [
# weakrefs needed, because of get_subclasses()
requires=[("translation.rweakref", True)]),
ChoiceOption("multimethods", "the multimethod implementation to use",
["doubledispatch", "mrd"],
"track types that override __hash__, __eq__ or __cmp__ and use a special dict strategy for those which do not",
......@@ -64,7 +64,7 @@ def test_check_documentation():
def check_file_exists(fn):
assert configdocdir.join(fn).check()
from pypy.doc.config.confrest import all_optiondescrs
from pypy.doc.config.generate import all_optiondescrs
configdocdir = thisdir.dirpath().dirpath().join("doc", "config")
for descr in all_optiondescrs:
prefix = descr._name
Documentation TODO
* architecture documents don't really show the separation between PyPy and
* architecture.rst is duplicate (both pypy and rpython)
* Consider moving information from pypy/doc/{build,windows}.rst to rpython/doc
* remove documentation on removed features
* various object spaces
* update / remove dead links
* work on configuration/options documentation generation
* Update coding guide
* Move links from project-documentation.rst to index.rst and consider killing
* make translation.rst a high-level overview and move details in their own
* redo various outdated pictures in translation.rst
.. comment: this document is very incomplete, should we generate
it automatically?
The ``__pypy__`` module
The ``__pypy__`` module is the main entry point to special features provided
by PyPy's standard interpreter. Its content depends on `configuration options`_
which may add new functionality and functions whose existence or non-existence
indicates the presence of such features.
by PyPy's standard interpreter. Its content depends on :doc:`configuration options <config/index>`
which may add new functionality and functions whose existence or non-existence
indicates the presence of such features.
.. _`configuration options`: config/index.html
Generally available functionality
- ``internal_repr(obj)``: return the interpreter-level representation of an
......@@ -22,28 +19,22 @@ Generally available functionality
It works like a simplified array of characters (actually, depending on the
configuration the ``array`` module internally uses this).
Transparent Proxy Functionality
If `transparent proxies`_ are enabled (with :config:`objspace.std.withtproxy`)
If :ref:`transparent proxies <tproxy>` are enabled (with :config:`objspace.std.withtproxy`)
the following functions are put into ``__pypy__``:
- ``tproxy(typ, controller)``: Return something that looks like it is of type
typ. Its behaviour is completely controlled by the controller. See the docs
about `transparent proxies`_ for detail.
about :ref:`transparent proxies <tproxy>` for detail.
- ``get_tproxy_controller(obj)``: If obj is really a transparent proxy, return
its controller. Otherwise return None.
.. _`transparent proxies`: objspace-proxies.html#tproxy
Functionality available on (not after translation)
- ``isfake(obj)``: returns True if ``obj`` is faked.
- ``interp_pdb()``: start a pdb at interpreter-level.
This diff is collapsed.
This diff is collapsed.
Building PyPy from Source
For building PyPy, we recommend installing a pre-built PyPy first (see
:doc:`install`). It is possible to build PyPy with CPython, but it will take a
lot longer to run -- depending on your architecture, between two and three
times as long.
Even when using PyPy to build PyPy, translation is time-consuming -- 30
minutes on a fast machine -- and RAM-hungry. You will need **at least** 2 GB
of memory on a 32-bit machine and 4GB on a 64-bit machine.
Clone the repository
If you prefer to compile your own PyPy, or if you want to modify it, you
will need to obtain a copy of the sources. This can be done either by
`downloading them from the download page`_ or by checking them out from the
repository using mercurial. We suggest using mercurial if you want to access
the current development.
.. _downloading them from the download page:
You must issue the following command on your
command line, DOS box, or terminal::
hg clone pypy
This will clone the repository and place it into a directory
named ``pypy``, and will get you the PyPy source in ``pypy/pypy`` and
documentation files in ``pypy/pypy/doc``.
We try to ensure that the tip is always stable, but it might
occasionally be broken. You may want to check out `our nightly tests`_:
find a revision (12-chars alphanumeric string, e.g. "963e808156b3")
that passed at least the
``{linux32}`` tests (corresponding to a ``+`` sign on the
line ``success``) and then, in your cloned repository, switch to this revision
hg up -r XXXXX
where XXXXX is the revision id.
.. _our nightly tests:<trunk>
Install build-time dependencies
To build PyPy on Unix using the C translation backend, you need at least a C
compiler and ``make`` installed. Further, some optional modules have additional
cffi, ctypes
libffi, pkg-config
lzma (PyPy3 only)
libncurses + cffi dependencies from above
Make sure to have these libraries (with development headers) installed before
building PyPy, otherwise the resulting binary will not contain these modules.
On Debian, this is the command to install all build-time dependencies::
apt-get install gcc make libffi-dev pkg-config libz-dev libbz2-dev \
libsqlite3-dev libncurses-dev libexpat1-dev libssl-dev
For the optional lzma module on PyPy3 you will also need ``liblzma-dev``.
On Fedora::
yum install gcc make libffi-devel pkgconfig zlib-devel bzip2-devel \
lib-sqlite3-devel ncurses-devel expat-devel openssl-devel
For the optional lzma module on PyPy3 you will also need ``xz-devel``.
On SLES11::
zypper install gcc make python-devel pkg-config \
zlib-devel libopenssl-devel libbz2-devel sqlite3-devel \
libexpat-devel libffi-devel python-curses
For the optional lzma module on PyPy3 you will also need ``xz-devel``.
On Mac OS X, most of these build-time dependencies are installed alongside
the Developer Tools. However, note that in order for the installation to
find them you may need to run::
xcode-select --install
Run the translation
Translate with JIT::
pypy rpython/bin/rpython --opt=jit pypy/goal/
Translate without JIT::
pypy rpython/bin/rpython --opt=2 pypy/goal/
(You can use ``python`` instead of ``pypy`` here, which will take longer
but works too.)
If everything works correctly this will create an executable ``pypy-c`` in the
current directory. The executable behaves mostly like a normal Python
interpreter (see :doc:`cpython_differences`).
.. _translate-pypy:
Translating with non-standard options
It is possible to have non-standard features enabled for translation,
but they are not really tested any more. Look, for example, at the
:doc:`objspace proxies <objspace-proxies>` document.
PyPy dynamically finds the location of its libraries depending on the location
of the executable. The directory hierarchy of a typical PyPy installation
looks like this::
The hierarchy shown above is relative to a PREFIX directory. PREFIX is
computed by starting from the directory where the executable resides, and
"walking up" the filesystem until we find a directory containing ``lib_pypy``
and ``lib-python/2.7``.
To install PyPy system wide on unix-like systems, it is recommended to put the
whole hierarchy alone (e.g. in ``/opt/pypy``) and put a symlink to the
``pypy`` executable into ``/usr/bin`` or ``/usr/local/bin``.
If the executable fails to find suitable libraries, it will report ``debug:
WARNING: library path not found, using compiled-in sys.path`` and then attempt
to continue normally. If the default path is usable, most code will be fine.
However, the ``sys.prefix`` will be unset and some existing libraries assume
that this is never the case.
.. TODO windows
Old documentation that needs review
.. The following stuff is old (and crufty?), and needs further investigation:
.. doc-index: This needs merging somehow
.. toctree::
This diff is collapsed.
......@@ -40,9 +40,9 @@ if not on_rtd: # only import and set the theme if we're building docs locally
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx',
'sphinx.ext.todo', 'sphinx.ext.ifconfig', 'sphinx.ext.graphviz',
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.extlinks',
'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.ifconfig',
'sphinx.ext.graphviz', 'pypyconfig']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
......@@ -215,7 +215,8 @@ latex_use_modindex = False
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'': None}
intersphinx_mapping = {'python': ('', None),
'rpython': ('', None)}
# -- Options for manpage output-------------------------------------------------
......@@ -225,3 +226,4 @@ man_pages = [
u'The PyPy Project', 1)
extlinks = {'source': ('', '')}
from pypy.doc.confrest import *
from pypy.config.makerestdoc import make_cmdline_overview
from pypy.config import pypyoption
from rpython.config.config import Config
from rpython.config import translationoption
all_optiondescrs = [pypyoption.pypy_optiondescription,
start_to_descr = dict([(descr._name, descr) for descr in all_optiondescrs])
class PyPyPage(PyPyPage):
def fill(self):
super(PyPyPage, self).fill()
self.menubar[:] = html.div(
html.a("general documentation", href="../index.html",
class_="menu"), " ",
html.a("config index", href="index.html",
class_="menu"), " ",
html.a("command-line overview", href="commandline.html",
class_="menu"), " ",
" ", id="menubar")
class Project(Project):
stylesheet = "../style.css"
title = "PyPy Configuration"
prefix_title = "PyPy Configuration"
Page = PyPyPage
def get_content(self, txtpath, encoding):
if txtpath.basename == "commandline.rst":
result = []
for line in
if line.startswith('.. GENERATE:'):
start = line[len('.. GENERATE:'):].strip()
descr = start_to_descr[start]
line = make_cmdline_overview(descr, title=False).text()
return "\n".join(result)
fullpath = txtpath.purebasename
start = fullpath.split(".")[0]
path = fullpath.rsplit(".", 1)[0]
basedescr = start_to_descr.get(start)
if basedescr is None:
if fullpath.count(".") == 0:
descr = basedescr
path = ""
conf = Config(basedescr)
subconf, step = conf._cfgimpl_get_home_by_path(
fullpath.split(".", 1)[1])
descr = getattr(subconf._cfgimpl_descr, step)
text = unicode(descr.make_rest_doc(path).text())
if txtpath.check(file=True):
content =
if content:
text += "\nDescription\n==========="
return u"%s\n\n%s" % (text, unicode(, encoding))
return text
import py
from pypy.config import pypyoption, makerestdoc
from pypy.doc.config.confrest import all_optiondescrs
from rpython.config import translationoption, config
all_optiondescrs = [pypyoption.pypy_optiondescription,
A shortcut speeding up primitive operations between built-in types.
This is a space-time trade-off: at the moment, this option makes a
translated pypy-c executable bigger by about 1.7 MB. (This can probably
be improved with careful analysis.)
Choose the multimethod implementation.
* ``doubledispatch`` turns
a multimethod call into a sequence of normal method calls.
* ``mrd`` uses a technique known as Multiple Row Displacement
which precomputes a few compact tables of numbers and
function pointers.
PyPy's Configuration Handling
......@@ -6,16 +5,17 @@ Due to more and more available configuration options it became quite annoying to
hand the necessary options to where they are actually used and even more
annoying to add new options. To circumvent these problems configuration
management was introduced. There all the necessary options are stored in a
configuration object, which is available nearly everywhere in the `RPython
configuration object, which is available nearly everywhere in the `RPython
toolchain`_ and in the standard interpreter so that adding new options becomes
trivial. Options are organized into a tree. Configuration objects can be
created in different ways, there is support for creating an optparse command
line parser automatically.
_`RPython toolchain`: translation.html
.. _RPython toolchain:
Main Assumption
Configuration objects are produced at the entry points and handed down to
where they are actually used. This keeps configuration local but available
......@@ -24,7 +24,7 @@ command line.
API Details
The handling of options is split into two parts: the description of which
options are available, what their possible values and defaults are and how they
......@@ -40,8 +40,9 @@ The Options are organized in a tree. Every option has a name, as does every
option group. The parts of the full name of the option are separated by dots:
e.g. ``config.translation.thread``.
Description of Options
All the constructors take a ``name`` and a ``doc`` argument as first arguments
to give the option or option group a name and to document it. Most constructors
......@@ -70,6 +71,7 @@ This class is used to group suboptions.
``children`` is a list of option descriptions (including
``OptionDescription`` instances for nested namespaces).
......@@ -81,10 +83,11 @@ Represents a choice out of several objects. The option can also have the value
``requires`` is a dictionary mapping values to lists of of two-element
Represents a choice between ``True`` and ``False``.
Represents a choice between ``True`` and ``False``.
``__init__(self, name, doc, default=None, requires=None, suggests=None, cmdline=DEFAULT, negation=True)``
``default`` specifies the default value of the option. ``requires`` is
......@@ -103,7 +106,6 @@ Represents a choice between ``True`` and ``False``.
Represents a choice of an integer.
``__init__(self, name, doc, default=None, cmdline=DEFAULT)``
......@@ -112,7 +114,6 @@ Represents a choice of an integer.
Represents a choice of a floating point number.
``__init__(self, name, doc, default=None, cmdline=DEFAULT)``
......@@ -121,12 +122,10 @@ Represents a choice of a floating point number.
Represents the choice of a string.
``__init__(self, name, doc, default=None, cmdline=DEFAULT)``
Configuration Objects
``Config`` objects hold the chosen values for the options (of the default,
if no choice was made). A ``Config`` object is described by an
......@@ -164,7 +163,7 @@ Description of the (useful) methods on ``Config``:
Production of optparse Parsers
To produce an optparse parser use the function ``to_optparse``. It will create
an option parser using callbacks in such a way that the config object used for
......@@ -182,14 +181,11 @@ creating the parser is updated automatically.
The usage of config objects in PyPy
The two large parts of PyPy, the Python interpreter and the `RPython
The two large parts of PyPy, the Python interpreter and the RPython
toolchain, have two separate sets of options. The translation toolchain options
can be found on the ``config`` attribute of all ``TranslationContext``
instances and are described in `rpython/config/`_. The interpreter options
instances and are described in :source:`rpython/config/`. The interpreter options
are attached to the object space, also under the name ``config`` and are
described in `pypy/config/`_.
.. include:: _ref.txt
described in :source:`pypy/config/`.
import py
# XXX importing/inheriting from an internal py lib class is hackish
from confrest_oldpy import Project, Page, relpath
html = py.xml.html
class PyPyPage(Page):
googlefragment = """
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "' type='text/javascript'%3E%3C/script%3E"));