windows.rst 14.9 KB
Newer Older
1 2 3
===============
PyPy on Windows
===============
4

Armin Rigo's avatar
Typo  
Armin Rigo committed
5
PyPy is supported on Windows platforms, starting with Windows 2000.
6 7
The following text gives some hints about how to translate the PyPy
interpreter.
8

9 10 11 12
PyPy supports only being translated as a 32bit program, even on
64bit Windows.  See at the end of this page for what is missing
for a full 64bit translation.

13 14 15 16 17 18 19 20
To build pypy-c you need a working python environment, and a C compiler.
It is possible to translate with a CPython 2.6 or later, but this is not
the preferred way, because it will take a lot longer to run – depending
on your architecture, between two and three times as long. So head to 
`our downloads`_ and get the latest stable version.

Microsoft Visual Studio is preferred as a compiler, but there are reports 
of success with the mingw32 port of gcc.
21 22


23 24
Translating PyPy with Visual Studio
-----------------------------------
25

26 27
We routinely test the `RPython translation toolchain`_ using 
Visual Studio 2008, Express
28
Edition.  Other configurations may work as well.
29

30
The translation scripts will set up the appropriate environment variables
31 32
for the compiler, so you do not need to run vcvars before translation.  
They will attempt to locate the same compiler version that
33 34 35 36
was used to build the Python interpreter doing the
translation.  Failing that, they will pick the most recent Visual Studio
compiler they can find.  In addition, the target architecture
(32 bits, 64 bits) is automatically selected.  A 32 bit build can only be built
37 38
using a 32 bit Python and vice versa. By default pypy is built using the 
Multi-threaded DLL (/MD) runtime environment.
39 40

**Note:** PyPy is currently not supported for 64 bit Windows, and translation
41
will fail in this case.
42

43
Python and a C compiler are all you need to build pypy, but it will miss some
44 45
modules that relies on third-party libraries.  See below how to get
and build them.
46

47 48 49 50 51 52 53 54 55 56
Please see the `non-windows instructions`_ for more information, especially note
that translation is RAM-hungry. A standard translation requires around 4GB, so
special preparations are necessary, or you may want to use the method in the
notes of the `build instructions`_ to reduce memory usage at the price of a
slower translation::

    set PYPY_GC_MAX_DELTA=200MB
    pypy --jit loop_longevity=300 ../../rpython/bin/rpython -Ojit targetpypystandalone
    set PYPY_GC_MAX_DELTA=

57 58 59
Preping Windows for the Large Build
-----------------------------------

60 61 62 63 64 65 66 67 68 69
Normally 32bit programs are limited to 2GB of memory on Windows. It is
possible to raise this limit, to 3GB on Windows 32bit, and almost 4GB
on Windows 64bit.

On Windows 32bit, it is necessary to modify the system: follow
http://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240617
to enable the "3GB" feature, and reboot. This step is not necessary on
Windows 64bit.

Then you need to execute::
70

71
    editbin /largeaddressaware translator.exe
72

73 74
where ``translator.exe`` is the pypy.exe or cpython.exe you will use to 
translate with. 
75

76 77
Installing external packages
----------------------------
78

79
On Windows, there is no standard place where to download, build and
80
install third-party libraries.  We recommend installing them in the parent
81 82
directory of the pypy checkout.  For example, if you installed pypy in
``d:\pypy\trunk\`` (This directory contains a README file), the base
83 84 85 86
directory is ``d:\pypy``. You must then set the
INCLUDE, LIB and PATH (for DLLs) environment variables appropriately.

Abridged method (for -Ojit builds using Visual Studio 2008)
87
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
88 89 90 91 92
Download the versions of all the external packages
from 
https://bitbucket.org/pypy/pypy/downloads/local.zip
Then expand it into the base directory (base_dir) and modify your environment to reflect this::

93 94 95
    set PATH=<base_dir>\bin;<base_dir>\tcltk\bin;%PATH%
    set INCLUDE=<base_dir>\include;<base_dir>\tcltk\include;%INCLUDE%
    set LIB=<base_dir>\lib;<base_dir>\tcltk\lib;%LIB%
96

97
Now you should be good to go. Read on for more information. 
98

99 100
The Boehm garbage collector
~~~~~~~~~~~~~~~~~~~~~~~~~~~
101

102 103 104 105
This library is needed if you plan to use the ``--gc=boehm`` translation
option (this is the default at some optimization levels like ``-O1``,
but unneeded for high-performance translations like ``-O2``).
You may get it at
106
http://hboehm.info/gc/gc_source/gc-7.1.tar.gz
107

108
Versions 7.0 and 7.1 are known to work; the 6.x series won't work with
109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125
pypy. Unpack this folder in the base directory. 
The default GC_abort(...) function in misc.c will try to open a MessageBox.
You may want to disable this with the following patch::

    --- a/misc.c    Sun Apr 20 14:08:27 2014 +0300
    +++ b/misc.c    Sun Apr 20 14:08:37 2014 +0300
    @@ -1058,7 +1058,7 @@
     #ifndef PCR
      void GC_abort(const char *msg)
       {
       -#   if defined(MSWIN32)
       +#   if 0 && defined(MSWIN32)
              (void) MessageBoxA(NULL, msg, "Fatal error in gc", MB_ICONERROR|MB_OK);
               #   else
                      GC_err_printf("%s\n", msg);
    
Then open a command prompt::
126

127 128 129
    cd gc-7.1
    nmake -f NT_THREADS_MAKEFILE
    copy Release\gc.dll <somewhere in the PATH>
130

131 132
The zlib compression library
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
133

134
Download http://www.gzip.org/zlib/zlib-1.2.3.tar.gz and extract it in
mattip's avatar
mattip committed
135
the base directory.  Then compile as a static library::
136

137 138
    cd zlib-1.2.3
    nmake -f win32\Makefile.msc
mattip's avatar
mattip committed
139 140
    copy zlib1.lib <somewhere in LIB>
    copy zlib.h zconf.h <somewhere in INCLUDE>
141

142 143
The bz2 compression library
~~~~~~~~~~~~~~~~~~~~~~~~~~~
mattip's avatar
mattip committed
144
Get the same version of bz2 used by python and compile as a static library::
145

Matti Picus's avatar
wip  
Matti Picus committed
146 147
    svn export http://svn.python.org/projects/external/bzip2-1.0.6
    cd bzip2-1.0.6
148
    nmake -f makefile.msc
mattip's avatar
mattip committed
149 150 151
    copy libbz2.lib <somewhere in LIB>
    copy bzlib.h <somewhere in INCLUDE>

152
    
153
The sqlite3 database library
154
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
155

156 157 158 159
PyPy uses cffi to interact with sqlite3.dll. Only the dll is needed, the cffi
wrapper is compiled when the module is imported for the first time.
The sqlite3.dll should be version 3.6.21 for CPython2.7 compatablility.

160 161
The expat XML parser
~~~~~~~~~~~~~~~~~~~~
162

163 164
Download the source code of expat on sourceforge:
http://sourceforge.net/projects/expat/ and extract it in the base
165 166
directory.  Version 2.1.0 is known to pass tests. Then open the project 
file ``expat.dsw`` with Visual
167
Studio; follow the instruction for converting the project files,
168 169 170
switch to the "Release" configuration, reconfigure the runtime for 
Multi-threaded DLL (/MD) and build the solution (the ``expat`` project 
is actually enough for pypy).
171

172
Then, copy the file ``win32\bin\release\libexpat.dll`` somewhere in
mattip's avatar
mattip committed
173 174
your PATH, ``win32\bin\release\libexpat.lib`` somewhere in LIB, and
both ``lib\expat.h`` and ``lib\expat_external.h`` somewhere in INCLUDE.
175

176 177
The OpenSSL library
~~~~~~~~~~~~~~~~~~~
178

179 180 181
OpenSSL needs a Perl interpreter to configure its makefile.  You may
use the one distributed by ActiveState, or the one from cygwin.  In
both case the perl interpreter must be found on the PATH.
182

Matti Picus's avatar
wip  
Matti Picus committed
183 184
    svn export http://svn.python.org/projects/external/openssl-0.9.8y
    cd openssl-0.9.8y
185 186
    perl Configure VC-WIN32
    ms\do_ms.bat
187
    nmake -f ms\nt.mak install
188

Matti Picus's avatar
Matti Picus committed
189 190 191
TkInter module support
~~~~~~~~~~~~~~~~~~~~~~

192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208
Note that much of this is taken from the cpython build process.
Tkinter is imported via cffi, so the module is optional. To recreate the tcltk
directory found for the release script, create the dlls, libs, headers and
runtime by running::

	svn export http://svn.python.org/projects/external/tcl-8.5.2.1 tcl85 
	svn export http://svn.python.org/projects/external/tk-8.5.2.0 tk85
	cd tcl85\win 
	nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 DEBUG=0 INSTALLDIR=..\..\tcltk clean all 
	nmake -f makefile.vc DEBUG=0 INSTALLDIR=..\..\tcltk install
	cd ..\..\tk85\win 
	nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl85 clean all 
	nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl85 install

Now you should have a tcktk\bin, tcltk\lib, and tcltk\include directory ready
for use. The release packaging script will pick up the tcltk runtime in the lib
directory and put it in the archive.
Matti Picus's avatar
Matti Picus committed
209

210 211
Using the mingw compiler
------------------------
212

213
You can compile pypy with the mingw compiler, using the --cc=mingw32 option;
214 215 216
gcc.exe must be on the PATH. If the -cc flag does not begin with "ming", it should be
the name of a valid gcc-derivative compiler, i.e. x86_64-w64-mingw32-gcc for the 64 bit
compiler creating a 64 bit target.
217

218 219 220 221
You probably want to set the CPATH, LIBRARY_PATH, and PATH environment variable to
the header files, lib or dlls, and dlls respectively of the locally installed packages 
if they are not in the mingw directory heirarchy. 

mattip's avatar
mattip committed
222
libffi for the mingw compiler
223 224
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

mattip's avatar
mattip committed
225
To enable the _rawffi (and ctypes) module, you need to compile a mingw
226 227
version of libffi.  Here is one way to do this, wich should allow you to try
to build for win64 or win32:
mattip's avatar
mattip committed
228 229 230 231 232 233 234 235 236 237 238 239

#. Download and unzip a `mingw32 build`_ or `mingw64 build`_, say into c:\mingw
#. If you do not use cygwin, you will need msys to provide make, 
   autoconf tools and other goodies.

    #. Download and unzip a `msys for mingw`_, say into c:\msys
    #. Edit the c:\msys\etc\fstab file to mount c:\mingw

#. Download and unzip the `libffi source files`_, and extract
   them in the base directory.  
#. Run c:\msys\msys.bat or a cygwin shell which should make you
   feel better since it is a shell prompt with shell tools.
mattip's avatar
mattip committed
240
#. From inside the shell, cd to the libffi directory and do::
241 242 243 244

    sh ./configure
    make
    cp .libs/libffi-5.dll <somewhere on the PATH>
mattip's avatar
mattip committed
245

246 247 248 249 250 251
If you can't find the dll, and the libtool issued a warning about 
"undefined symbols not allowed", you will need to edit the libffi
Makefile in the toplevel directory. Add the flag -no-undefined to
the definition of libffi_la_LDFLAGS

If you wish to experiment with win64, you must run configure with flags::
mattip's avatar
mattip committed
252

253
    sh ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32
mattip's avatar
mattip committed
254

255 256
or such, depending on your mingw64 download.

Armin Rigo's avatar
typo  
Armin Rigo committed
257
hacking on PyPy with the mingw compiler
mattip's avatar
mattip committed
258
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Armin Rigo's avatar
typo  
Armin Rigo committed
259
Since hacking on PyPy means running tests, you will need a way to specify
260 261
the mingw compiler when hacking (as opposed to translating). As of
March 2012, --cc is not a valid option for pytest.py. However if you set an
262
environment variable CC to the compliter exe, testing will use it.
263

264
.. _`mingw32 build`: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds
265 266
.. _`mingw64 build`: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds
.. _`msys for mingw`: http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29   
267
.. _`libffi source files`: http://sourceware.org/libffi/
Laura Creighton's avatar
Laura Creighton committed
268
.. _`RPython translation toolchain`: translation.html
269 270 271
.. _`our downloads`: http://pypy.org/download.html   
.. _`non-windows instructions`: getting-started-python.html#translating-the-pypy-python-interpreter
.. _`build instructions`: http://pypy.org/download.html#building-from-source
272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288

What is missing for a full 64-bit translation
---------------------------------------------

The main blocker is that we assume that the integer type of RPython is
large enough to (occasionally) contain a pointer value cast to an
integer.  The simplest fix is to make sure that it is so, but it will
give the following incompatibility between CPython and PyPy on Win64:
  
CPython: ``sys.maxint == 2**32-1, sys.maxsize == 2**64-1``

PyPy: ``sys.maxint == sys.maxsize == 2**64-1``

...and, correspondingly, PyPy supports ints up to the larger value of
sys.maxint before they are converted to ``long``.  The first decision
that someone needs to make is if this incompatibility is reasonable.

289 290 291 292 293 294 295 296 297 298
Assuming that it is, the first thing to do is probably to hack *CPython*
until it fits this model: replace the field in PyIntObject with a ``long
long`` field, and change the value of ``sys.maxint``.  This might just
work, even if half-brokenly: I'm sure you can crash it because of the
precision loss that undoubtedly occurs everywhere, but try not to. :-)

Such a hacked CPython is what you'll use in the next steps.  We'll call
it CPython64/64.

It is probably not too much work if the goal is only to get a translated
299
PyPy executable, and to run all tests before translation.  But you need
300 301 302 303 304 305 306
to start somewhere, and you should start with some tests in
rpython/translator/c/test/, like ``test_standalone.py`` and
``test_newgc.py``: try to have them pass on top of CPython64/64.

Keep in mind that this runs small translations, and some details may go
wrong.  The most obvious one is to check that it produces C files that
use the integer type ``Signed`` --- but what is ``Signed`` defined to?
307
It should be equal to ``long`` on every other platform, but on Win64 it
308 309 310
should be something like ``long long``.

What is more generally needed is to review all the C files in
311
rpython/translator/c/src for the word ``long``, because this means a
312 313 314
32-bit integer even on Win64.  Replace it with ``Signed`` most of the
times.  You can replace one with the other without breaking anything on
any other platform, so feel free to.
315

316 317
Then, these two C types have corresponding RPython types: ``rffi.LONG``
and ``lltype.Signed`` respectively.  The first should really correspond
318
to the C ``long``.  Add tests that check that integers cast to one
319
type or the other really have 32 and 64 bits respectively, on Win64.
320

321
Once these basic tests work, you need to review ``rpython/rlib/`` for
322
uses of ``rffi.LONG`` versus ``lltype.Signed``.  The goal would be to
323 324 325 326 327 328 329 330 331
fix some more ``LONG-versus-Signed`` issues, by fixing the tests --- as
always run on top of CPython64/64.  Note that there was some early work
done in ``rpython/rlib/rarithmetic`` with the goal of running all the
tests on Win64 on the regular CPython, but I think by now that it's a
bad idea.  Look only at CPython64/64.

The major intermediate goal is to get a translation of PyPy with ``-O2``
with a minimal set of modules, starting with ``--no-allworkingmodules``;
you need to use CPython64/64 to run this translation too.  Check
332 333 334
carefully the warnings of the C compiler at the end. By default, MSVC
reports a lot of mismatches of integer sizes as warnings instead of
errors.
335

336 337 338
Then you need to review ``pypy/module/*/`` for ``LONG-versus-Signed``
issues.  At some time during this review, we get a working translated
PyPy on Windows 64 that includes all ``--translationmodules``, i.e.
339
everything needed to run translations.  Once we have that, the hacked
340 341 342 343 344 345 346
CPython64/64 becomes much less important, because we can run future
translations on top of this translated PyPy.  As soon as we get there,
please *distribute* the translated PyPy.  It's an essential component
for anyone else that wants to work on Win64!  We end up with a strange
kind of dependency --- we need a translated PyPy in order to translate a
PyPy ---, but I believe it's ok here, as Windows executables are
supposed to never be broken by newer versions of Windows.
347 348

Happy hacking :-)