Authors: Ralf Gommers, Matthew Brett, Nathaniel Smith, Olivier Grisel and Carl Kleffner
Grant objective: provide a modern, freely available, easy to install compiler toolchain that allows building and distributing the scientific Python stack on Windows.
Problem statement: the situation with building the scientific Python stack on Windows is reaching breaking point. There is no freely available compiler toolchain that can be used to build 64-bit installers for Scipy or any Scipy installers for Python 3.5. There are no Windows wheels for Numpy, Scipy and many other projects on PyPi. Windows issues are a major burden for release managers.
Key benefits for end users expected to be quickly realized after achieving the objective of this grant are:
For developers the key benefits are:
Funding request: We request $5k in total from NumFOCUS, $1k of which to come from Numpy/Scipy funds, as well as endorsement of a request for $5k to the PSF working group for Python in Scientific Computing and Data Analysis.
Python.org Windows installers are built with specific versions of Microsoft Visual C++ (MSVC). When we build extension modules with MSVC, we need the specific version of MSVC with which Python was compiled. Thus building extensions for each of Python 2.7, 3.4 and 3.5 needs a different copy of MSVC installed and configured  . Installation and configuration is not always easy. Although it is possible to find freely available versions of MSVC for these Pythons, there is no freely available Fortran compiler, making it impossible to build Scipy and other packages depending on Fortran code.
For these reasons, Numpy and Scipy have been using a Windows GCC toolchain based on MinGW for their public releases. However, the MinGW project is now dormant and the toolchain is aging. Symptoms of this aging are that it does not support 64-bit compilation or versions of Fortran beyond f77.
The obvious replacement for MinGW is the newer MinGW-w64 project. This does support 64-bit compilation and modern GFortran. However, there are problems to overcome in configuring MinGW-w64 to compile extensions with acceptable mathematical precision and compatibility with code compiled with MSVC.
Carl Kleffner has been working on solving these and other problems for the last 2 years . His current MinGW-w64-based toolchain is capable of building the core packages of the Scipy stack (see , ); there are rough edges however. The aim of this proposal is to document this toolchain, and make it user-friendly and maintainable, in order for the scientific Python community to be able to switch to it for the Windows binaries of the releases of its core projects. There is significant work involved in achieving this user-friendliness and maintainability to the level required for acceptance by the core Scipy Stack projects. Furthermore, significant work is needed to support MSVC 2015 - needed to support Python 3.5 and up. This effort requires funding, to allow Carl Kleffner (the proposed contractor) to spend more time on it than only spare hours in the evening.
The overall funding request for the planned work is split in 3 phases, for a total amount of $10000. The request for the first phase is $1000; for the second phase $4000 and for the third phase $5000. There are several reasons for this phasing:
As stated above, the request to NumFOCUS now is to approve the phase 1 request of $1000 and the phase 2 request of $4000. In light of the significance of this work for the whole Scipy Stack, the Numpy and Scipy projects are willing to together contribute $1000 of the funds donated to them via NumFOCUS, which covers phase 1.
The proposal is to execute the work on a contractor basis, with a set of milestones to be achieved for each tranche of funds. The contractor will be Carl Kleffner. The other people involved in the work, to support Carl and act as community liaisons, are:
Main milestones for phase 1:
For phase 2:
And for phase 3:
The current 32-bit Windows installers for Numpy and Scipy are built with MinGW 3.4.5 (see  for details). They’re compiled against three variants of ATLAS, which differ in the instruction sets they support (no SSE, SS2 or SSE3). The installers are .exe files with a GUI, and therefore cannot be used in batch installed. Building wheels for these ATLAS based builds isn’t possible; the wheel format doesn’t support selecting the right instruction set at install time. Having no Windows wheels for Numpy and Scipy on PyPy has also blocked many other projects from providing Windows wheels. The difficulty of building Windows installers is a major burden on release managers for scientific Python projects.
Development of core libraries like NumPy and SciPy is hobbled by the need to work around bugs and limitations of old compilers (e.g., lack of support for C99 and Fortran 95). Issues with the current outdated MinGW setup are the main bottleneck in making releases, and regularly lead to delays of a month or more. Windows CI isn’t available, therefore build issues and test failures are usually detected too late.
Another important issue is that Cython, and related technologies that depend on a working compiler, is hard to use in introductory classes because setting up the Microsoft Visual Studio compilers is difficult and error-prone.
The target for supported Python versions is 2.7 and 3.5. Python 2.7 support must work but is not required to be pushed to upstream MinGW-w64. For Python 3.5 (and up), upstream MinGW-w64 support for the universal common runtime in MSVC 2015 is necessary in order to have a situation that’s maintainable longer term.
There are a number of benefits in addition to the ones mentioned in the executive summary. These are:
GCC as well as the all necessary libraries for a tool chain can be compiled with the -disabled-shared option at configure time. The GCC runtime object code will be statically linked into the binaries. As a consequence no dependencies to external GCC runtime libraries will be created. Deployment of Python extensions is therefore greatly improved.
Two alternative options to improve the build situation on Windows were considered:
Option (1) is potentially easier to maintain than the proposed work (see  for a report of some experimentation in this direction done in 2009). However, this would rely on MSVC remaining freely available, which is not a given. It also would not provide all the benefits that this proposal brings: the toolchain wouldn’t be pip-installable, and it wouldn’t support C99. Finally: no one is working on this, while significant work has already been done on the MinGW-w64 toolchain.
Option (2) has been explored, but isn’t feasible. MKL is now free, but ifort (the Intel Fortran compiler) not yet. So that’s still not a free toolchain. In addition, there are legal issues with distributing the binaries even if ifort would become free. There has been contact with Intel; our conclusion was that there is little chance that a free solution will be made available by Intel in the near future. (In fact, our contacts at Intel have encouraged us to move ahead with the MinGW-w64 solution, and helped us locate BSD-licensed code to improve the numerical precision issues that was previously contributed by Intel to Android.)
There are several mutually incompatible 64-bit MinGW-based toolchains being distributed. The most active and promising project is MinGW-w64 . Therefore the static toolchain will be based on this project.
The static toolchains (separate ones for 32-bit and 64-bit) have to be built on Windows, in the MSYS2 Posix shell. Cross-compiling on LINUX is under examination, but isn’t a priority.
Customizations not present in the standard MinGW-w64 releases include: