anaconda - Compile fortran module with f2py and Python 3.6 on Windows 10

Question

I'm trying (and failing) to compile a fortran module (specifically igrf12.f from the BGS) using f2py and Python 3.6 on Windows 10. Python was installed using Anaconda 4.4.10.

My setup:

• Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 15 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)] on win32
• Windows 10 Enterprise (version 1703)
• NumPy 1.14.0

I followed f2py directions from the SciPy documentation and a very helpful guide from Dr.Michael Hirsch. Dr.Hirsch has created the pyigrf12 module, but installation through pip failed for me, which is what initially sparked my interest in f2py.

I will outline a few of the methods I am using. Regardless of the method, I always begin by creating a *.pyf signature file using f2py igrf12.f -m pyigrf12 -h igrf12.pyf and adding intent(in/out) attributes appropriately.

• Method 1: Use C:MinGW and --compiler=mingw32
• Method 2: Use C:MinGW and --compiler=msvc
• Method 3: use anaconda version of mingw and --compiler=mingw32
• Method 4: use anaconda version of mingw and --compiler=msvc

Method 1:

For background, I have MinGW at C:MinGW and I have added C:MinGWin to my user Path. Unfortunately, I did not install this version of MinGW (I inherited this computer from a colleague), so I do not know where it was sourced. The gcc --version and gfortran --version is 5.3.0.

I run f2py -c igrf12.pyf igrf12.f --compiler=mingw32. This fails with this error message:

Building import library (arch=AMD64):
"C:UsersSholesAppDataLocalContinuumanaconda3libslibpython36.a" 
(from C:UsersSholesAppDataLocalContinuumanaconda3python36.dll)
    objdump.exe: C:UsersSholesAppDataLocalContinuumanaconda3python36.dll: File format not recognized

Traceback (most recent call last):
  File "C:UsersSholesAppDataLocalContinuumanaconda3Scripts\f2py.py", line 28, in <module>
    main()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpyf2pyf2py2e.py", line 648, in main
    run_compile()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpyf2pyf2py2e.py", line 633, in run_compile
    setup(ext_modules=[ext])
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscore.py", line 169, in setup
    return old_setup(**new_attr)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscore.py", line 148, in setup
    dist.run_commands()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 955, in run_commands
    self.run_command(cmd)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 974, in run_command
    cmd_obj.run()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscommanduild.py", line 47, in run
    old_build.run(self)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscommanduild.py", line 135, in run
    self.run_command(cmd_name)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 974, in run_command
    cmd_obj.run()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscommanduild_ext.py", line 117, in run
    force=self.force)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsccompiler.py", line 733, in new_compiler
    compiler = klass(None, dry_run, force)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 104, in __init__
    build_import_library()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 416, in build_import_library
    return _build_import_library_amd64()
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 472, in _build_import_library_amd64
    generate_def(dll_file, def_file)
  File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 302, in generate_def
    raise ValueError("Symbol table not found")
ValueError: Symbol table not found

The issue seems to involve building libpython36.a from python36.dll.

After a quick google search, a suggestion on a github forum for pywafo was to use the msvc compiler instead of mingw32, leading to Method 2.

Method 2: For background, the files related to my msvc are being pulled from C:Program Files (x86)Microsoft Visual Studio2017CommunityVCToolsMSVC14.10.25017inHostX86x64. Not sure if that's helpful.

I run f2py -c igrf12.pyf igrf12.f --compiler=msvc. This produces two files:

• pyigrf12.cp36-win_amd64.pyd in my current working directory and,
• libigrf12.BMWM6WD5Y3O3UTOEQITBXCIICXVMBEZS.gfortran-win_amd64.dll in .UNKNOWN.libs

When I try to import pyigrf12, first I receive ImportError: DLL load failed: The specified module could not be found. Using Dependency Walker, I receive a lot of errors:

but the most obvious one to address is related to the libigrf12.BMWM6WD5Y3O3UTOEQITBXCIICXVMBEZS.gfortran-win_amd64.dll. I move this libigrf12 dll file to my current working directory, alongside pyigrf12.cp36-win_amd64.pyd.

Now when I try import pyigrf12, I receive ImportError: DLL load failed: %1 is not a valid Win32 application.. Some stackoverflow posts seem to indicate that this is a problem with a conflict between a 32-bit dll and 64-bit Python. Can anyone offer insight into this? After more searching, I decided to try to use the anaconda version of mingw and libpython.

Method 3:

I run conda install mingw libpython, and it installs mingw 4.7-1 and libpython 2.1-py36_0. I run f2py -c igrf12.pyf igrf12.f --compiler=mingw32, and it fails with the following error message:

Building msvcr library: 
"C:UsersSholesAppDataLocalContinuumanaconda3libslibvcruntime140.a" 
(from C:UsersSholesAppDataLocalContinuumanaconda3vcruntime140.dll)
    objdump.exe: C:UsersSholesAppDataLocalContinuumanaconda3vcruntime140.dll: File format not recognized
    Traceback (most recent call last):
      File "C:UsersSholesAppDataLocalContinuumanaconda3Scripts\f2py.py", line 28, in <module>
        main()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpyf2pyf2py2e.py", line 648, in main
        run_compile()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpyf2pyf2py2e.py", line 633, in run_compile
        setup(ext_modules=[ext])
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscore.py", line 169, in setup
        return old_setup(**new_attr)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscore.py", line 148, in setup
        dist.run_commands()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 955, in run_commands
        self.run_command(cmd)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 974, in run_command
        cmd_obj.run()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscommanduild.py", line 47, in run
        old_build.run(self)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscommanduild.py", line 135, in run
        self.run_command(cmd_name)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilscmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libdistutilsdist.py", line 974, in run_command
        cmd_obj.run()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilscommanduild_ext.py", line 117, in run
        force=self.force)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsccompiler.py", line 733, in new_compiler
        compiler = klass(None, dry_run, force)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 107, in __init__
        msvcr_success = build_msvcr_library()
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 399, in build_msvcr_library
        generate_def(dll_file, def_file)
      File "C:UsersSholesAppDataLocalContinuumanaconda3libsite-packages
umpydistutilsmingw32ccompiler.py", line 302, in generate_def
        raise ValueError("Symbol table not found")
    ValueError: Symbol table not found

Now the issue seems to relate to building libvcruntime140.a from vcruntime140.dll. Once again the dll file format is not recognized by objdump.exe.

Method 4:

My final attempt was to run f2py -c igrf12.pyf igrf12.f --compiler=msvc with the anaconda version of mingw installed. For this case, gfortran failed with this error:

   C:UsersSholesAppDataLocalContinuumanaconda3Scriptsgfortran.bat -Wall -g -Wall -g -shared 
......AppDataLocalTempmpugo__0q9Releaseigrf12.o -Lc:userssholesappdatalocalcontinuumanaconda3mingwlibgccx86_64-w64-mingw324.7.0 -LC:UsersSholesAppDataLocalContinuumanaconda3libs -LC:UsersSholesAppDataLocalContinuumanaconda3PCbuildamd64 -o C:UsersSholesAppDataLocalTempmpugo__0q9Releaseextra-dlllibigrf12.75XJA5DX6DTO7YIZ7X6ZHJYTRDCCYQYR.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,C:UsersSholesAppDataLocalTempmpugo__0q9Releaselibigrf12.75XJA5DX6DTO7YIZ7X6ZHJYTRDCCYQYR.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64

gfortran.exe: error: unrecognized command line option '-mlong-double-64'

At this point, I just want to know if it is possible to create fortran extensions with my set up and f2py. I don't have any background compiling C or fortran extensions on Windows, and based on all of the questions related to issues with Python 3.6 scipy and numpy installation on Windows, it seems like this is a common problem without a simple solution.

Any feedback or insight would be greatly appreciated.

Answer 1

Finally got this working.

Short version:

Make sure you use 64-bit compilers (triple check the build for mingw-w64) for 64-bit Python. Not as obvious as it sounds to an f2py newb on Windows.

Long version:

I uninstalled my existing copy of MinGW (I suspect it was a 32 bit version) and instead downloaded a specific 64-bit build of mingw-w64 7.2.0 from sourceforge, specifically x86_64-7.2.0-release-posix-seh-rt_v5-rev1.7z. This stackoverflow question was useful.

I unzipped and copied the "mingw64" folder into my C: drive (C:mingw64). I added C:mingw64in to my user Path.

I uninstalled the anaconda version of MinGW with conda uninstall mingw. Note, this is only necessary if you've previously installed MinGW using conda.

Upon running f2py -c igrf12.pyf igrf12.f --compiler=mingw32 (in same directory as igrf12.pyf, see Scipy Documentation for how to generate *.pyf file), pyigrf12.cp36-win_amd64.pyd is created without any errors. I can finally import pyigrf12 successfully and access the underlying Fortran subroutines (e.g. igrf12syn).

Note, I can also run f2py -c igrf12.pyf igrf12.f --compiler=msvc successfully, but then I have to manually copy and paste the libigrf12....gfortran-win_amd64.dll (generated in .UNKNOWN.libs) into the same directory as pyigrf12.cp36-win_amd64.pyd to avoid ImportError: DLL load failed: The specified module could not be found. mentioned in Method 2 of my question.

Just to re-iterate: Make sure C:mingw64in is added to your path!

By the way, f2py was painless for me on macOS Sierra and Ubuntu. If the above still doesn't work for you, I recommend trying on Linux, macOS or Windows Subsystem for Linux.