SOFA Statistics

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

[[:home | Home]]

[[:proj:project | Project]]

====== Windows Installer Notes ======

===== What Next =====

If already set up dev pipeline on machine go straight to "Make Package - Overview".

Otherwise, "Windows Development and Packaging - From Scratch"

===== Make Package - Overview =====

* Update sofawin/licence.txt if necessary
  * cd D:\sofa_dev_win\sofastatistics
  * git pull (SSH passphrase "matilda loves linux better")
  * bring across licence.txt if changed
  * Copy contents of D:\sofa_dev_win\sofastatistics\sofastats over top of D:\sofa_dev_win\sofa4packaging\sofastats (leaving behind tests, eclipse project files, __pycache__ folders, and any exes such as convert.exe and wkhtmltopdf.exe)
  * Eliminate debug = True (esp dev_debug = True in home.py)
  * Delete build and dist folders under D:\sofa_dev_win\sofa4packaging\sofastats (to ensure clean build)
  * Create executable

cd D:\sofa_dev_win\sofa4packaging\sofastats\
  pyinstaller sofastats.spec

* Under D:\sofa_dev_win\sofa4packaging\sofastats\ copy everything but __pycache__, build, and dist into dist\sofastats.
  * Change the version at the top of In D:\sofa_dev_win\packaging\1 sofastats.nsi:

!define SOFA_VERSION "1.5.3"

* In D:\sofa_dev_win\packaging right click on  "1 sofastats.nsi" and "compile NSIS script" then click on "Test Installer" button.
  * Run SOFA from icon on Desktop
  * (Assuming OK) copy sofastats-x.x.x_for_win.exe to the USB for safe-keeping
  * Compress to zip folder so sofastats-x.x.x_for_win.zip

===== Windows Development and Packaging - From Scratch =====

In short, how we get from a clean Windows machine to a development (to fix Windows-specific bugs) and packaging environment for SOFA.

==== Create working folders ====

Make D:\sofa_dev_win

Under that put:

* **sofastatistics** (actually, don't make it directly - instead, add it when cloning the git repo). This is the code. Can make changes here to get SOFA working on Windows and push the changes up into origin. Update this folder then copy across contents to sofa4packaging/sofastats to keep the version to be packaged up-to-date
  * **dependencies** - binaries such as wkhtmltopdf.exe and specific dependencies of those binaries e.g. delegates.xml (needed by convert.exe). You'll find them under sofa_win/sofa_dev_win/dependencies on the Ubuntu dev machine. Keep the Ubuntu dev machine versions up-to-date with what is used on the Windows machine when developing there and running SOFA unpackaged.
  * **packaging** - put all the files managing the actual building of packages e.g. spec files, nsi files etc. You'll find them under sofa_win/sofa_dev_win/packaging on the Ubuntu dev machine.
  * **sofa4packaging** - put contents of sofastatistics folder and add launch.py, import2run.py, and sofastats.spec as well from /packaging. Leave behind __pycache__, app.fil and sofastats.pot.
  * **store** - put junk here to avoid polluting / over-complicating other folders.

==== Python and dependencies installed ====

Set up prerequisites for running SOFA working on Windows at all - install latest version of Python 3 e.g. 3.9 (https://www.python.org/downloads/). Allow Python to be set in PATH so don't need to specify full path to python exe every time. Allow to be installed for all users. Also install all required pip and system dependencies (perhaps PostgreSQL). See list of dependencies further down under "Set launch.py dependency imports". Note pip install Pillow not PIL; matplotlib not pylab; pywin32 to enable associated imports (and https://github.com/KBNLresearch/iromlab/issues/100) etc. Have to translate from import requirements to module requirements. Usually straight forward.

Target pip installations at the python you will be building against (possibly only one). No need to be admin for pip installations unless Python was installed as admin.

If installed locally you might find Python somewhere like:

C:\Users\Giles Paton-Simpson\AppData\Roaming\Python\Python39

Run

cmd

ctrl-shift Enter if needing to run terminal as admin

python -m pip install requests

Re: PostgreSQL: use weak password for throw-away PG latest (e.g. 13) installation on Win 10.

==== Binary dependencies ====

Put copies of every dependency under D:\sofa_dev_win\dependencies and also D:\sofa_dev_win\packaging\sofastats

=== wkhtmltopdf (for HTML to PDF) ===

wkhtmltopdf.exe - download installer from https://wkhtmltopdf.org/downloads.html then go to installed location e.g. C:\Program Files\wkhtmltopdf and put copy in D:\sofa_dev_win\store and under D:\sofa_dev_win\sofa4packaging\sofastats

=== convert (for PDF to PNG) ===

convert.exe - unzip / install the PORTABLE version of Imagemagick e.g. ImageMagick-7.0.10-47-portable-Q16-x64.zip into D:\sofa_dev_win\store\imagemagick. Naming convention explained below:

Versions with Q8 in the name are 8 bits-per-pixel component (e.g. 8-bit red, 8-bit green, etc.),
  whereas, Q16 in the filename are 16 bits-per-pixel component. A Q16 version permits you to read
  or write 16-bit images without losing precision but requires twice as much resources as the Q8
  version. Versions with dll in the filename include ImageMagick libraries as dynamic link libraries.
  Unless you have a Windows 32-bit OS, we recommend this version of ImageMagick for 64-bit Windows:

from https://imagemagick.org/script/download.php

Also need delegates.xml

Note - delegates.xml must be edited so convert.exe can find the correct ghostscript executable namely gswin64c.exe (otherwise it defaults to gswin32c.exe). The ghostscript executable will be in the same folder as convert.exe so we don't need to worry about path thankfully.

Replace:

<delegate decode="ps:alpha" stealth="True" command="&quot;@PSDelegate@&quot; ...

with:

<delegate decode="ps:alpha" stealth="True" command="&quot;gswin64c.exe&quot; ...

=== ghostscript (what convert delegates specific PDF to PNG tasks to) ===

gswin64c.exe and gsdll64.dll from https://www.ghostscript.com/download/gsdnld.html
==== Get dev toolchain working e.g. eclipse, git ====

Install git and git bash etc - https://gitforwindows.org

Generate SSH key using Git GUI (Git Bash). Use passphrase "matilda loves linux better" (also in keepass2). Use OpenSSH-generated key made for Windows machine. Don't try to reuse Linux one - bad practice.

Change git remote as required so has SSH in it. id_rsa.pub is somewhere like C:\Users\Giles Paton-Simpson\.ssh. Log into launchpad (launchpad@p-s.co.nz using pwd from keepass2) and add additional SSH key.

Extras if paint self in corner with SSH key: https://stackoverflow.com/questions/50918607/remove-cached-password-for-git-on-windows Reinstalling won't solve it. You will have to update it in your Credential Manager. I.e. go to Control Panel > User Accounts > Credential Manager > Windows Credentials. You will see Git credentials in the list (e.g. git:https://...). Click on it, update the password, and execute git pull/push command from your Git bash and it it won't throw any error message.

Then get latest SOFA code using git within Git Cmd:

D:
  cd sofa_dev_win
  git clone https://git.launchpad.net/sofastatistics

==== Get SOFA working on Windows ====

cd "D:\sofa_dev_win\sofastatistics\sofastats"
  python -m start
 
Note - SOFA puts parent path into sys.path so we start in the same folder as start.py

Iron out inevitable bugs.

The pathlib library is a good start when working with Windows path names (a common source of problems).

==== Install pyinstaller ====

Run

cmd

ctrl-shift Enter if needing to run terminal as admin

python -m pip install pyinstaller

==== Set launch.py dependency imports ====

Run special script sofa_dev_win/packaging/import_dependencies.py to identify actual dependencies so they can be called by launch.py

#import CUBRID-Python ## 2014 Python 2 only
  import PIL  ## actually pillow installed
  import PyPDF2
  import adodbapi
  import agw
  import numpy
  import openpyxl
  #import pgdb  ## part of psycopg2 installation
  import psycopg2
  import pylab  ## part of matplotlib
  import pyodbc
  import pymysql
  import requests
  import sqlite3  ## part of Python
  import win32api  ## part of pywin32
  import win32com  ## part of pywin32
  import win32con  ## part of pywin32
  import wx  ## wxPython
  import wx.dataview  ## part of wxPython
  import wx.grid  ## part of wxPython
  import wx.html2  ## part of wxPython
  import wx.lib  ## part of wxPython
  import wx.lib.agw  ## part of wxPython
  import wx.lib.agw.hyperlink  ## part of wxPython
  import wx.lib.agw.hypertreelist  ## part of wxPython
  
  import import2run

Note - must specify standard library modules called upon and also explicitly call submodules as required e.g.

import sqlite3
  import wx.dataview
  import wx.grid
  import wx.html2
  import wx.lib
  import wx.lib.agw
  import wx.lib.agw.hyperlink
  import wx.lib.agw.hypertreelist

Otherwise errors like: ModuleNotFoundError: No module named 'wx.lib.agw.hypertreelist'

==== Identify Windows C Runtime (CRT) dependency path (nothing to install) ====

This dependency is required if using Python >= 3.5 and Windows systems < Windows 10. I.e. it is required.

E.g. C:\Windows\WinSxS\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.17134.1_none_50c6cb8431e7428f

See https://github.com/pyinstaller/pyinstaller/issues/1566#issuecomment-342916812. Note - the end part of the full path

(e.g. ..._31bf3856ad364e35_10.0.17134.1_none_50c6cb8431e7428f) will vary but there will only be one on your system

Context at: https://pyinstaller.readthedocs.io/en/v3.3.1/usage.html#windows

This dependency will be used as an argument in the pyinstaller command run later e.g.

%%--%%path C:\Windows\WinSxS\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.17134.1_none_50c6cb8431e7428f

==== Run pyinstaller for full run (making spec and using) ====

Note - sofa_win_dev/sofa4packaging/sofastats must contain latest code plus dependencies plus launch.py, import2run.py, and sofastats.spec as well from /packaging. Leave behind pycache, app.fil and sofastats.pot.

Note - usually people only use the --onedir option to start with for debugging reasons and the --onefile option later.

We want the --onedir option during development because it allows us to inject in the py modules we want into the folder so the exe will use them instead of what was originally baked in. In short, we can override individual modules while testing until we are happy. We could use the --onefile option to create a self-contained binary at that point but I like exposing the code - certainly makes it much, much easier to fix problems.

Edit import2run.py if required to ensure it is actually importing SOFA and not just printing output as a test

The actual command:

cd D:\sofa_dev_win\sofa4packaging\sofastats\
  pyinstaller --clean --exclude start --path C:\Windows\WinSxS\x86_microsoft-windows-m..namespace-downlevel_31bf3856ad364e35_10.0.17134.1_none_50c6cb8431e7428f launch.py

The binary executable is D:\sofa_dev_win\sofa4packaging\sofastats\dist\launch\launch.exe

Shift all D:\sofa_dev_win\sofa4packaging\sofastats\ in (except for __pycache__, build, or dist) and rename launch folder to sofastats so calls to sofastats package work.

==== Rename, relocate, and edit spec file ====

Warning - must wipe build folder first for a clean exe build

ghostscript - use the commandline version of ghostscript. http://stackoverflow.com/questions/2598669/ghostscript-whats-are-the-differences-between-linux-and-windows-variants. On Windows you have two executables, gswin64c.exe and gswin64.exe instead of gs only. The first one is to run Ghostscript on the commandline ("DOS box"), the second one will open two GUI windows: one to render the output, another one which is console-like and shows GS stdout/stderr or takes your command input if you run GS in interactive mode.

ImageMagick - use the portable version (big fat convert.exe with everything baked in vs scrawny non-portable version which relies the dependencies)

binaries=[
      ('D:/sofa_dev_win/dependencies/wkhtmltopdf.exe', '.'),  ## the docs for 3.4 show the structure as (binary, subfolder) where '.' means the main folder
      ('D:/sofa_dev_win/dependencies/convert.exe', '.'),  ## need portable version
      ('D:/sofa_dev_win/dependencies/gswin64c.exe', '.'),
      ('D:/sofa_dev_win/dependencies/gsdll64.dll', '.'),
  ],
  datas=[
      ('D:/sofa_dev_win/dependencies/delegates.xml', '.'), 
  ],

exe - icon='D:\sofa_dev_win\packaging\sofa_32x32.ico'

exe - upx=False

exe - name='sofastats'

exe - exclude_binaries=True  ## we want the binaries to go into the sofastats folder (we're one-dir remember) alongside the sofastats.exe binary so they'll be grabbed by col instead. If this is set to False and we're in one-dir then expect failure ;-)

pyz - level=9 ## heavily compressed?

col - upx=False

col - name='sofastats'  ## so folder is called sofastats (which it needs to be because that is the expected package name in internal calls e.g. from sofastats import ...)

Initially (so debugging is trivial - go on - don't be a martyr!):

exe - console=True
exe - debug=True

Then for prod:

exe - console=False
exe - debug=False

Put in D:\sofa_dev_win\sofa4packaging\sofastats\ as sofastats.spec. Needs to be in same folder as scripts being packaged i.e. launch.py etc

==== Run pyinstaller off spec file ====

cd D:\sofa_dev_win\sofa4packaging\sofastats\
  pyinstaller sofastats.spec

Shift all D:\sofa_dev_win\sofa4packaging\sofastats\ in (except for __pycache__, build, or dist). Folder already named sofastats because of spec edits.

==== Test binary and SOFA code folders on clean machine ====

Find a clean Windows 10 machine and test on it using the installer.

==== Make NSIS installer ====

Use UltraModernUI

Get latest version from [[https://sourceforge.net/projects/ultramodernui/files/UltraModernUI/]]

Note - don't need to specify every file - can just use the recursive syntax

Section ""
    RMDir /r $INSTDIR
    ; Set output path to the installation directory.
    SetOutPath $INSTDIR
    File /r "${DEV_PATH}\*.*"

==== Note on DAO ====

For MS Access

Re: dao file, it is because we can't dynamically get it from the frozen (or some similar issue) but it works if we generate it, grab it as a module, and import it. Start with installing pywin. Then go to site-packages\win32com\Pythonwin.exe and run COM Makepy Utility for DAO 3.6. Then go to win32com\gen_py folder and copy content of one of the py files and save as the module to import e.g. dao36_from_gen_py_after_makepy.py.

Please fill all the letters into the box to prove you're human. U Z P Q R Please keep this field empty:

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Share Alike 4.0 International

SOFA Statistics

User Tools

Site Tools

Page Tools