SOFA Statistics

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

[[:home | Home]]

[[:proj:project | Project]]

====== Mac Development Environment ======

===== Virtualising Mac OSX =====

==== VirtualBox Setup ====

Get iBoot-Ivy-Bridge.iso from http://www.tonymacx86.com/downloads.php?do=cat&id=3

Set up VirtualBox machine with iso file as CD (Settings>Storage)

20GB should be plenty for VM.

Other settings:

System>Motherboard
  * 1000MB (too much can be a problem)
  * CD/DVD as first boot under Boot Order
  * Chipset ICH9
  * Pointing device: USB tablet (if you foolishly select PS/2 Mouse you won't get a mouse or keyboard)
  * Extended Features: Enable I/O APIC only
System>Processor
  * 1 CPU (too many is a problem)
  * Extended Features: Enable PAE/NX
System>Acceleration>Hardware Virtualization
  * Enable VT-x/AMD-V
  * Enable Nested Paging

==== Process ====

- Put Snow Leopard CD into drive
  - Boot into VM\\ Should see iBoot-Ivy-Bridge under CD image (under friendly apple image)
  - Using VB menu Devices>CD/DVD Devices, untick iBoot iso
  - Tick Host Drive ASUS DRW etc
  - F5 (have to click inside window once to give focus)\\ Wait\\ Should see words under CD image change to Mac OS X Install DVD
  - Space bar to click\\ Should get grey background and dark grey apple logo\\ Wait
  - Eventually get to point where need to select device to install onto - and nothing there! Never fear. See item 4 in http://www.macbreaker.com/2012/02/how-to-install-mac-os-x-snow-leopard-on.html.
    - Utilities>Disk Utility>Erase
    - Call it sofamac and proceed with Erase - should ask for confirmation and then partition.
  - Once finished, select new sofamac device to install onto.
  - Power off VirtualBox and put iBoot iso back as CD. Restart.
  - Select sofa (right arrow key then Enter, not tab)
  - Set up account (use Grant Paton-Simpson as name and supply brief password). Use g as username unless you love typing ;-)
  - Power off. Take snapshot.
  - Power up.
  - Apple main menu > Software Updates ... Specify Mac OSX Update Combined (and untick everything else)
  - Confirm upgrade with >> sw_vers
  - New snapshot.
  - Install MultiBeast 3.10.1.pkg (http://www.multibeast.com/ and http://www.tonymacx86.com/downloads.php?do=file&id=128)
  - Select easybeast, not DSDT version. Don't think you select anything else.
  - Open up Finder and go to the folder "Extra" in the main hard drive, and open the file org.Chameleon.boot.plist. Between <dict> and </dict> in the file, insert the following line.\\ <key>Graphics Mode</key>\\ <string>1280x1024x32</string>.\\ Save As to desktop but add .plist on end and don't tick checkbox under encoding/charset. Then drag over original location.
  - Exit VM.
  - VBoxManage setextradata "macpackager_upgraded" "CustomVideoMode2" "1280x1024x32"
  - Remove iBoot-Ivy-Bridge under CD.
  - Set as Full Screen mode once in VM
  - Turn off "Check for updates:" as per Step 11 http://www.macbreaker.com/2012/02/snow-leopard-virtualbox.html
  - Don't manually clone anything - use the clone functionality once in snapshot view. I did machine state and not keeping snapshots. Worked just like you'd hope. Awesome.

Can't use /usr/share/virtualbox/VBoxGuestAdditions.iso on Mac.

===== Setting up Xcode =====

* https://developer.apple.com/downloads/index.action?q=xcode (log in as kiwiappdev)
  * Successfully followed path from xcode 3.2, 3.2.1, to 3.2.2 - install packages from inside dmgs

===== Homebrew =====

* Open terminal

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

* brew doctor
  * brew install git (Note - can be very, very slow so don't bail out too early)
  * sudo chown -R `whoami` /usr/local # otherwise symlinking won't work - see http://developpeers.com/blogs/fix-for-homebrew-permission-denied-issues. Note - /usr/local gets made by brew it seems so can't do this step too early

===== ImageMagick (convert) =====

Cannot use standard installers as are all 64-bit (will get Bad CPU error message). So ignore the binary-releases page. Homebrew works when osx 10.6.8 and xcode 3.2.2. (unlike 10.6.0 and 3.2) so ...

brew install imagemagick

Then manually redo the imagemagick bit (homebrew has done all the dependencies as well so still useful). Brew ignores half the commands I gave it e.g. --enable-static so need to take care of that myself.

http://www.imagemagick.org/download/releases/ImageMagick-6.9.0-9.tar.gz

Extract contents of tar.gz file onto desktop.

cd into folder

./configure --disable-osx-universal-binary --prefix=/usr/local/Cellar/imagemagick/6.9.0-9 --disable-shared --disable-installed --enable-static --with-png --with-gslib LDFLAGS='-L/usr/local/opt/libpng/lib -L/usr/local/Cellar/ghostscript/9.15/lib' CPPFLAGS='-I/usr/local/opt/libpng/include -I/usr/local/Cellar/ghostscript/9.15/include'

The --prefix determines where the code ends up - and we might as well keep everything consistent with what it would have been if the homebrew approach had worked.

http://www.imagemagick.org/script/advanced-unix-installation.php

--enable-static. So it bakes more into the binary rather than relying on having access to shared files elsewhere. convert ends up much larger in size but has much fewer dependencies.

The FLAGS are so imagemagick detects the PNG - otherwise even if we specify --with-png it won't be able to fulfil it. There will be messages at the end of the configure step like png.h usability no.

If we were compiling imagemagick with ghostscript support built in, which we are not (;-)), we would need to add flags for ghostscript as well as for png - the --with-gslib is not enough on its own. We leave ghostscript out so we only use delegates.xml to point to what we need - we do NOT want to bake any dependencies for gs into convert. It can only get in the way of how we will link to ghostscript.

See http://codetheory.in/convert-split-pdf-files-into-images-with-imagemagick-and-ghostscript/

At one point there seemed to be a permissions problem with /usr/lib again so needed to sudo chown -R `whoami` /usr/local again and then brew link various libs: libtiff jbig2dec little-cms2 ghostscript. Might not be required but try it if a problem.

===== Ghostscript (gs) =====

Don't bother with a brew install. Just download latest source, extract, cd into folder with configure inside and:

./configure --disable-dynamic --with-drivers=FILES

I thought I could get away with --with-drivers=PNG but it failed but =FILES dropped it enough (about 17MB) and worked.

--disable-dynamic bakes in a whole lot so gs doesn't try to find it outside e.g. gs_init.ps. No need to have a parallel share/ghostscript/...gs_init.ps
===== convert and gs =====

- Don't compile ghostscript with gslib on board
  - Do export MAGICK_CONFIGURE_PATH='/...' whatever path you put delegates.xml in
  - Do put colors.xml in same folder (now you've got all the essential configuration files I need)
  - Ensure PATH is set to start with the folder gs is in. Removes need to hardwire in the path to gs all over the place.

export PATH=<folder_gs_etc_are_in>:${PATH}

Success!

If delegates.xml can't be found, the error message might be something like:

convert: UnableToOpenConfigureFile `delegates`.xml' @warning/configure.c/GetConfigureOptions/706.

If the command used to handle something doesn't work the error message from imagemagick is not very useful. It will be something like:

convert: NoImagesDefined `/Users/g/Desktop/test.png' @error/convert.c/ConvertImageCommand/3212.
====== Linking Dependencies to @executable_path ======

otool makes it possible to identify dylib dependencies.

====== Ghostscript Extras ======

-P- to prevent ghostscript looking for shared resources in same folder.

-I to determine where it should look first. Note - on Linux/OSX use a colon as the delimiter. See entry on -sFONTMAP in http://ghostscript.com/doc/9.15/Use.htm#Font_related_parameters.

./gs -P- -I"/Users/g/Desktop/ghostscript/9.15/share/ghostscript/Resource/Init/:/Users/g/Desktop/Desktop/ghostscript/9.15/share/ghostscript/lib/:/Users/g/Desktop/ghostscript/9.15/share/Resource/Font/:/Users/g/Desktop/ghostscript/9.15/share/fonts/" -o "/Users/g/Desktop/success.png" -sDevice=png16m -r300 -dFirstPage=1 -dLastPage=1 '/Users/g/Desktop/python_argentina.pdf'

-o is not same as sOutputFile=. It is a shorthand for misc other changes. See http://ghostscript.com/doc/9.15/Use.htm#File_output

-dFirstPage and -dLastPage are useful for multipage PDF files.

-r resolution. If only one number is e.g. 300x300.

gs -? displays very useful help including where gs thinks it resources are. E.g.:

Search path:
     /usr/local/Cellar/ghostscript/9.15/share/ghostscript/Resource/Init :
     /usr/local/Cellar/ghostscript/9.15/share/ghostscript/lib :
     /usr/local/Cellar/ghostscript/9.15/share/ghostscript/Resource/Font :
     /usr/local/Cellar/ghostscript/9.15/share/ghostscript/fonts :
     /usr/local/Cellar/ghostscript/9.15/share/fonts/default/ghostscript :
     /usr/local/Cellar/ghostscript/9.15/share/fonts/default/Type1 :
     /usr/local/Cellar/ghostscript/9.15/share/fonts/default/TrueType :
     /usr/lib/DPS/outline/base : /usr/openwin/lib/X11/fonts/Type1 :
     /usr/openwin/lib/X11/fonts/TrueType

One of the most important is Resource/Init which contains gs_init.ps.
===== Python 2.7.9 =====

* brew install openssl
  * brew link openssl --force
  * brew install python --with-brewed-openssl (Note - very slow etc as prev comments)

See http://stackoverflow.com/questions/25372911/python-pip-error-on-osx

===== wxPython =====

Decided to stick to 2.8.12 rather than risking change to 3. So couldn't use homebrew which installs 3.0.

http://sourceforge.net/projects/wxpython/files/wxPython/2.8.12.1/ and get unicode mac python 2.7 version (have to hover to see all relevant details)

Is installed to /usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7

Create a wxredirect.pth file so python can detect wxPython. See htp:/stackoverflow.com/questions/21864202/can-not-import-wxpython-mac. utf-8 BOM fails so avoid that text format.

Inside wxredirect.pth, add a single-line command (it seems that a multiline-command fails).

import site; site.addsitedir('/usr/local/lib/wxPython-unicode-2.8.12.1/lib/python2.7')

Then save as: /usr/local/Cellar/python/2.7.9_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/wxredirect.pth

If you open the Cellar version of Python - namely /usr/local/Cellar/python/2.7.9_1/Frameworks/Python.framework/Versions/2.7/bin/python2.7 (warning - there may be other versions of python2.7 on the machine - only the ones with a pth file pointing to the correct location for wxPython will allow a successful import wx statement) you should be able to import wx.

===== PostgreSQL =====

brew install postgresql

Then, so the system can find postgresql even though it is not in the standard location:

export PATH=/usr/local/Cellar/postgresql/9.4.0/bin:$PATH
===== Pip installations =====

Ensure you use the version of pip that homebrew installed i.e.

/usr/local/Cellar/python/2.7.9/bin/pip2.7 install <package_name>

Then install

* pyPDF
  * psycopg2
  * numpy
  * pymysql
  * requests
  * matplotlib

===== wkhtmltopdf =====

http://wkhtmltopdf.org/downloads.html

Select: OS X 10.6+ (Carbon)  32-bit  recommended; produces selectable text and smaller file sizes

===== pdftk =====

Supports 10.6 onwards so Snow Leopard is supported (unlike Leopard)

https://www.pdflabs.com/tools/pdftk-server/

Right click on big green "Mac OS X Download" button down screen and choose to save linked file as ... pdftk_server-2.02-mac_osx-10.6-setup.pkg

On Mac, for some unknown reason, downloads as Unknown unless menu-click it and select Save As, and accept displayed name. Bizarre! But this workaround works fine.

Test with pdftk --version

Installs to /opt/pdflabs/pdftk/bin

===== Frameworks Libraries =====

Put the following libraries into a sibling folder of the executable so they can be called the same way while developing as when packaged. Use soft links.

* wkhtmltopdf (in /usr/local/bin/)
  * pdftk (in /usr/local/bin/)
  * convert (in /usr/local/bin/)

cd /usr/local/Cellar/python/2.7.9/bin
  ln -s /usr/local/bin/wkhtmltopdf wkhtmltopdf
  ln -s /usr/local/bin/pdftk pdftk
  ln -s /usr/local/bin/convert convert

===== Testing =====

cd /usr/local/Cellar/python/2.7.9/bin # important this is our current working directory
  # as we use os.getcwd() to reference locations for linked libraries
  python2.7 /Users/g/Desktop/sofadev/sofa.main/start.py # or whatever path to start.py is

====== Packaging ======

===== Overview =====

Pyinstalled bakes all the python dependencies into its binary (but not wkhtmltopdf, pdftk, or convert). If we let it, pyinstaller would also bake in all the scripts of SOFA. But we do NOT want this - we want the binary to have to use humanly-editable scripts just like on Windows and Linux. Easier to debug when packaging and when using. Recompiling is not fun or quick and it discourages exploration. So we cheat. We supply it with launch_mac.py which requires all the python dependencies e.g. wx.webkit. And we starve it of import2run.py by only having the import2run_hide.py version available at the time the binary is created. Once the binary is made, we put it in a folder alongside all the SOFA scripts including a working import2run.py which imports start.py. We also make sure the binary can find wkhtmltopdf etc. We can do this by putting files in the right places in the app bundle (viewable using Open Package Contents). Note - pdftk expects 3 dylinbs to be in the ../lib/ folder.

===== Installing Pyinstaller =====

The mailing list recommended I use the dev version. And this can be installed using:

/usr/local/Cellar/python/2.7.9/bin/pip2.7 install -e git://github.com/pyinstaller/pyinstaller.git#egg=PyInstaller

===== To Edit Specs or Not To Edit Specs =====

There is less need to edit spec files than there used to be but manual changes are still needed when putting special files in Framework folders e.g. wkhtmltopdf, pdftk, and convert, and the dylibs required by pdftk.

Will need to copy the following (note - these are in these locations sometimes only because we symlinked them there earlier):

/usr/local/bin/wkhtmltopdf ->
  /usr/local/bin/pdftk ->
  /usr/local/bin/convert ->
  /opt/pdflabs/pdftk/lib -> (has all the dylibs required by pdftk)
===== Making Spec Files =====

Just run the pyinstaller process and you get a spec file produced as a byproduct. Modify it, save it somewhere safe, and use it thus:

===== Making Spec File =====

--name. Determines name of binary and app bundle. Do not include .spec as that is added to the end of whatever you use - so if you name it myapp.spec you will get a spec file made named myapp.spec.spec which is not what you want ;-) )

--clean. So we don't have to manually wipe files/folders first.

--strip. To reduce size of installer. A good idea as the resulting files substantially reduce in size (still too big but a lot less so).

--onedir vs --onefile. Strangely, the onedir output folder version (as opposed to the app bundle version) completely works. convert actually makes the png images. But the app version doesn't. Possibly the onefile version would also work if put into a folder with all the sofa .py scripts e.g. export_output.py put in the folder it is in etc.

--icons. e.g. /Volumes/grantshare/mac_extras/sofastatistics.icns

Note - references to /Volumes/grantshare will only work if Grant's computer is on and you have accessed it from the Mac (Go>Network> etc) first.

Note - always run script on Grant's computer first to populate macmaker folder.

Example:

/usr/local/Cellar/python/2.7.9/bin/python2.7 /usr/local/bin/pyinstaller --clean --name='/Users/grantpaton-simpson/Desktop/sofa_<desc here>' <other options here> '/Volumes/grantshare/macmaker/sofa.main/launch_mac.py'

/usr/local/Cellar/python/2.7.9/bin/python2.7 /usr/local/bin/pyinstaller --clean --onefile --windowed --strip --name='SOFA Statistics' --icons='/Volumes/grantshare/mac_extras/sofastatistics.icns' '/Volumes/grantshare/macmaker/sofa.main/launch_mac.py'
===== Using Spec File(s) =====

Save the different spec files and then run them thus:

cd ~ && /usr/local/Cellar/python/2.7.9/bin/python2.7 /usr/local/bin/pyinstaller --onedir --windowed '/Volumes/transfer/SOFA Statistics_save.spec'

I have already shifted a version into macmaker so I can keep a copy on the main dev system and share with the mac dev environment.

===== What Must Go Where =====

In the folder the binary lives in e.g. inside Contents/MacOS we need all the sofa.main content e.g. export_data.py, output.py etc including the folders SOFA uses to populate the sofastats and sofastats_recovery folders. E.g.

* export_output.py
  * start.py
  * _internal folder

Under the level above e.g. Contents, we need a Frameworks folder containing key binaries:

* Frameworks
    * convert
    * gs
    * pdftk
    * wkhtmltopdf
    * and all the dylibs etc that these libraries require (other than pdftk - see below)

Also a lib folder containing the dylibs required by pdftk (namely, the contents of /opt/pdflabs/pdftk/lib. pdftk expects its dylibs to be in ../lib)

* lib
    * libgcc_s.1.dylib
    * libgcj.11.dylib
    * libiconv.2.dylib
    * libstdc++.6.dylib

Then find out what convert needs to run thus:

otool -L /Users/grantpaton-simpson/dist/SOFA\ Statistics.app/Contents/Frameworks/convert

Then for each dylib etc it requires, put the resource in the Frameworks folder and modify the convert binary so it looks in its own path. Do this using install_name_tool.

The command required is like this:

install_name_tool -change '/usr/local/Cellar/imagemagick/6.9.0-3/lib/libMagickCore-6.Q16.2.dylib' '@executable_path/libMagickCore-6.Q16.2.dylib'

I have automated this with a script called dependable.py:

#! /usr/local/Cellar/python/2.7.9/bin/python2.7
	# -*- coding: utf-8 -*-
	"""
	Inspired by http://thecourtsofchaos.com/2013/09/16/how-to-copy-and-relink-binaries-on-osx/
  
	Copy everything into a working folder so we don't mess up any original binaries.
  
	Must run as sudo otherwise no permissions.
	"""
	from __future__ import print_function
	import os
	import shutil
	import subprocess
  
	working_folder = "/Users/grantpaton-simpson/Desktop/indeps/"
	processed_binpaths = []
  
	def relink_dep(working_binpath, dep_path):
	    """
	    working_binpath - never want to change original versions of binaries - only 
	    copies in the working folder.
	    """
	    old_link = dep_path
	    dep_fname = os.path.split(dep_path)[1]
	    new_link = "@executable_path/" + dep_fname
	    retval = subprocess.call(["install_name_tool", "-change", old_link, new_link,
		working_binpath])
	    if retval != 0:
		raise Exception("Problem relinking %s inside %s" % (old_link, working_binpath))
  
	def get_deps(orig_binpath, working_folder):
	    """
	    Adjust copy of file originally at orig_binpath so expects dependencies to be in
	    same path it is in.
  
	    Pass any dep files that have not been processed already back through get_deps.
	    """
	    # check not a repeat
	    if orig_binpath in processed_binpaths:
		print("Not copying or relinking %s. Already handled" % orig_binpath)
		return
	    processed_binpaths.append(orig_binpath)
	    # make working copy of binary (we never mess with the original)
	    bin_fname = os.path.split(orig_binpath)[1]
	    working_binpath = os.path.join(working_folder, bin_fname)
	    shutil.copy(orig_binpath, working_binpath)
	    # identify dependencies
	    output = subprocess.check_output(["otool", "-L", working_binpath])
	    dep_path_output = [x.strip().split(" ")[0] for x in output.split("\n") if x.strip() != ""]
	    if len(dep_path_output) < 2:
		print("%s has no dependencies" % working_binpath)
		return
	    else:
		# Handle dependencies - flexibly relink to each and send them through again in case 
		# they have their own dependencies.
		dep_paths = dep_path_output[1:] # always displays binary as first item so ignore that one
		for dep_path in dep_paths:
		    relink_dep(working_binpath, dep_path)
		    get_deps(dep_path, working_folder) # back down the rabbit hole with you!
  
	def fix_binpath(binpath, working_folder_root, bin_shortname=None):
	    if raw_input("Are you running as sudo? y/n ") != "y":
		print("Maybe next time")
		return
	    if not bin_shortname:
		bin_shortname = os.path.split(binpath)[1]
	    working_folder = working_folder_root + "_" + bin_shortname 
	    try:
		shutil.rmtree(working_folder)
	    except OSError:
		pass
	    os.mkdir(working_folder)
	    get_deps(binpath, working_folder)
	    subprocess.call(["chmod", "-R", "777", working_folder])
	    print("Processed:\n" + ";\n".join(processed_binpaths))
	    print("Finished")
  
	fix_binpath(binpath="/usr/local/Cellar/imagemagick/6.9.0-9/bin/convert",
	    working_folder_root="/Users/grantpaton-simpson/Desktop/indeps")
  
	fix_binpath(binpath="/usr/local/Cellar/ghostscript/9.15/bin/gs",
	    working_folder_root="/Users/grantpaton-simpson/Desktop/indeps")

===== Practical Packaging =====

To simplify things, maybe do the dependable.py strategy on pdftk so ALL files are in one folder. One less thing to have to handle.

That would mean we have stable Frameworks (convert, gs, delegates.xml, colors.xml, wkhtmltopdf etc etc) and Resources folders (has sofastatistics.icns and all the pyinstaller bits and pieces) and the only thing that changes is the MacOS folder.

So modify spec file to copy across those two folders into app once made, and copy contents from sofa.main into MacOS and we have a package ready to go.

Please fill all the letters into the box to prove you're human. H J H M U Please keep this field empty:

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Share Alike 4.0 International

SOFA Statistics

User Tools

Site Tools

Page Tools