HACKING

This HACKING file describes the development environment.  	-*- org -*-

  Copyright (C) 2008, 2009, 2011 ViewPlus Technologies, Inc. and Abilitiessoft, Inc.
  Copyright (C) 2012, 2013, 2014,2015 Swiss Library for the Blind, Visually Impaired and Print Disabled

  Copying and distribution of this file, with or without modification,
  are permitted in any medium without royalty provided the copyright
  notice and this notice are preserved. This file is offered as-is,
  without any warranty.

This file attempts to describe the maintainer-specific notes to follow
when hacking liblouis. 

* Table of Contents 							:TOC:
 - [[#developing][Developing]]
   - [[#where-to-get-it][Where to get it]]
   - [[#build-requirements][Build requirements]]
   - [[#gnulib][Gnulib]]
   - [[#how-to-build][How to build]]
   - [[#install-with-homebrew][Install with Homebrew]]
   - [[#docker][Docker]]
   - [[#how-to-run-tests][How to run tests]]
   - [[#how-to-debug][How to debug]]
   - [[#how-to-find-memory-leaks][How to find memory leaks]]
   - [[#how-to-analyze-performance][How to analyze performance]]
   - [[#how-to-build-for-windows][How to build for Windows]]
 - [[#release-procedure][Release Procedure]]
   - [[#update-the-news-entry][Update the NEWS entry]]
   - [[#set-the-version-number][Set the version number]]
   - [[#commit-and-tag][Commit and tag]]
   - [[#make-the-release][Make the release]]
   - [[#upload][Upload]]
   - [[#online-documentation][Online documentation]]
   - [[#web-site-maintenance][Web site maintenance]]
   - [[#announce][Announce]]

* Developing
** Where to get it
The development sources are available through git at github.com:

  https://github.com/liblouis/liblouis

** Build requirements
To build Automake, Autoconf, and Libtool are used. If you are getting
the sources from git (or change configure.ac), you'll need to have
these tools installed to (re)build. Optionally (if you want to
generate man pages) you'll also need help2man. All of these programs
are available from ftp://ftp.gnu.org/gnu.

If you want to run the YAML based test suite you will have to install
~libyaml~.

On Mac OS, the programs can be optained with Homebrew (http://brew.sh):

  brew install automake libtool pkg-config texinfo

Note that if you are using Homebrew to install liblouis (see below),
the build dependencies are installed automatically.

** Gnulib
Gnulib (http://www.gnu.org/software/gnulib) is used to provide
portable basic functionality to programs and libraries. We use two
instances of gnulib, one to provide portable functions such as
~malloc~, ~strndup~, etc to the library and another one to provide
portable functionality such as ~getopt~, ~progname~ or
~version-etc~ to the tools.

The first time invocation to import gnulib for the library was

  gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files \
	      malloc-gnu realloc-gnu strndup

and for the tools

  gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files \
	      getopt-gnu malloc-gnu progname version-etc

More modules might have been added since. The currently-used gnulib
modules and other gnulib information are recorded in
~gnulib/m4/gnulib-cache.m4~ and ~tools/gnulib/m4/gnulib-cache.m4~.

If you want to update from the current gnulib, install gnulib, and
then run the following commands in the top-level directory.

  gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files
  gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files

** How to build
After getting the sources from git, with 

  git clone https://github.com/liblouis/liblouis.git

and installing the tools above, change to the liblouis directory and
and bootstrap the project with the following command

  ./autogen.sh

to do a fresh build. Then run configure as usual:

  ./configure

You have the choice to compile liblouis for either 16- or 32-bit
Unicode. By default it is compiled for the former. To get 32-bit
Unicode run configure with --enable-ucs4 .

After running configure run "make" and then "make install". You must
have root privileges for the installation step.

** Install with Homebrew
Homebrew (http://brew.sh) is a package manager for Mac OS X that
installs software from source. There is nothing special about the
installation process in the sense that under the hood it happens
exactly as described above, with the only difference that Homebrew
automates it completely.

First, use the ~brew tap~ command to add the repository that includes
the liblouis formula:

  brew tap liblouis/liblouis

Now you are ready to install liblouis:

  brew install liblouis

** Docker
Docker (https://www.docker.com) can be useful both for creating a
development environment for liblouis, and for shipping the
application.

Setting up a developer environment can take long and can be
problematic especially for Windows. Thanks to Docker we can set up the
environment for you, we can easily distribute it as an image, which
can be run by anybody and will behave exactly the same for everybody.

Docker images of liblouis are being built automatically each time
something changes in the code (see https://registry.hub.docker.com/repos/liblouis).
In order to use them, first get Docker at
http://docs.docker.com/introduction/get-docker. Download the latest
liblouis image with:

  docker pull liblouis/liblouis

Then, enter the development environment by running the image in a
Docker container:

  docker run -it liblouis/liblouis bash

Local files and directories can be "mounted" inside the container, in
order to make it easier to edit files and to persist changes across
runs. For example, to use local table files:

  docker run -it -v $(pwd)/tables:/tmp/liblouis/tables liblouis/liblouis bash

See the Docker documentation for more info.

The same Docker image can be used as a development environment and as
the application itself. For example, to run the lou_translate tool
from inside a Docker container:

  docker run -it liblouis/liblouis lou_translate en-us-g1.ctb

To rebuild the image yourself, run the following command in the root
directory of the liblouis source:

  docker build -t liblouis/liblouis .

A .dockerignore file is required if you want to compile the source both on te host and in the Docker
container. The .dockerignore file can be updated from .gitignore with:

  make .dockerignore

** How to run tests
Tests are run with

  make check

** How to debug
First you have to build liblouis with debugging info enabled.

  $ ./configure CFLAGS='-g -O0 -Wall -Wextra'
  $ make

Starting the programs under the tools directory within gdb is a little
tricky as they are linked with libtool. See the info page of libtool
for more information. To start lou_checktable for table wiskunde.ctb
for example you'd have to issue the following commands:

  $ libtool --mode=execute gdb ./tools/lou_checktable
  (gdb) run tables/wiskunde.ctb

** How to find memory leaks
Valgrind is a tool that can be used to find memory errors. It is
recommended that you compile liblouis without any optimizations and
with all warnings enabled before running it through Valgrind:

  $ ./configure CFLAGS='-g -O0 -Wall'
  $ make

Then use Valgrind to analyze liblouis. For example you can run
lou_translate trough Valgrind:

  $ libtool --mode=execute valgrind -v --tool=memcheck \
    --leak-check=full --leak-resolution=high --log-file=valgrind.log \
    ./tools/lou_translate en-us-g2.ctb

Type a few words at the prompt, check translation and terminate
lou_translate. Now open the file valgrind.log and see if there are any
memory leaks reported.

You can also just run lou_checktable for example:

  $ libtool --mode=execute valgrind -v --tool=memcheck \
    --leak-check=full --leak-resolution=high --log-file=valgrind.log \
    ./tools/lou_checktable tables/nl-BE-g0.utb

Again open valgrind.log to see if any memory leaks were reported.

For the full experience run lou_allround under Valgrind:

  $ libtool --mode=execute valgrind -v --tool=memcheck \
    --leak-check=full --show-reachable=yes \
    --leak-resolution=high --track-origins=yes \
    --log-file=valgrind.log ./tools/lou_allround

*** Electric Fence
Electric Fence is a tool that helps detect memory access that overruns
the boundaries of a malloc() memory allocation, and access to memory
that has been free()d.

Under Debian the usage is fairly straightforward:

#+BEGIN_SRC shell
$ sudo apt install sudo apt install electric-fence
#+END_SRC

Then compile [[How to debug][as above]] and invoke the debugger [[How to debug][as above]]. Inside =gdb=
set up =LOUIS_TABLEPATH= and =LD_PRELOAD= as follows:

#+BEGIN_SRC gdb-script
set environment LOUIS_TABLEPATH ./tables,./tests/tables,./tests/tables/moreTables,./tests/tablesWithMetadata,./tests/tables/emphclass
set environment LD_PRELOAD /usr/lib/libefence.so.0.0
run
#+END_SRC

If there are problems with memory access the program will run into a
segmentation fault which you can consequently analyze in the debugger.

** How to analyze performance
Gprof helps you analyze the performance of programs. You have to
compile liblouis as follows:

  $ ./configure --disable-shared
  $ make clean all CFLAGS='-g -O0 -pg' LDFLAGS='-all-static'

Then translate some stuff with a large table:

  $ ./tools/lou_translate tests/tables/large.ctb 

Finally look at the call profile:

  $ libtool --mode=execute gprof ./tools/lou_translate gmon.out

** How to build for Windows
See the README.windows file and the windows subdirectory.

*** How to cross-compile for Windows
To compile for win32, use the MinGW win32 cross-compiler as shown
below. Use the prefix option to install the binaries to a temporary
place where you can create a zip file. The LDFLAGS='-all-static'
ensures that libgcc is linked in statically. Otherwise the users need
to have libgcc_s_sjlj-1.dll.

Some users want the dlls unversioned. To achieve that add
'-avoid-version' to LDFLAGS.

  ./configure --build=i686-pc-linux-gnu --host=i686-w64-mingw32 --prefix=/tmp/liblouis-mingw32
  make LDFLAGS='-avoid-version -Xcompiler -static-libgcc'
  make install
  zip -r liblouis-mingw32msvc.zip /tmp/liblouis-mingw32

To compile for win64, use the MinGW-w64 cross-compiler:

  ./configure --build=i686-pc-linux-gnu --host=x86_64-w64-mingw32 --prefix=/tmp/liblouis-w64-mingw32
  make LDFLAGS='-avoid-version -Xcompiler -static-libgcc'
  make install
  zip -r liblouis-w64-mingw32.zip /tmp/liblouis-w64-mingw32

Two makefile rules have been provided to do this automatically for you. Docker will be used if the
right cross-compiler is not installed. To compile for win32 using the MinGW cross-compiler, run make
distwin32. This will produce a ZIP file called liblouis-<version>-win32.zip. To compile for win64
using the MinGW-w64 cross-compiler, run make distwin64. This will produce a ZIP file called
liblouis-<version>-win64.zip.

*** TODO How to build for Windows using Cygwin
(possibly use a Vagrantfile as demonstration + explain that Cygwin
binaries can not be used outside the Cygwin environment)

*** TODO How to build for Windows using MinGW
(possibly use a Vagrant file as demonstration)

* Release Procedure
These steps describe what a maintainer does to make a release; they
are not needed for ordinary patch submission.

** Update the NEWS entry
If any new tables were added or renamed please note them with their
name in the NEWS entry. This is usefull for projects like NVDA.

** Set the version number
Update the version number in NEWS (with version, date, and release
type), ChangeLog, configure.ac and windows/include/config.h.

Don't forget to update the libtool versioning info in configure.ac,
i.e. LIBLOUIS_REVISION and possibly LIBLOUIS_CURRENT and LIBLOUIS_AGE.

** Commit and tag
Commit the changes and tag this version

  git tag -s v2.6.0 -m "Release 2.6.0"
  git push origin v2.6.0

If you know the exact version number that needs to be tagged use

  git tag -s v2.6.0 -m "Release 2.6.0" <commit>
  git push origin v2.6.0

** Make the release
Check out a clean copy in a different directory, like /tmp. Run
autogen.sh and configure with no special prefixes. Run make distcheck.
This will make sure that all needed files are present, and do a
general sanity check. Run make dist. This will produce a tarball.

  ./autogen.sh && ./configure && make && make distcheck && make dist

** Upload
Add the tarball to the github liblouis releases page, i.e. add it
under https://github.com/liblouis/liblouis/releases with the specific
release and add a link to it in $WEBSITE/downloads/index.md. See below
for instructions on how to update the web site.

** Online documentation
The online documentation is part of the liblouis web site. To add it to the
site simply copy doc/liblouis.html to $WEBSITE/documentation/liblouis.html.
Make sure you add the proper YAML front matter. Again see below for
instructions on how to update the web site.

** Web site maintenance
The liblouis web site at liblouis.org is maintained with the help of
github pages (https://pages.github.com/). To edit the site just check
out the repo at https://github.com/liblouis/liblouis.github.io. You'll
need to know a few things about Jekyll (http://jekyllrb.com/) and
markdown, the markup that is used to edit the content. In order to
update the site simply edit, commit and push.

For the new release update the project web site.

- Add a post containing the current NEWS to the _posts directory and
- add the download artifacts to the download page

** Announce
Send an announcement to the liblouis list
liblouis-liblouisxml@freelists.org. See ANNOUNCEMENT for an example.