HACKING

This HACKING file describes the development environment.  	-*- org -*-

  Copyright (C) 2008, 2009, 2011 ViewPlus Technologies, Inc. and Abilitiessoft, Inc.
  Copyright (C) 2012, 2013, 2014,2015 Swiss Library for the Blind, Visually Impaired and Print Disabled

  Copying and distribution of this file, with or without modification,
  are permitted in any medium without royalty provided the copyright
  notice and this notice are preserved. This file is offered as-is,
  without any warranty.

This file attempts to describe the maintainer-specific notes to follow
when hacking liblouis. 

* Table of Contents 							:TOC:
- [[#developing][Developing]]
  - [[#where-to-get-it][Where to get it]]
  - [[#build-requirements][Build requirements]]
  - [[#gnulib][Gnulib]]
  - [[#how-to-build][How to build]]
  - [[#install-with-homebrew][Install with Homebrew]]
  - [[#docker][Docker]]
  - [[#how-to-run-tests][How to run tests]]
  - [[#how-to-debug][How to debug]]
  - [[#how-to-find-memory-leaks][How to find memory leaks]]
  - [[#static-analysis][Static analysis]]
  - [[#how-to-analyze-performance][How to analyze performance]]
  - [[#how-to-build-for-windows][How to build for Windows]]
- [[#release-procedure][Release Procedure]]
  - [[#update-the-news-entry][Update the NEWS entry]]
  - [[#set-the-version-number][Set the version number]]
  - [[#commit-and-tag][Commit and tag]]
  - [[#make-the-release][Make the release]]
  - [[#upload][Upload]]
  - [[#online-documentation][Online documentation]]
  - [[#web-site-maintenance][Web site maintenance]]
  - [[#announce][Announce]]

* Developing
** Where to get it
The development sources are available through git at github.com:

  https://github.com/liblouis/liblouis

** Build requirements
To build Automake, Autoconf, and Libtool are used. If you are getting
the sources from git (or change configure.ac), you'll need to have
these tools installed to (re)build. Optionally (if you want to
generate man pages) you'll also need ~help2man~. All of these programs
are available from ftp://ftp.gnu.org/gnu.

If you want to run the YAML based test suite you will have to install
~libyaml~.

On Mac OS, the programs can be optained with Homebrew (http://brew.sh):

#+BEGIN_SRC shell
$ brew install automake libtool pkg-config texinfo
#+END_SRC

Note that if you are using Homebrew to install liblouis (see below),
the build dependencies are installed automatically.

** Gnulib
Gnulib (http://www.gnu.org/software/gnulib) is used to provide
portable basic functionality to programs and libraries. We use two
instances of gnulib, one to provide portable functions such as
~malloc~, ~strndup~, etc to the library and another one to provide
portable functionality such as ~getopt~, ~progname~ or
~version-etc~ to the tools.

The first time invocation to import gnulib for the library was

#+BEGIN_SRC shell
$ gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files \
	      malloc-gnu realloc-gnu strndup
#+END_SRC

and for the tools

#+BEGIN_SRC shell
$ gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files \
	      getopt-gnu malloc-gnu progname version-etc
#+END_SRC

More modules might have been added since. The currently-used gnulib
modules and other gnulib information are recorded in
~gnulib/m4/gnulib-cache.m4~ and ~tools/gnulib/m4/gnulib-cache.m4~.

If you want to update from the current gnulib, install gnulib, and
then run the following commands in the top-level directory.

#+BEGIN_SRC shell
$ gnulib-tool --add-import --lib=libgnu --source-base=gnulib \
              --m4-base=gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl --no-vc-files
$ gnulib-tool --add-import --lib=libgnutools --source-base=tools/gnulib \
              --m4-base=tools/gnulib/m4 --aux-dir=build-aux --libtool \
	      --macro-prefix=gl_tools --no-vc-files
#+END_SRC

** How to build
After getting the sources from git, with 

#+BEGIN_SRC shell
$ git clone https://github.com/liblouis/liblouis.git
#+END_SRC

and installing the tools above, change to the liblouis directory and
and bootstrap the project with the following command

#+BEGIN_SRC shell
$ ./autogen.sh
#+END_SRC

to do a fresh build. Then run configure as usual:

#+BEGIN_SRC shell
$ ./configure
#+END_SRC

You have the choice to compile liblouis for either 16- or 32-bit
Unicode. By default it is compiled for the former. To get 32-bit
Unicode run configure with ~--enable-ucs4~.

After running configure run ~make~ and then ~make install~. You must
have root privileges for the installation step.

** Install with Homebrew
Homebrew (http://brew.sh) is a package manager for Mac OS X that
installs software from source. There is nothing special about the
installation process in the sense that under the hood it happens
exactly as described above, with the only difference that Homebrew
automates it completely.

First, use the ~brew tap~ command to add the repository that includes
the liblouis formula:

#+BEGIN_SRC shell
$ brew tap liblouis/liblouis
#+END_SRC

Now you are ready to install liblouis:

#+BEGIN_SRC shell
$ brew install liblouis
#+END_SRC

** Docker
Docker (https://www.docker.com) can be useful both for creating a
development environment for liblouis, and for shipping the
application.

Setting up a developer environment can take long and can be
problematic especially for Windows. Thanks to Docker we can set up the
environment for you, we can easily distribute it as an image, which
can be run by anybody and will behave exactly the same for everybody.

Docker images of liblouis are being built automatically each time
something changes in the code (see https://registry.hub.docker.com/repos/liblouis).
In order to use them, first get Docker at
http://docs.docker.com/introduction/get-docker. Download the latest
liblouis image with:

#+BEGIN_SRC shell
$ docker pull liblouis/liblouis
#+END_SRC

Then, enter the development environment by running the image in a
Docker container:

#+BEGIN_SRC shell
$ docker run -it liblouis/liblouis bash
#+END_SRC

Local files and directories can be "mounted" inside the container, in
order to make it easier to edit files and to persist changes across
runs. For example, to use local table files:

#+BEGIN_SRC shell
$ docker run -it -v $(pwd)/tables:/tmp/liblouis/tables liblouis/liblouis bash
#+END_SRC

See the Docker documentation for more info.

The same Docker image can be used as a development environment and as
the application itself. For example, to run the lou_translate tool
from inside a Docker container:

#+BEGIN_SRC shell
$ docker run -it liblouis/liblouis lou_translate en-us-g1.ctb
#+END_SRC

To rebuild the image yourself, run the following command in the root
directory of the liblouis source:

#+BEGIN_SRC shell
$ docker build -t liblouis/liblouis .
#+END_SRC

A ~.dockerignore~ file is required if you want to compile the source
both on te host and in the Docker container. The ~.dockerignore~ file
can be updated from ~.gitignore~ with:

#+BEGIN_SRC shell
$ make .dockerignore
#+END_SRC

** How to run tests
Tests are run with

#+BEGIN_SRC shell
$ make check
#+END_SRC

** How to debug
First you have to build liblouis with debugging info enabled.

#+BEGIN_SRC shell
$ ./configure CFLAGS='-g -O0 -Wall -Wextra'
$ make
#+END_SRC

Starting the programs under the tools directory within gdb is a little
tricky as they are linked with libtool. See the info page of libtool
for more information. To start ~lou_checktable~ for table
~wiskunde.ctb~ for example you'd have to issue the following commands:

#+BEGIN_SRC shell
$ libtool --mode=execute gdb ./tools/lou_checktable
(gdb) run tables/wiskunde.ctb
#+END_SRC

** How to find memory leaks
Valgrind is a tool that can be used to find memory errors. It is
recommended that you compile liblouis without any optimizations and
with all warnings enabled before running it through Valgrind:

#+BEGIN_SRC shell
$ ./configure CFLAGS='-g -O0 -Wall'
$ make
#+END_SRC

Then use Valgrind to analyze liblouis. For example you can run
~lou_translate~ trough Valgrind:

#+BEGIN_SRC shell
$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --leak-resolution=high --log-file=valgrind.log \
  ./tools/lou_translate en-us-g2.ctb
#+END_SRC

Type a few words at the prompt, check translation and terminate
~lou_translate~. Now open the file ~valgrind.log~ and see if there are
any memory leaks reported.

You can also just run lou_checktable for example:

#+BEGIN_SRC shell
$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --leak-resolution=high --log-file=valgrind.log \
  ./tools/lou_checktable tables/nl-BE-g0.utb
#+END_SRC

Again open valgrind.log to see if any memory leaks were reported.

For the full experience run lou_allround under Valgrind:

#+BEGIN_SRC shell
$ libtool --mode=execute valgrind -v --tool=memcheck \
  --leak-check=full --show-reachable=yes \
  --leak-resolution=high --track-origins=yes \
  --log-file=valgrind.log ./tools/lou_allround
#+END_SRC

*** AddressSanitizer
[[https://github.com/google/sanitizers/wiki/AddressSanitizer][AdressSanitizer]] is a memory error detector for C/C++. It is part of
both LLVM and gcc. To check liblouis build it as follows:

#+BEGIN_SRC shell
$ ./autogen.sh
$ ./configure CFLAGS='-fsanitize=address -O1 -fno-omit-frame-pointer -g'
$ make
$ ./tools/lou_translate tables/en-ueb-g2.ctb
#+END_SRC

Run a few translations, end the program and marvel at the output of
AddressSanitizer. At the time of this writing it complained pretty
hard:

#+BEGIN_EXAMPLE
SUMMARY: AddressSanitizer: 12384 byte(s) leaked in 12 allocation(s).
#+END_EXAMPLE

*** Electric Fence
Electric Fence is a tool that helps detect memory access that overruns
the boundaries of a ~malloc()~ memory allocation, and access to memory
that has been released with ~free()~.

Under Debian the usage is fairly straightforward:

#+BEGIN_SRC shell
$ sudo apt install sudo apt install electric-fence
#+END_SRC

Then compile [[#how-to-debug][as above]] and invoke the debugger [[#how-to-debug][as above]]. Inside ~gdb~
set up ~LOUIS_TABLEPATH~ and ~LD_PRELOAD~ as follows:

#+BEGIN_SRC shell
$ libtool --mode=execute gdb ./tools/lou_allround
(gdb) set environment LOUIS_TABLEPATH ./tables,./tests/tables,./tests/tables/moreTables,./tests/tablesWithMetadata,./tests/tables/emphclass
(gdb) set environment LD_PRELOAD /usr/lib/libefence.so.0.0
(gdb) run
#+END_SRC

If there are problems with memory access the program will run into a
segmentation fault which you can consequently analyze in the debugger.

** Static analysis
[[http://clang-analyzer.llvm.org/scan-build.html][scan-build]] is a command line utility to run a static analyzer over the
codebase. It helps to find problems such as memory leaks, dereference
of null pointer, etc.

#+BEGIN_SRC shell
$ make maintainer-clean
$ ./autogen.sh
$ scan-build ./configure
$ scan-build make
...
scan-build: 101 bugs found.
scan-build: Run 'scan-view /tmp/scan-build-2019-08-15-141827-5185-1' to examine bug reports.
#+END_SRC

then look at the generated reports and fix (or report) the issues.

** How to analyze performance
Gprof helps you analyze the performance of programs. You have to
compile liblouis as follows:

#+BEGIN_SRC shell
$ ./configure --disable-shared
$ make clean all CFLAGS='-g -O0 -pg' LDFLAGS='-all-static'
#+END_SRC

Then translate some stuff with a large table:

#+BEGIN_SRC shell
$ ./tools/lou_translate tests/tables/large.ctb 
#+END_SRC

Finally look at the call profile:

#+BEGIN_SRC shell
$ libtool --mode=execute gprof ./tools/lou_translate gmon.out
#+END_SRC

** How to build for Windows
See the ~README.windows~ file and the windows subdirectory.

*** How to cross-compile for Windows
To compile for either win32 or win64 two Dockerfiles are provided
which set up the right environment to cross-compile liblouis. There is
a Makefile target that will build the Docker images, build liblouis
and extract the zip files with the windows build artifacts.

To build for both win32 and win64 type the following command:

#+BEGIN_SRC shell
$ make distwin
#+END_SRC

This will place ~liblouis-$(VERSION)-win32.zip~ and
~liblouis-$(VERSION)-win64.zip~ in the current directory. The build is
compiled with ~--enable-ucs4~ and contains ~lou_checkyaml.exe~.

*** TODO How to build for Windows using Cygwin
(possibly use a Vagrantfile as demonstration + explain that Cygwin
binaries can not be used outside the Cygwin environment)

*** TODO How to build for Windows using MinGW
(possibly use a Vagrant file as demonstration)

* Release Procedure
These steps describe what a maintainer does to make a release; they
are not needed for ordinary patch submission.

** Update the NEWS entry
If any new tables were added, renamed or removed please note them with
their file name in the NEWS entry. This is usefull for projects like
NVDA.

** Set the version number
Update the version number in ~NEWS~ (with version, date, and release
type), ~configure.ac~ and ~windows/include/config.h~.

Don't forget to update the libtool versioning info in ~configure.ac~,
i.e. ~LIBLOUIS_REVISION~ and possibly ~LIBLOUIS_CURRENT~ and
~LIBLOUIS_AGE~. Check the [[https://www.gnu.org/software/libtool/manual/html_node/Interfaces.html#Interfaces][libtool documentation on versioning]] for an
explanation of these values.

** Commit and tag
Commit the changes and tag this version

#+BEGIN_SRC shell
$ git tag -s v2.6.0 -m "Release 2.6.0"
$ git push origin v2.6.0
#+END_SRC

If you know the exact version number that needs to be tagged use

#+BEGIN_SRC shell
$ git tag -s v2.6.0 -m "Release 2.6.0" <commit>
$ git push origin v2.6.0
#+END_SRC

** Make the release
Check out a clean copy in a different directory, like /tmp. Run
autogen.sh and configure with no special prefixes. Run make distcheck.
This will make sure that all needed files are present, and do a
general sanity check. Run make dist. This will produce a tarball.

#+BEGIN_SRC shell
$ ./autogen.sh && ./configure && make && make distcheck && make dist
#+END_SRC

** Upload
Add the tarball to the github liblouis releases page, i.e. add it
under https://github.com/liblouis/liblouis/releases with the specific
release and add a link to it in $WEBSITE/downloads/index.md. See below
for instructions on how to update the web site.

** Online documentation
The online documentation is part of the liblouis web site. To add it to the
site simply copy doc/liblouis.html to $WEBSITE/documentation/liblouis.html.
Make sure you add the proper YAML front matter. Again see below for
instructions on how to update the web site.

** Web site maintenance
The liblouis web site at liblouis.org is maintained with the help of
[[https://pages.github.com/][github pages]]. To edit the site just check out the repo at
https://github.com/liblouis/liblouis.github.io. You'll need to know a
few things about [[http://jekyllrb.com/][Jekyll]] and markdown, the markup that is used to edit
the content. In order to update the site simply edit, commit and push.

For the new release update the project web site.

- Add a post containing the current NEWS to the _posts directory and
- add the download artifacts to the download page

** Announce
Send an announcement to the liblouis list
~liblouis-liblouisxml@freelists.org~. See ~ANNOUNCEMENT~ for an
example.