blob: 283646aef4c765e750179434d64a775dcd41fe03 [file] [log] [blame]
Eric Fiselierd720d1f2015-08-22 19:40:49 +00001==============
2Testing libc++
3==============
4
5.. contents::
6 :local:
7
8Getting Started
9===============
10
Louis Dionnef58d8292020-03-11 17:03:00 -040011libc++ uses LIT to configure and run its tests.
Marshall Clowca321972019-09-05 00:38:36 +000012
Louis Dionne089ced02020-04-07 15:02:37 -040013The primary way to run the libc++ tests is by using ``make check-cxx``.
Marshall Clowca321972019-09-05 00:38:36 +000014
15However since libc++ can be used in any number of possible
16configurations it is important to customize the way LIT builds and runs
17the tests. This guide provides information on how to use LIT directly to
18test libc++.
Eric Fiselierd720d1f2015-08-22 19:40:49 +000019
20Please see the `Lit Command Guide`_ for more information about LIT.
21
Sylvestre Ledrub09c9382020-03-22 22:42:03 +010022.. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
Eric Fiselierd720d1f2015-08-22 19:40:49 +000023
Louis Dionne089ced02020-04-07 15:02:37 -040024Usage
25-----
Eric Fiselierd720d1f2015-08-22 19:40:49 +000026
Louis Dionne089ced02020-04-07 15:02:37 -040027After building libc++, you can run parts of the libc++ test suite by simply
Louis Dionnee35df792020-04-15 15:05:14 -040028running ``llvm-lit`` on a specified test or directory. If you're unsure
Louis Dionne3efccd72020-10-23 09:32:50 -040029whether the required libraries have been built, you can use the
Louis Dionne4eaeee42022-08-02 16:15:55 -040030``cxx-test-depends`` target. For example:
Eric Fiselier4ac88092015-10-14 20:44:44 +000031
32.. code-block:: bash
33
Louis Dionne089ced02020-04-07 15:02:37 -040034 $ cd <monorepo-root>
Louis Dionne10735312021-01-12 10:56:57 -050035 $ make -C <build> cxx-test-depends # If you want to make sure the targets get rebuilt
Louis Dionne089ced02020-04-07 15:02:37 -040036 $ <build>/bin/llvm-lit -sv libcxx/test/std/re # Run all of the std::regex tests
37 $ <build>/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test
38 $ <build>/bin/llvm-lit -sv libcxx/test/std/atomics libcxx/test/std/threads # Test std::thread and std::atomic
Eric Fiselier4ac88092015-10-14 20:44:44 +000039
Louis Dionne4e6ca9b2022-08-03 09:24:54 -040040.. note::
41 If you used the Bootstrapping build instead of the default runtimes build, the
42 ``cxx-test-depends`` target is instead named ``runtimes-test-depends``, and
43 you will need to prefix ``<build>/runtimes/runtimes-<target>-bins/`` to the
44 paths of all tests.
45
Louis Dionne94870d82020-06-26 12:08:59 -040046In the default configuration, the tests are built against headers that form a
47fake installation root of libc++. This installation root has to be updated when
Louis Dionne4eaeee42022-08-02 16:15:55 -040048changes are made to the headers, so you should re-run the ``cxx-test-depends``
49target before running the tests manually with ``lit`` when you make any sort of
Louis Dionne94870d82020-06-26 12:08:59 -040050change, including to the headers.
51
Eric Fiselier4ac88092015-10-14 20:44:44 +000052Sometimes you'll want to change the way LIT is running the tests. Custom options
Louis Dionne4eaeee42022-08-02 16:15:55 -040053can be specified using the ``--param <name>=<val>`` flag. The most common option
54you'll want to change is the standard dialect (ie ``-std=c++XX``). By default the
Eric Fiselier4ac88092015-10-14 20:44:44 +000055test suite will select the newest C++ dialect supported by the compiler and use
Louis Dionne4eaeee42022-08-02 16:15:55 -040056that. However, you can manually specify the option like so if you want:
Eric Fiselier4ac88092015-10-14 20:44:44 +000057
58.. code-block:: bash
59
Louis Dionne089ced02020-04-07 15:02:37 -040060 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers # Run the tests with the newest -std
Louis Dionne4eaeee42022-08-02 16:15:55 -040061 $ <build>/bin/llvm-lit -sv libcxx/test/std/containers --param std=c++03 # Run the tests in C++03
Eric Fiselier4ac88092015-10-14 20:44:44 +000062
Louis Dionne4eaeee42022-08-02 16:15:55 -040063Other parameters are supported by the test suite. Those are defined in ``libcxx/utils/libcxx/test/params.py``.
64If you want to customize how to run the libc++ test suite beyond what is available
65in ``params.py``, you most likely want to use a custom site configuration instead.
Eric Fiselier4ac88092015-10-14 20:44:44 +000066
Louis Dionne4eaeee42022-08-02 16:15:55 -040067The libc++ test suite works by loading a site configuration that defines various
68"base" parameters (via Lit substitutions). These base parameters represent things
69like the compiler to use for running the tests, which default compiler and linker
70flags to use, and how to run an executable. This system is meant to be easily
71extended for custom needs, in particular when porting the libc++ test suite to
72new platforms.
Eric Fiselier4ac88092015-10-14 20:44:44 +000073
Louis Dionne4eaeee42022-08-02 16:15:55 -040074Using a Custom Site Configuration
Louis Dionne089ced02020-04-07 15:02:37 -040075---------------------------------
76
77By default, the libc++ test suite will use a site configuration that matches
78the current CMake configuration. It does so by generating a ``lit.site.cfg``
Louis Dionne6da743b2020-08-29 17:13:02 -040079file in the build directory from one of the configuration file templates in
80``libcxx/test/configs/``, and pointing ``llvm-lit`` (which is a wrapper around
81``llvm/utils/lit/lit.py``) to that file. So when you're running
82``<build>/bin/llvm-lit``, the generated ``lit.site.cfg`` file is always loaded
83instead of ``libcxx/test/lit.cfg.py``. If you want to use a custom site
84configuration, simply point the CMake build to it using
85``-DLIBCXX_TEST_CONFIG=<path-to-site-config>``, and that site configuration
86will be used instead. That file can use CMake variables inside it to make
87configuration easier.
Louis Dionne089ced02020-04-07 15:02:37 -040088
89 .. code-block:: bash
90
Louis Dionne19608c62020-06-12 15:19:55 -040091 $ cmake <options> -DLIBCXX_TEST_CONFIG=<path-to-site-config>
Louis Dionne10735312021-01-12 10:56:57 -050092 $ make -C <build> cxx-test-depends
Louis Dionne19608c62020-06-12 15:19:55 -040093 $ <build>/bin/llvm-lit -sv libcxx/test # will use your custom config file
Louis Dionne089ced02020-04-07 15:02:37 -040094
Nikolas Klausera9ad6702022-08-13 13:23:16 +020095Additional tools
96----------------
97
98The libc++ test suite uses a few optional tools to improve the code quality.
99
100These tools are:
Nikolas Klauserbec39912023-01-09 03:01:26 +0100101- clang-tidy (you might need additional dev packages to compile libc++-specific clang-tidy checks)
Nikolas Klausera9ad6702022-08-13 13:23:16 +0200102
Louis Dionne9e7cf262020-04-02 17:14:45 -0400103Writing Tests
104-------------
105
106When writing tests for the libc++ test suite, you should follow a few guidelines.
107This will ensure that your tests can run on a wide variety of hardware and under
108a wide variety of configurations. We have several unusual configurations such as
109building the tests on one host but running them on a different host, which add a
110few requirements to the test suite. Here's some stuff you should know:
111
112- All tests are run in a temporary directory that is unique to that test and
113 cleaned up after the test is done.
114- When a test needs data files as inputs, these data files can be saved in the
Marek Kurdej0dd96e52020-10-29 14:39:09 +0100115 repository (when reasonable) and referenced by the test as
Louis Dionne9e7cf262020-04-02 17:14:45 -0400116 ``// FILE_DEPENDENCIES: <path-to-dependencies>``. Copies of these files or
117 directories will be made available to the test in the temporary directory
118 where it is run.
119- You should never hardcode a path from the build-host in a test, because that
120 path will not necessarily be available on the host where the tests are run.
121- You should try to reduce the runtime dependencies of each test to the minimum.
122 For example, requiring Python to run a test is bad, since Python is not
123 necessarily available on all devices we may want to run the tests on (even
124 though supporting Python is probably trivial for the build-host).
125
Eric Fiselier9e3e1372016-07-19 23:07:03 +0000126Benchmarks
127==========
128
129Libc++ contains benchmark tests separately from the test of the test suite.
130The benchmarks are written using the `Google Benchmark`_ library, a copy of which
131is stored in the libc++ repository.
132
133For more information about using the Google Benchmark library see the
134`official documentation <https://github.com/google/benchmark>`_.
135
136.. _`Google Benchmark`: https://github.com/google/benchmark
137
138Building Benchmarks
139-------------------
140
Eric Fiselier93f212c2016-08-29 19:50:49 +0000141The benchmark tests are not built by default. The benchmarks can be built using
142the ``cxx-benchmarks`` target.
Eric Fiselier9e3e1372016-07-19 23:07:03 +0000143
144An example build would look like:
145
146.. code-block:: bash
147
148 $ cd build
Shivam Guptae3cb5702021-10-02 07:35:15 +0530149 $ ninja cxx-benchmarks
Eric Fiselier9e3e1372016-07-19 23:07:03 +0000150
151This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be
152built against the just-built libc++. The compiled tests are output into
Shivam Guptae3cb5702021-10-02 07:35:15 +0530153``build/projects/libcxx/benchmarks``.
Eric Fiselier9e3e1372016-07-19 23:07:03 +0000154
155The benchmarks can also be built against the platforms native standard library
156using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This
157is useful for comparing the performance of libc++ to other standard libraries.
158The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and
159``<test>.native.out`` otherwise.
160
161Also See:
162
163 * :ref:`Building Libc++ <build instructions>`
164 * :ref:`CMake Options`
165
166Running Benchmarks
167------------------
168
169The benchmarks must be run manually by the user. Currently there is no way
170to run them as part of the build.
171
172For example:
173
174.. code-block:: bash
175
Shivam Guptae3cb5702021-10-02 07:35:15 +0530176 $ cd build/projects/libcxx/benchmarks
Eric Fiselier9e3e1372016-07-19 23:07:03 +0000177 $ ./algorithms.libcxx.out # Runs all the benchmarks
178 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks
179
180For more information about running benchmarks see `Google Benchmark`_.