1# Apache Thrift - integration test suite
2
3This is the cross everything integration test suite for Apache Thrift.
4
5## Run
6
7### A. Using Make
8
9The test can be executed by:
10
11    make cross
12
13This starts the [test.py](test.py) script which does the real cross test with
14different transports, protocols and languages.
15
16Note that this skips any language that is not built locally. It also skips
17tests that are known to be failing. If you need more control over which tests
18to run, read following section.
19
20### B. Using test script directly
21
22Alternatively, you can invoke [test.py](test.py) directly. You need to run`make
23precross` once before executing it for the first time.
24
25For example, if you changed something in `nodejs` library and need to verify
26the patch, you can skip everything except `nodejs` itself and some reference
27implementation (currently `cpp` and `java` are recommended) like this:
28
29    ./configure --without-c_glib --without-erlang --without-lua ...
30    make precross -j8
31    test/test.py --server cpp,java --client nodejs
32    test/test.py --server nodejs --client cpp,java
33
34Another useful flag is --regex. For example, to run all tests that involve
35Java TBinaryProtocol:
36
37    test/test.py --regex "java.*binary"
38
39## Test case definition file
40
41The cross test cases are defined in [tests.json](tests.json).
42The root element is collection of test target definitions.
43Each test target definition looks like this:
44
45    {
46      "name": "somelib",
47
48      "client": {
49        "command": ["somelib_client_executable"],
50        "workdir": "somelib/bin",
51        "protocols": ["binary"],
52        "transports": ["buffered"],
53        "sockets": ["ip"],
54      },
55      "server": {
56        "command": ["somelib_server_executable"],
57        "workdir": "somelib/bin",
58        "protocols": ["binary"],
59        "transports": ["buffered"],
60        "sockets": ["ip", "ip-ssl"],
61      }
62    }
63
64Either client or server definition or both should be present.
65
66Parameters that are common to both `client` and `server` can be put to target
67definition root:
68
69    {
70      "name": "somelib",
71
72      "workdir": "somelib/bin",
73      "protocols": ["binary"],
74      "transports": ["buffered"],
75      "sockets": ["ip"],
76
77      "client": { "command": ["somelib_client_executable"] },
78      "server": {
79        "command": ["somelib_server_executable"],
80        "sockets": ["ip-ssl"]
81      }
82    }
83
84For the complete list of supported keys and their effect, see source code
85comment at the opt of [crossrunner/collect.py](crossrunner/collect.py).
86
87
88## List of known failures
89
90Since many cross tests currently fail (mainly due to partial incompatibility
91around exception handling), the test script specifically report for "not known
92before" failures.
93
94For this purpose, test cases known to (occasionally) fail are listed in
95`known_failures_<platform>.json` where `<platform>` matches with python
96`platform.system()` string.
97
98Currently, only Linux version is included.
99
100FYI, the file is initially generated by
101
102    test/test.py --update-expected-failures=overwrite
103
104after a full test run, then repeatedly
105
106    test/test.py --skip-known-failures
107    test/test.py --update-expected-failures=merge
108
109to update the known failures, run
110
111    make fail
112
113## Test executable specification
114
115### Command line parameters
116
117Unit tests for languages are usually located under lib/<lang>/test/
118cross language tests according to [ThriftTest.thrift](ThriftTest.thrift) shall be
119provided for every language including executables with the following command
120line interface:
121
122**Server command line interface:**
123
124    $ ./TestServer -h
125    Allowed options:
126      -h | --help                  produce help message
127      --port=arg (9090)            Port number to listen
128      --domain-socket=arg          Unix Domain Socket (e.g. /tmp/ThriftTest.thrift)
129      --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
130      --server-type=arg (simple)   type of server, "simple", "thread-pool",
131                                   "threaded", or "nonblocking"
132      --transport=arg (buffered)   transport: buffered, framed, http, anonpipe, zlib
133      --protocol=arg (binary)      protocol: binary, compact, header, json
134      --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
135      --abstract-namespace         Create the domain socket in the Abstract Namespace
136                                   (no connection with filesystem pathnames)
137      --ssl                        Encrypted Transport using SSL
138      --zlib                       Wrapped Transport using Zlib
139      --processor-events           processor-events
140      -n=arg | --workers=arg (=4)  Number of thread pools workers. Only valid for
141                                   thread-pool server type
142
143**Client command line interface:**
144
145    $ ./TestClient -h
146    Allowed options:
147      -h | --help                  produce help message
148      --host=arg (localhost)       Host to connect
149      --port=arg (9090)            Port number to connect
150      --domain-socket=arg          Domain Socket (e.g. /tmp/ThriftTest.thrift),
151                                   instead of host and port
152      --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
153      --anon-pipes hRead hWrite    Windows Anonymous Pipes pair (handles)
154      --abstract-namespace         Create the domain socket in the Abstract Namespace
155                                   (no connection with filesystem pathnames)
156      --transport=arg (buffered)   Transport: buffered, framed, http, evhttp, zlib
157      --protocol=arg (binary)      Protocol: binary, compact, header, json
158      --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
159      --ssl                        Encrypted Transport using SSL
160      --zlib                       Wrap Transport with Zlib
161      -n=arg | --testloops=arg (1) Number of Tests
162      -t=arg | --threads=arg (1)   Number of Test threads
163
164If you have executed the **make check** or **make cross** then you will be able to browse
165[gen-html/ThriftTest.html](gen-html/ThriftTest.html) with the test documentation.
166
167### Return code
168
169The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors.
170In order to signal failed tests, the return code shall be composed from these bits to indicate
171failing tests:
172
173      #define TEST_BASETYPES     1  // 0000 0001
174      #define TEST_STRUCTS       2  // 0000 0010
175      #define TEST_CONTAINERS    4  // 0000 0100
176      #define TEST_EXCEPTIONS    8  // 0000 1000
177      #define TEST_UNKNOWN      64  // 0100 0000 (Failed to prepare environment etc.)
178      #define TEST_TIMEOUT     128  // 1000 0000
179      #define TEST_NOTUSED      48  // 0011 0000 (reserved bits)
180
181Tests that have not been executed at all count as errors.
182
183**Example:**
184
185During tests, the test client notices that some of the Struct tests fail.
186Furthermore, due to some other problem none of the Exception tests is executed.
187Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure
188of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS).
189
190
191## SSL
192Test Keys and Certificates are provided in multiple formats under the following
193directory [test/keys](keys)
194