Contents |
TAE help
To know about various TAE options, you can use "tae --help". This provides detailed information about all the TAE options and their usage.
# <TAEBASE>/tae/tae --help
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~ OpenClovis Test Automation Environment (TAE) - Version 0.8 ~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Usage:
tae [options] [command]
Options:
-h, --help This help page
fetch
-f, --force Force action despite stamp shows it has been completed in
previous session, but executing unfinished prerequisites
-i, --ignore Ignore prerequisites. Same as -f (--force), but it does not
check for prerequisites. This should be used only if you
specifically want to skip prerequisites.
-q, --quiet Be more quiet (can be issued multiple times)
-v, --verbose Be more verbose (can be issued multiple times)
-s, --suppress
Do not show testcase error details on TAE console output
-u, --unload-tipc
Unload the tipc kernel module before starting SAFplus Platform during the
start stage
-b, --no-nodeinfo
Do not show nodeinfo/bladeinfo for each fixture node. By
default this information is requested and printed for each node
-w <cols>, --width=<cols>
Use this width instead of the actual width of the terminal
when printing results to output
-c, --clean Run a 'make clean' before configure
-r, --revert Run an 'svn revert' on any SVN-based images before running
'svn update' on the tree
--show-config Shows all the config attributes parsed from the config files
and then quits
-o <variable>=[<value>]
Allow manual override of a config attribute
-d Start tae in debug mode. Any Python error will launch a JIT
Python debug (pdb) session, allowing quick code analyzis.
This applies to Python errors in testcases too, which would
otherwise let TAE to continue
-n, --netid Manual override of the TIPC netid that will be used on the
target
-p <port>, --gms-port=<port>
Manual override of the GMS port number used on the target
-m <mcast-ip>, --gms-mcast=<mcast-ip>
Manual override of the GMS multicast IP address
-L, --list List test cases in model
-l <file>, --log=<file>
Put detailed tae log here (default is log/tae.log)
-t <filter>, --testcase-filter=<filter>
Execute test cases matching this filter expression only (by
default all testcases in model are executed). A filter
expression is a testcase identifier, with wildcard characters
also allowed. Examples:
TAE-TST-ACC.TC001
TAE-TST-ACC.TC0??
*-ACC*
*TC001
-T <file>, --testcase-file=<file>
Execute test cases specified in this file. The first word of
each line is treated as a filter expression which will be
matched against testcase ids. Lines started with a comment
character (#) are ignored.
-C <config-file>, --config=<file>
Config file to define the model, fixture, mapping, and other
operational parameters. This information can be provided in
one single file, or can be separated into multiple config
files. To read more than one config file this option can be
used repeatedly. If the same configuration attribute is defined
in more than one file, the last read file takes precedence.
If no config file is specified, tae will attempt to
read all local files with '*.cfg' extensions in alphabetical
order.
Deprecated config file options:
The following four options are still allowed, but their use is deprecated.
Use the -C option instead to read any and all config files.
-E <file>, --environment=<file>
Environment definition config file.
-F <file>, --fixture=<file>
Test fixture config file.
-M <file>, --model=<file>
Model definition config file.
-P <file>, --mapping=<file>
Model node name to fixture node mapping config.
Command:
Command describes what needs to be done in terms of the sequential stages
of a full test run. The following stages are defined (in the order of
progression):
fetch, populate, configure, build, package, deploy, start, test, stop
The following forms are allowed as command:
<stage> Run up to this stage, doing all prerequisites (any previous
stages that were not done yet)
When no command is specified, the last stage (report) is assumed to be the
target stage.
When combined with the -f (--force) option, the command can be in either of
the following forms:
<stage> Redo this stage even if it has been done before. Also do all
prerequisite stages if they have not been completed yet
<stage> Redo all stages from fetch to the named stage, even if all or
some have been done previously.
<stage>: Starting from named stage, redo everything
<st1>:<st2> Redo everything from stage st1 through stage st2
When the -i (--ignore) option used, TAE does not check for prerequisites,
but attempt to run the specified stage or stages. It implies the -f (--force)
option.
The meanings of these stages are:
fetch Fetch model for purpose of test case extract
populate Grab the images and populate working dirs for build
configure Configure the project for target fixture
build Build project for target fixture
package Package project for target fixture deployment
deploy Deploy images to target fixture
start Bring up SAFplus Platform on target fixture
test Run all or selected tests
stop Shut down SAFplus Platform on target fixture
TAE configuration files
To run TAE with your model, you need to create a directory (rest of the document will refer to this directory using the symbol <TAECFGDIR>). It is generally a good idea to create separate <TAECFGDIR> for testing different models. A <TAECFGDIR> should contain at least following configuration files (these are just sample files, you need to change the configuration according to your requirement):
mapping.cfg
<!-- This file maps SAFplus Platform model node names to fixture node names -->
<map_config ver="1.0.0.0">
<mapping>
<SCNode0 node="node1" role="controller" order="1" asp_dir="/root/simulation/asp1"/>
<SCNode1 node="node2" role="controller" order="2" asp_dir="/root/simulation/asp2"/>
<WorkerNode0 node="node3" role="worker" order="3" asp_dir="/root/simulation/asp3"/>
<WorkerNode1 node="node4" role="worker" order="4" asp_dir="/root/simulation/asp4"/>
<WorkerNode2 node="node5" role="worker" order="5" asp_dir="/root/simulation/asp5"/>
<WorkerNode3 node="node6" role="worker" order="6" asp_dir="/root/simulation/asp6"/>
<WorkerNode4 node="node7" role="worker" order="7" asp_dir="/root/simulation/asp7"/>
<WorkerNode5 node="node8" role="worker" order="8" asp_dir="/root/simulation/asp8"/>
</mapping>
<gms mcast_port="8788" />
<tipc netid="8239" />
</map_config>
- Where SCNode0, SCNode1, WorkerNode0, etc are the node instances of the model. You also need to mention the role of the node instance. The "order" field is used to set the order of SAFplus Platform bring up on the nodes. "mcast_port" in gms tag can be used to override the value of "GMS_MCAST_PORT" present in "<model>/src/target.conf" Similarly "netid" in tipc tag can be used to override the value of "TIPC_NETID" (these values will be overridden in the "package" stage of TAE)
- asp_dir attribute is required only when TAE has to run in simulation setup(It deploys the asp images to specified directory), otherwise this attribute has to be removed.
fixture.cfg
<fixture_config ver="1.0.0.0">
<build_server ip='localhost' os='kubuntu' user='build_user' password='password' />
<nodes>
<node1 ip='10.10.3.10' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node2 ip='10.10.3.20' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node3 ip='10.10.3.30' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node4 ip='10.10.3.40' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node5 ip='10.10.3.50' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node6 ip='10.10.3.60' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node7 ip='10.10.3.90' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
<node8 ip='10.10.3.100' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
</nodes>
<description>
HP 14-slot chassis with MPCBL0001 cards
</description>
<lockfile value="/home/build_user/run_tae/test_on_hp/hp_chassis.lock" />
<!--simulation value="yes"/-->
</fixture_config>
- Normally the build server (where the SAFplus Platform code along with the model is compiled) is localhost (the same machine from where TAE is being run). One can use any other machine on the network for this purpose by specifying the ip, username and password of that machine. <nodes> tag contains the information about the machines where SAFplus Platform will be running during the test run.
- "crossbuild" attribute in node<number> tag is used for the purpose of building the SAFplus Platform binaries for the machine which is architecturally different than the build machine. The value of "crossbuild" attribute should be the name of the toolchain present for that kind of achitecture (located in <sdk-dir>/buildtools/"). If architecture of build and target machine are same, then "crossbuild" field has to be removed.
- lockfile tag can be used to maintain the integrity of the test. Once specified it doesn't allow any other test to use the same fixture. This way you can avoid interrupting an already running test on the same fixture.
- simulation value is used only when tae has to be run in simulation mode i.e. more than one node on same machine.
model.cfg
<model_config ver="1.0.0.0">
<project value="YOUR_PROJECT_NAME" />
<models>
<amsTestGeneric>
<image_source value="dir:~/run_tae/test_on_hp/asptest3.1/" />
<!--image_source value="svn://<svn-user>:<password>@clovisforge.openclovis.org:8888/svn/asptest/branches/3.1" /-->
<make_options value="AMS_PERF_TEST=1" />
</amsTestGeneric>
</models>
<asp>
<image_source value="dir:~/run_tae/test_on_hp/clasp3.1/" />
<!--image_source value="http://<ip>/<path-to-the-tarball>/openclovis-sdk-3.1-latest.tar.gz" /-->
<!--image_source value="svn://<svn-user>:<password>@clovisforge.openclovis.org:8888/svn/clasp/branches/3.1/maint/" /-->
</asp>
<testcase_dir value="~/run_tae/test_on_hp/asptest3.1" />
<buildserver>
<sdk_root_dir value="/opt/clovis" />
<asp_dir value="~/run_tae/test_on_hp/clasp3.1" />
<project_dir value="~/run_tae/test_on_hp/asptest3.1" />
</buildserver>
<!--skip_stages value="start" /-->
</model_config>
- This file contains the model specific configurations like model name, make options, source code directory etc. In the above sample file "amsTestGeneric" is the model name.
- Below are the description of all the tags in the model.cfg file:
- <image_source>: In this attribute, we have to specify the location of SAFplus Platform source code and model directory. This can be any directory on build server (In the example it is ~/run_tae/test_on_hp/clasp3.1/ directory on build server) or it can be the path from where we have to checkout from SVN or any other repository or it can be any link on internet or ftp details.
- <make_options>: In this tag we provide all the flags with which we have to compile the code. In the example above it is AMS_PERF_TEST=1. So, code will be compiled with "make AMS_PERF_TEST=1".
- <testcase_dir>: It is the directory where all test cases of the model will be present.
- <sdk_root_dir>: In this we specify the directory where sdk is installed.
- <asp_dir>: In this we specify the directory where SAFplus Platform is checked out.
- <project_dir>: In this we specify the directory where model is checked out.
- <skip_stages>: By specifying this we can skip some of the steps of complete tae run.
- There are many ways to specify the location of SAFplus Platform source code. "dir:" can be used to specify the local directory as the source code directory for SAFplus Platform in the <asp> <image_source> tag and <model-name><image_source> tag for the model directory.
env.cfg
<env_config ver="1.0.0.0">
<svn>
<timeout>
<checkout value='10000' />
<update value='10000' />
<revert value='3000' />
<status value='6000' />
</timeout>
</svn>
<report_url value="scp://<user>:<password>@<ip>/<taereport-dir>/import" />
<!--report_url value="scp://testuser:test@192.168.0.100/taereport/import" /-->
</env_config>
- This file contains the timeout values for various TAE operations. After the test run TAE will send the report tarball to the machine specified in <report_url>. Below are the list of all tags used in this config file:
- checkout: It is time in seconds for which tae will wait for checkout to happen before timing out.
- update: It is time in seconds for which tae will wait for svn update to complete before timing out.
- revert: It is time in seconds for which tae will wait for svn revert before timing out.
- status: It is time in seconds for which tae will wait for svn status before timing out.
- report_url: specify the command to upload the test results to the TAE Report Server (to the web GUI). Comment this line out if you do not want your test results to be submitted.
Selecting and running the tests
If following command is issued from the <TAECFGDIR> directory, it lists all the test cases present in the model.
# <TAEBASE>/tae/tae -L
#---------------------------------------------------------------------- # ID State Brief description #---------------------------------------------------------------------- AMF-CMP-KIL.TC001 E Active Comp Kill Test : Verify SI/CSI assignments after fail-over AMF-CSI-DEP.TC001 E CSI Dependency Test : Verify CSI assignments order after switch-over AMF-HA-TST.TC001 E SI Swap Test : Verify SI/CSI assignments after switch-over AMF-HA-TST.TC002 E SU Lock Test : Verify SI/CSI assignments after switch-over AMF-NOD-LCK.TC001 E Node Lock Test : Verify SI/CSI assignments after switch-over AMF-RDN-PRC.TC001 E AMF Reduction Procedure Test : Verify SI/CSI assignments in the reduced mode AMF-SU-SDN.TC001 E SU Shutdown Test : Verify SI/CSI assignments after switch-over
you can then put some of the test cases in a file (say tests.lst) and specify the test-case-filter-file name while running tae using -T switch
# <TAEBASE>/tae/tae -T tests.lst -f fetch:stop
- Note that the above command should be issued from <TAECFGDIR> directory for the model
- It will execute all the steps from fetch to stop (fetch, populate, configure, build, package, deploy, start, test, stop).
- Some of the steps can be skipped by specifying the stage-name in <skip_stages value="" /> tag present in "model.cfg" file. Alternatively, you can also specify something like "-f build:test" if the requirement is to build the model from already existing code base and then run the test cases.
- If neither of "-t" or "-T" switch is specified, then TAE runs all the test cases present in the "<model>/src/test" directory. One can also use "-t *" to run all the test cases. Remember, the regular expression syntax is supported while specifying the filter for testcases, which can be very handy sometime.
TAE logs
TAE log files will be present in "log" directory of your model <TAECFGDIR>. Following log files are created by TAE for various different purposes:
- tae.log : contains detailed log of all the commands executed by the TAE while performing the test.
- bash_127.0.0.1_build_server.log : Contains the logs of commands executed during the SAFplus Platform and model build. It also contains the "make" output in case of build error.
- bash_local.log : Contains the commands performed on the local machine like scp the tarball to the remote nodes etc.
- Node level postmortem files (e.g. SysCtrlI0.postmortem.tgz) : Contains the node level logs as well as the runtime files of the node.