Doc:latest/taeguide/runtae

Revision as of 18:18, 13 March 2013 by Stone (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


Contents

TAE help

To know about various TAE options, you can use "tae --help". This provides detailed information about all the TAE options and their usage.

# <TAEBASE>/tae/tae --help

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~  OpenClovis Test Automation Environment (TAE) - Version 0.8 ~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Usage:
  tae [options] [command]
Options:
  -h, --help    This help page
  fetch
  -f, --force   Force action despite stamp shows it has been completed in
                previous session, but executing unfinished prerequisites
  -i, --ignore  Ignore prerequisites. Same as -f (--force), but it does not
                check for prerequisites. This should be used only if you
                specifically want to skip prerequisites.
  -q, --quiet   Be more quiet (can be issued multiple times)
  -v, --verbose Be more verbose (can be issued multiple times)
  -s, --suppress
                Do not show testcase error details on TAE console output
  -u, --unload-tipc
                Unload the tipc kernel module before starting SAFplus Platform during the
                start stage
  -b, --no-nodeinfo
                Do not show nodeinfo/bladeinfo for each fixture node. By
                default this information is requested and printed for each node
  -w <cols>, --width=<cols>
                Use this width instead of the actual width of the terminal
                when printing results to output
  -c, --clean   Run a 'make clean' before configure
  -r, --revert  Run an 'svn revert' on any SVN-based images before running
                'svn update' on the tree
  --show-config Shows all the config attributes parsed from the config files
                and then quits
  -o <variable>=[<value>]
                Allow manual override of a config attribute
  -d            Start tae in debug mode. Any Python error will launch a JIT
                Python debug (pdb) session, allowing quick code analyzis.
                This applies to Python errors in testcases too, which would
                otherwise let TAE to continue
  -n, --netid   Manual override of the TIPC netid that will be used on the
                target
  -p <port>, --gms-port=<port>
                Manual override of the GMS port number used on the target
  -m <mcast-ip>, --gms-mcast=<mcast-ip>
                Manual override of the GMS multicast IP address
  -L, --list    List test cases in model
  -l <file>, --log=<file>
                Put detailed tae log here (default is log/tae.log)
  -t <filter>, --testcase-filter=<filter>
                Execute test cases matching this filter expression only (by
                default all testcases in model are executed). A filter
                expression is a testcase identifier, with wildcard characters
                also allowed. Examples:
                    TAE-TST-ACC.TC001
                    TAE-TST-ACC.TC0??
                    *-ACC*
                    *TC001
  -T <file>, --testcase-file=<file>
                Execute test cases specified in this file. The first word of
                each line is treated as a filter expression which will be
                matched against testcase ids. Lines started with a comment
                character (#) are ignored.
  -C <config-file>, --config=<file>
                Config file to define the model, fixture, mapping, and other
                operational parameters. This information can be provided in
                one single file, or can be separated into multiple config
                files. To read more than one config file this option can be
                used repeatedly. If the same configuration attribute is defined
                in more than one file, the last read file takes precedence.
                If no config file is specified, tae will attempt to
                read all local files with '*.cfg' extensions in alphabetical
                order.

  Deprecated config file options:
  The following four options are still allowed, but their use is deprecated.
  Use the -C option instead to read any and all config files.
  -E <file>, --environment=<file>
                Environment definition config file.
  -F <file>, --fixture=<file>
                Test fixture config file.
  -M <file>, --model=<file>
                Model definition config file.
  -P <file>, --mapping=<file>
                Model node name to fixture node mapping config.

Command:

  Command describes what needs to be done in terms of the sequential stages
  of a full test run. The following stages are defined (in the order of
  progression):

    fetch, populate, configure, build, package, deploy, start, test, stop

  The following forms are allowed as command:

    <stage>     Run up to this stage, doing all prerequisites (any previous
                stages that were not done yet)

  When no command is specified, the last stage  (report) is assumed to be the
  target stage.

  When combined with the -f (--force) option, the command can be in either of
  the following forms:

    <stage>     Redo this stage even if it has been done before. Also do all
                prerequisite stages if they have not been completed yet
    <stage>    Redo all stages from fetch to the named stage, even if all or
                some have been done previously.
    <stage>:    Starting from named stage, redo everything
    <st1>:<st2> Redo everything from stage st1 through stage st2

  When the -i (--ignore) option used, TAE does not check for prerequisites,
  but attempt to run the specified stage or stages. It implies the -f (--force)
  option.

  The meanings of these stages are:

    fetch       Fetch model for purpose of test case extract
    populate    Grab the images and populate working dirs for build
    configure   Configure the project for target fixture
    build       Build project for target fixture
    package     Package project for target fixture deployment
    deploy      Deploy images to target fixture
    start       Bring up SAFplus Platform on target fixture
    test        Run all or selected tests
    stop        Shut down SAFplus Platform on target fixture

TAE configuration files

To run TAE with your model, you need to create a directory (rest of the document will refer to this directory using the symbol <TAECFGDIR>). It is generally a good idea to create separate <TAECFGDIR> for testing different models. A <TAECFGDIR> should contain at least following configuration files (these are just sample files, you need to change the configuration according to your requirement):

mapping.cfg

<!-- This file maps SAFplus Platform model node names to fixture node names -->
<map_config ver="1.0.0.0">
  <mapping>
    <SCNode0  node="node1" role="controller" order="1" asp_dir="/root/simulation/asp1"/>
    <SCNode1  node="node2" role="controller" order="2" asp_dir="/root/simulation/asp2"/>
    <WorkerNode0  node="node3" role="worker" order="3" asp_dir="/root/simulation/asp3"/>
    <WorkerNode1  node="node4" role="worker" order="4" asp_dir="/root/simulation/asp4"/>
    <WorkerNode2  node="node5" role="worker" order="5" asp_dir="/root/simulation/asp5"/>
    <WorkerNode3  node="node6" role="worker" order="6" asp_dir="/root/simulation/asp6"/>
    <WorkerNode4  node="node7" role="worker" order="7" asp_dir="/root/simulation/asp7"/>
    <WorkerNode5  node="node8" role="worker" order="8" asp_dir="/root/simulation/asp8"/>
  </mapping>
  <gms mcast_port="8788" />
  <tipc netid="8239" />
</map_config>
  • Where SCNode0, SCNode1, WorkerNode0, etc are the node instances of the model. You also need to mention the role of the node instance. The "order" field is used to set the order of SAFplus Platform bring up on the nodes. "mcast_port" in gms tag can be used to override the value of "GMS_MCAST_PORT" present in "<model>/src/target.conf" Similarly "netid" in tipc tag can be used to override the value of "TIPC_NETID" (these values will be overridden in the "package" stage of TAE)
  • asp_dir attribute is required only when TAE has to run in simulation setup(It deploys the asp images to specified directory), otherwise this attribute has to be removed.

fixture.cfg

<fixture_config ver="1.0.0.0">
 <build_server ip='localhost' os='kubuntu' user='build_user' password='password' />
 <nodes>
  <node1 ip='10.10.3.10' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node2 ip='10.10.3.20' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node3 ip='10.10.3.30' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node4 ip='10.10.3.40' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node5 ip='10.10.3.50' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node6 ip='10.10.3.60' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node7 ip='10.10.3.90' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
  <node8 ip='10.10.3.100' os='wrs-pnele-1.4' crossbuild='i586-wrl-pnele1.4-2.6.14-mpcbl0001' user="root" password="password-of-this-node" />
 </nodes>
 <description>
    HP 14-slot chassis with MPCBL0001 cards
 </description>
 <lockfile value="/home/build_user/run_tae/test_on_hp/hp_chassis.lock" />
 <!--simulation value="yes"/-->
</fixture_config>
  • Normally the build server (where the SAFplus Platform code along with the model is compiled) is localhost (the same machine from where TAE is being run). One can use any other machine on the network for this purpose by specifying the ip, username and password of that machine. <nodes> tag contains the information about the machines where SAFplus Platform will be running during the test run.
  • "crossbuild" attribute in node<number> tag is used for the purpose of building the SAFplus Platform binaries for the machine which is architecturally different than the build machine. The value of "crossbuild" attribute should be the name of the toolchain present for that kind of achitecture (located in <sdk-dir>/buildtools/"). If architecture of build and target machine are same, then "crossbuild" field has to be removed.
  • lockfile tag can be used to maintain the integrity of the test. Once specified it doesn't allow any other test to use the same fixture. This way you can avoid interrupting an already running test on the same fixture.
  • simulation value is used only when tae has to be run in simulation mode i.e. more than one node on same machine.

model.cfg

<model_config ver="1.0.0.0">
<project value="YOUR_PROJECT_NAME" />
<models>
  <amsTestGeneric>
    <image_source value="dir:~/run_tae/test_on_hp/asptest3.1/" />
    <!--image_source value="svn://<svn-user>:<password>@clovisforge.openclovis.org:8888/svn/asptest/branches/3.1" /-->
    <make_options value="AMS_PERF_TEST=1" />
  </amsTestGeneric>
</models>

<asp>
  <image_source value="dir:~/run_tae/test_on_hp/clasp3.1/" />
  <!--image_source value="http://<ip>/<path-to-the-tarball>/openclovis-sdk-3.1-latest.tar.gz" /-->
  <!--image_source value="svn://<svn-user>:<password>@clovisforge.openclovis.org:8888/svn/clasp/branches/3.1/maint/" /-->
</asp>

<testcase_dir  value="~/run_tae/test_on_hp/asptest3.1" />

<buildserver>
    <sdk_root_dir value="/opt/clovis" />
    <asp_dir      value="~/run_tae/test_on_hp/clasp3.1" />
    <project_dir  value="~/run_tae/test_on_hp/asptest3.1" />
</buildserver>

<!--skip_stages value="start" /-->
</model_config>
  • This file contains the model specific configurations like model name, make options, source code directory etc. In the above sample file "amsTestGeneric" is the model name.
  • Below are the description of all the tags in the model.cfg file:
    • <image_source>: In this attribute, we have to specify the location of SAFplus Platform source code and model directory. This can be any directory on build server (In the example it is ~/run_tae/test_on_hp/clasp3.1/ directory on build server) or it can be the path from where we have to checkout from SVN or any other repository or it can be any link on internet or ftp details.
    • <make_options>: In this tag we provide all the flags with which we have to compile the code. In the example above it is AMS_PERF_TEST=1. So, code will be compiled with "make AMS_PERF_TEST=1".
    • <testcase_dir>: It is the directory where all test cases of the model will be present.
    • <sdk_root_dir>: In this we specify the directory where sdk is installed.
    • <asp_dir>: In this we specify the directory where SAFplus Platform is checked out.
    • <project_dir>: In this we specify the directory where model is checked out.
    • <skip_stages>: By specifying this we can skip some of the steps of complete tae run.
  • There are many ways to specify the location of SAFplus Platform source code. "dir:" can be used to specify the local directory as the source code directory for SAFplus Platform in the <asp> <image_source> tag and <model-name><image_source> tag for the model directory.

env.cfg

<env_config ver="1.0.0.0">
  <svn> 
    <timeout>
      <checkout value='10000' />
      <update   value='10000' />
      <revert   value='3000' />
      <status   value='6000' />
    </timeout> 
  </svn>
  <report_url value="scp://<user>:<password>@<ip>/<taereport-dir>/import" />
  <!--report_url value="scp://testuser:test@192.168.0.100/taereport/import" /-->
</env_config>
  • This file contains the timeout values for various TAE operations. After the test run TAE will send the report tarball to the machine specified in <report_url>. Below are the list of all tags used in this config file:
    • checkout: It is time in seconds for which tae will wait for checkout to happen before timing out.
    • update: It is time in seconds for which tae will wait for svn update to complete before timing out.
    • revert: It is time in seconds for which tae will wait for svn revert before timing out.
    • status: It is time in seconds for which tae will wait for svn status before timing out.
    • report_url: specify the command to upload the test results to the TAE Report Server (to the web GUI). Comment this line out if you do not want your test results to be submitted.

Selecting and running the tests

If following command is issued from the <TAECFGDIR> directory, it lists all the test cases present in the model.

        # <TAEBASE>/tae/tae -L
	#----------------------------------------------------------------------
	# ID              State  Brief description
	#----------------------------------------------------------------------
	AMF-CMP-KIL.TC001   E    Active Comp Kill Test : Verify SI/CSI assignments after fail-over
	AMF-CSI-DEP.TC001   E    CSI Dependency Test : Verify CSI assignments order after switch-over
	AMF-HA-TST.TC001    E    SI Swap Test : Verify SI/CSI assignments after switch-over
	AMF-HA-TST.TC002    E    SU Lock Test : Verify SI/CSI assignments after switch-over
	AMF-NOD-LCK.TC001   E    Node Lock Test : Verify SI/CSI assignments after switch-over
	AMF-RDN-PRC.TC001   E    AMF Reduction Procedure Test : Verify SI/CSI assignments in the reduced mode
	AMF-SU-SDN.TC001    E    SU Shutdown Test : Verify SI/CSI assignments after switch-over
	

you can then put some of the test cases in a file (say tests.lst) and specify the test-case-filter-file name while running tae using -T switch

# <TAEBASE>/tae/tae -T tests.lst -f fetch:stop
  • Note that the above command should be issued from <TAECFGDIR> directory for the model
  • It will execute all the steps from fetch to stop (fetch, populate, configure, build, package, deploy, start, test, stop).
  • Some of the steps can be skipped by specifying the stage-name in <skip_stages value="" /> tag present in "model.cfg" file. Alternatively, you can also specify something like "-f build:test" if the requirement is to build the model from already existing code base and then run the test cases.
  • If neither of "-t" or "-T" switch is specified, then TAE runs all the test cases present in the "<model>/src/test" directory. One can also use "-t *" to run all the test cases. Remember, the regular expression syntax is supported while specifying the filter for testcases, which can be very handy sometime.

TAE logs

TAE log files will be present in "log" directory of your model <TAECFGDIR>. Following log files are created by TAE for various different purposes:

  • tae.log : contains detailed log of all the commands executed by the TAE while performing the test.
  • bash_127.0.0.1_build_server.log : Contains the logs of commands executed during the SAFplus Platform and model build. It also contains the "make" output in case of build error.
  • bash_local.log : Contains the commands performed on the local machine like scp the tarball to the remote nodes etc.
  • Node level postmortem files (e.g. SysCtrlI0.postmortem.tgz) : Contains the node level logs as well as the runtime files of the node.