Contents |
Organization
The TAE is intended to be used both as a individual developer's test bench and as an enterprise-wide test automation solution. It therefore is organised in two independent parts: first, the web interface and historical database that is generally deployed on a single machine, and second, the testing engine which often deployed on every developer's machine and on an automated (nightly) build and test server.
This allows the test automation environment to be run in a variety of different situations and by a variety of different users. For example, it could be run by an automated nightly verification system or by a developer fixing a bug. Also tests can be run at different physical sites (or even by multiple companies cooperating on the same project) yet all the results can be uploaded to a single centralized database and web interface for project analysis.
Lastly, the TAE is designed to allow programs written for a variety of purposes -- test, demo, eval, and real systems -- to be run. By supplying a heavy "class" framework, most automated test frameworks' client libraries force the application into a specific unit test mold. So the only software that can be run are test suites designed from the bottom-up to fit within the automated test framework. This framework allows software that was not written for the express purpose of testing to be annotated with compile-time optional tests (similar to assert()) that can be used to verify the correct execution of that software. This means that your final application code can be self-testing and report errors to the TAE, just like your unit tests.
Each layer is summarized in this document and contains a link to a detailed design.
Layer 1: Unit testing API (Clovis Test API)
The smallest layer is the API used in a program to run tests. This layer consists of a set of C macros that group and run tests. When not testing -- when a "release" build is created for example -- these macros are compiled out. This is very similar to unit testing packages found for many languages.
If this layer is run alone, it will simply print test successes and failures to the screen.
When run within the layer 3 framework, test successes and failures will be posted to the TAE report server.
Layer 2: System Test API
This API layer allows tests to be written in Python programming language and runs them on a machine that sits outside of the device-under-test.
Through these APIs, your tests can access the entire cluster and start/stop applications (or unit tests), cause failures of processes or nodes, and access CLIs and other external interfaces.
If this layer is run alone, it will simply print test successes and failures to the screen.
When run within the layer 3 framework, test successes and failures will be posted to the TAE report server.
Layer 3: Test Execution Management
This layer consists of a separate machine (test driver) that controls the test "fixture" (the set of machines that constitute a test environment). It is capable of taking software from either a local directory, a ftp site, an http site or from subversion repository, unpacking it (if needed), installing it on the fixture, compiling it, and running a series of tests.
These tests constitute executable programs that use the Layer 1 APIs, and Layer 2 Python modules that can "drive" the progress of the test suite and inject errors and events such as process, network, and machine failures.
If this layer is run alone, it will run through a complete test suite and report successes and failures.
Layer 4: Automated Run and Longitudinal Reports
The Automated Run and Longitudinal Reporting layer consists of a single globally accessible server and a source repository (subversion etc). The server contains a web site that allows users to see which source code branches are failing which tests on what hardware and it keeps a history of this information.