Using Code Coverage
In this chapter:
- Code coverage in the IDE
- Enabling code coverage
- Importing gcc code coverage data from a project
- Associated views
Code coverage is a way to measure how much code a particular process has executed during a test or benchmark. Using code-coverage analysis, you can then create additional test cases to increase coverage and determine a quantitative measure of code coverage, which is an indirect measure of the quality of your software (or better, a direct measure of the quality of your tests).
Several types of metrics are commonly used in commercial code-coverage tools, ranging from simple line or block coverage (i.e. “this statement was executed”) to condition-decision coverage (i.e. “all terms in this Boolean expression are exercised”). A given tool usually provides a combination of types.
The coverage tool in the IDE is a visual front end to the gcov metrics produced by the gcc compiler. These coverage metrics are essentially basic block coverage and branch coverage.
The IDE presents these metrics as line coverage, showing which lines are fully covered, partially covered, and not covered at all. The IDE also presents percentages of coverage in terms of the actual code covered (i.e. not just lines).
Block coverage, sometimes known as line coverage, describes whether a block of code, defined as not having any branch point within (i.e. the path of execution enters from the beginning and exits at the end) is executed or not.
By tracking the number of times the block of code has been executed, the IDE can determine the total coverage of a particular file or function. The tool also uses this information to show line coverage by analyzing the blocks on each line and determining the level of coverage of each.
Branch coverage can track the path of execution taken between blocks of code. Although this metric is produced by the gcc compiler, currently the IDE doesn't provide this information.
The IDE's code coverage tool works in conjunction with the compiler (gcc), the QNX C library (libc), and optionally the remote target agent (qconn). When code coverage is enabled for an application, the compiler instruments the code so that at run time, each branch execution to a basic block is counted. During the build, the IDE produces data files in order to recreate the program's flow graph and to provide line locations of each block.
|Since the IDE creates secondary data files at compilation
time, you must be careful when building your programs in a
multitargeted build environment, such as QNX Neutrino.
You must either:
Note also that the compiler's optimizations could produce unexpected results, so you should perform coverage tests on an unoptimized, debug-enabled build.
When you build a program with the Build with Code Coverage build option enabled and then launch it using a C/C++ QNX Qconn (IP) launch configuration, the instrumented code linked into the process connects to qconn, allowing the coverage data to be read from the process's data space.
But if you launch a coverage-built process with coverage disabled in the launch configuration, this causes the process to write the coverage information to a data file (.gcda) at run time, rather than read it from the process's data space.
|You can later import the data into the IDE code coverage tool. For information about importing gcc coverage data from a project, see “Importing gcc code coverage data from a project” later in this chapter.|
Once a coverage session has begun, you can immediately view the data. The QNX Code Coverage perspective contains a Code Coverage Sessions view that lists previous as well as currently active sessions. You can explore each session and browse the corresponding source files that have received coverage data.
|Code Coverage might not work as expected because the code coverage data for C++ projects includes other functions that are also in the source file, such as static initializer and global constructor functions. In addition, the files included by include statements aren't included in the overall coverage total; only those functions that are in the original source are included for code coverage.|
To build executables with code coverage enabled:
- In the C/C++ Projects view, right-click your project and select Properties. The properties dialog for your project appears.
- In the left pane, select QNX C/C++ Project.
- In the Build Options pane, check Build with Code Coverage.
- In the Build Variants tab, check only one build variant.
If more than one build variant is checked, the IDE displays an error message and disables the OK button.
- Click OK.
- In the C/C++ Projects view, right-click your project and select Clean….
- Be sure that your project is selected, and check the Start a build immediately box, and then click OK to rebuild your project.
If you're using your own custom build environment, rather than QNX Makefiles, you'll have to manually pass the coverage option to the compiler.
To enable code coverage for non-QNX projects:
If you're using qcc/gcc, compile and link with the following options:
- Also, include the -p option for linking.
For example, your Makefile might look something like the Makefile below, which belongs to the Code Coverage example project included with the IDE (although, this example includes additional comments):
DEBUG = -g CC = qcc LD = qcc CFLAGS += -Vgcc_ntox86 $(DEBUG) -c -Wc,-Wall -I. -O0 -Wc,-ftest-coverage -Wc,-fprofile-arcs LDFLAGS+= -Vgcc_ntox86 $(DEBUG) -ftest-coverage -fprofile-arcs -p # CC refers to the program for compiling C programs (the default is qcc. Use # CXX as the program for compiling C++ programs. # CFLAGS are additional flags to give to the C compiler. Use CFLAGS for the C++ compiler. # -c compiles or assemble the source files, but doesn't link, and the -Wc captures # the warning messages. The linking stage isn't done. The ultimate output is in the form of an object file for each source file. # -Wall turns on all optional warnings that are desirable for normal code. # -I. adds the current directory to the list of directories to search for # header files. Directories named by -I are searched before the standard system # include directories. # -O0 is an optimization flag that indicates 'Do not optimize.' # LDFLAGS are additional flags to give to compilers when they invoke the ld linker. # -ftest-coverage -Wc means that Code Coverage is enabled for your project, # and the data is used for test coverage analysis. # -fprofile-arcs adds code so that program flow arcs are instrumented. During # execution, the program records how many times each branch and call is # executed and how many times it is taken or returns, and it saves this # data to a file with the extension .gcda for each source file. # For Code Coverage, you'll need the -fprofile-arcs -ftest-coverage options in # both the compile and link lines, as well as the -p option in the link line. # # -p is used to generate extra code to write profile information for the analysis # program. You must use the -p option when compiling source files that you want # data about, and you must also use it when linking. dir := $(shell pwd) BINS = rbt_client rbt_server # This next line is the rule for <cmd>all</cmd> that incrementally builds your # system by performing a <cmd>make</cmd> of all the top-level targets the # Makefile knows about. It does this by expressing a dependency on the results # of that system, which in turn have their own rules and dependencies. all: $(BINS) # The following line shows a simple rule for cleaning your build environment. # It cleans your build environment by deleting all files that are normally # created by running make. # It has a Target named <cmd>clean</cmd> to left of the colon, no # dependencies (to the right of the colon), and two commands that are indented # by tabs on the lines that follow. clean: rm -f *.o *.img *.bb *.bbg *.gcno *.gcda $(BINS) # The following lines are Dependency Rules, which are rules without any command. # If any file to the right of the colon changes, the target to the left of the # colon is no longer considered current (out of date). # Dependency Rules are often used to capture header file dependencies. rbt_server: rbt_server.o # Alternatively, to manually capture dependencies, several automated # dependency generators exist. rbt_server.o : rbt_server.c rbt_server.h $(CC) $(CFLAGS) $(dir)/$< rbt_client: rbt_client.o rbt_client.o: rbt_client.c rbt_server.h $(CC) $(CFLAGS) $(dir)/$< profileCPP-std: $(objects) $(CC) -Vgcc_ntox86 $^ -g -p -o $@ -lcpp
To enable Code Coverage for your project, you must use the options -fprofile-arcs -ftest-coverage when compiling and linking.
For example, in the Makefile, you'll have the following gcc options set for Code Coverage:
CFLAGS += -g -fprofile-arcs -ftest-coverage LDFLAGS+=-p -g -fprofile-arcs -ftest-coverage
To start a program and measure the code coverage:
- Create a C/C++ QNX QConn (IP) launch configuration as you normally would, but don't click OK yet.
- On the launcher, click the Tools tab.
- Click Add/Delete Tool. The Tools selection dialog appears.
- Select the Code Coverage tool.
- Click OK.
- Click the Code Coverage tab, and fill in these fields:
- Code Coverage data format
- Select gcc 4.3 or later to enable code coverage metrics collection if your application was compiled with gcc 4.3 or later.
- Comments for this Code Coverage session
- Your notes about the session, for your own personal use. The comments appear at the top of the generated reports.
- Code Coverage data scan interval (sec)
- Sets how often the Code Coverage tool polls for data. A low setting can cause continuous network traffic. The default setting of 5 seconds should be sufficient.
- Collect data for
- By default, all code coverage data built with code coverage in a project is included in the current Code Coverage session. To include referenced projects or to only collect data from certain sources, disable the option All Sources in this application compiled with code coverage, and then click Select to select the projects or files that you want to collect code coverage data for.
- Opens the Projects to include Code Coverage data from dialog so you can choose projects to include your coverage data (projects and files).
Select any project from this list that you wish to gather
code coverage data for. Note that projects must be built with code coverage
enabled to capture data.
- Check Switch to this tool's perspective on launch if you want to automatically go to the QNX Code Coverage perspective when you run or debug.
- Click Apply.
- Click Run or Debug.
Previously, and for older compilers, if you launched a code coverage-enabled build process and chose to disable code coverage in the launch configuration, the process wrote the coverage information to a data file (.gcda) at run time, rather than read it from the process's data space. This meant that you could choose to import this data into the IDE Code Coverage tool at a later time. The newer gcc compiler doesn't stream the data coverage; the IDE waits for the generation of the data file before it copies it back to host machine.
In addition, the IDE generates notes files (.gcno) when it compiles projects that have enabled code coverage.
There are multiple ways to import a file.
It isn't necessary to move the file you want to import into the Workspace location.
By default, the location of the .da files for the earlier versions of gcc 2.95.3 and 4.2 compilers that the IDE supported were located in /tmp (or another location, if specified). However, now the .gcda files are located in a folder structure created under the /tmp location. For the gcc 4.3 compiler, the .da files are also located in the same location under /tmp.
If you buildt a project using an earlier version of gcc (version 2.95.3), the .bb and .bbg files were created in the same location where the source file was located.
When copying a project_name.da file into your workspace, you must copy it to the top level of the directory structure. If you build the project using gcc version 4.3, the .bb and .bbg files are created under the variant_name/o_g directory. When copying the project_name.da file into your workspace, you must copy it to the same location where the .bb and .bbg files are located. In this case, it is the variant_name/o_g directory.
To import gcc code coverage data from a project:
- Create and build a project with code coverage selected. For information about enabling code coverage, see “Enabling code coverage” earlier in this chapter.
- Create a launch configuration where code coverage is disabled.
- Run this configuration.
- Optional: Observe the target's directory using the Target File System Navigator tab in the Tasks view (bottom of the Workbench window) in the
location where the file project_name.da resides.
By default, you will not have the Target File System Navigator tab in your Tasks view. To add this tab to your view:
- Select .
- Expand QNX Targets.
- Select Target File System Navigator.
- Click OK.
If a previous project was built using gcc version 2.95, the .bb and .bbg files were created in the same location as the source file. Now, if a project is built using gcc version 4.3, .bb and .bbg files are created under variant_name/o_g directory.
- Optional: For the target, right-click on the file project_name.da and select .
- Optional: In the Select Target Folder window, specify a folder location to copy the file, and click OK.
The project_name.da will be visible under the C/C++ tab for the corresponding project.
- On the QNX Code Coverage tab, select
Now, you'll browse on the remote target to the folder that contains the data file.
- Specify the name of the session, click Browse to locate a project, and then click OK.
- Specify the name of the session, project and platform used to generate the code coverage data. Click Next.
- For the Coverage protocol type, browse to the location where the data files are located (such as on the remote target, within the workspace, or on the filesystem).
- Optional: If there are referenced projects to include data for, select the referenced projects to import code coverage data from. Also specify a comment about the import session, if desired.
- Click Finish.
Now, the Code Coverage tab shows the session name and imported gcc code coverage data for the selected project.
The QNX Code Coverage perspective includes the following views:
- Code Coverage Sessions view (controlling your session and examining data line-by-line)
- Code Coverage Properties View (seeing your coverage at a glance)
- Code Coverage Report View (examining your coverage report)
The Code Coverage Sessions view lets you control and display multiple code-coverage sessions:
The view shows the following as a hierarchical tree for each session:
|Session item||Description||Possible icons|
|Code coverage session||Launch configuration name, coverage tool, and start time (e.g. ccov102_factor [GCC Code Coverage] (7/2/03 2:48 PM))|
|Project||Project name and amount of coverage (e.g. ccov102_factor [ 86.67% ])|
|File||Filename and amount of coverage (e.g. ccov102_factor.c [ 86.67% ])|
|Function||Function name and amount of coverage (e.g. main [ 100% ])|
The IDE uses several icons in this view:
|Green||Full (100%) coverage|
|(cell is highlighted)||Out-of-date source file|
|x||Red||An error marker to indicate some type of error (e.g. a code coverage data file was not found, or an error reading data or notes files).|
The IDE also adds a coverage markup icon () to indicate source markup in the editor. (See the “Examining data line-by-line” section, below.)
To reduce the size of the hierarchical tree, you can click the Collapse All () button.
To combine several sessions:
- In the Code Coverage Sessions view, select the sessions you want to combine.
- Right-click your selections and select Combine/Copy Sessions. The IDE prompts you for a session name and creates a combined session.
The IDE can show the line-by-line coverage information for your source code. In the left margin, the editor shows you a “covered” icon () beside each line of source. In the right margin, the editor shows a summary of the coverage by showing green sections for fully-covered code, yellow for partial coverage, and red for no coverage.
- To open a file in the QNX Code Coverage perspective:
- In the Code Coverage Sessions view, expand a session and double-click a file or function.
Code coverage markers are added to the left pane of the opened file.
- To show coverage information from a particular session:
- In the Code Coverage Sessions view, select a session. The IDE shows all of the various markers.
To automatically show coverage information when opening a file:
- Open the Preferences dialog ( ).
- In the left pane, select .
- In the right pane, check the desired markers in the Coverage markup when file is opened field.
- Click OK. The next time you open a file, the markers appear automatically. To add markers from another session, add them manually, as described above.
- To remove all coverage markers:
- In the Code Coverage Sessions view's title bar, click the Remove All Coverage Markers button ().
The Properties view shows a summary of the code coverage for a project, file, or function you've selected in the Code Coverage Sessions view.
The Properties view tells you how many lines were covered, not covered, and so on:
The Code Coverage Report view provides a summary (in XML) of your session. The view lets you “drill down” into your project and see the coverage for individual files and functions:
To generate a report, simply right-click a coverage session and select Generate Report.
By default, the IDE shows reports in the Code Coverage Report view, but you can also have the IDE show reports in an external browser. Using an external browser lets you compare several reports simultaneously.
To toggle between viewing reports in the Code Coverage Report view and in an external browser:
- Open the Preferences dialog ( ).
- In the left pane, select .
- In the right pane, enable or disable the Use external Web browser check box.
- Click OK.
To save a report:
- Right-click in the Code Coverage Report view to show the context menu.
- Click Save As... to save the report.
- To refresh a report:
- In the Code Coverage Report view's title bar, click the Refresh button ().
- To print a report:
- In the Code Coverage Report view's title bar, click the Print button ().
By default, the report generated by the IDE doesn't include the code coverage information from other included files; however, you can choose to view this information, if desired.
- Select .
- In the left pane, expand QNX and select Code Coverage.
- In the right pane, select Show code coverage information from included files.
- Click OK.