Skip to content

RFC: Support arbitrary tagging and selection of testcases. #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brarcher merged 2 commits into from
Jul 12, 2016
Merged

RFC: Support arbitrary tagging and selection of testcases. #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7ca2b6d
Reduce Timeout Double Scaling Tests scale to avoid UT failures
Jun 23, 2016
7b0086f
Support arbitrary tagging and selection of testcases.
Jun 29, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 101 additions & 9 deletions doc/check.texi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -974,6 +974,8 @@ easier for the developer to write, run, and analyze tests.
* Test Fixtures::
* Multiple Suites in one SRunner::
* Selective Running of Tests::
* Selecting Tests by Suite or Test Case::
* Selecting Tests Based on Arbitrary Tags::
Copy link
Contributor

@brarcher brarcher Jul 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll need to add:

* Selecting Tests by Suite or Test Case

here. Otherwise, the index is the following:

...
4.5 Multiple Suites in one SRunner      
4.6 Selective Running of Tests      
4.6.2 Selecting Tests Based on Arbitrary Tags       
4.7 Testing Signal Handling and Exit Values 
...

Copy link
Author

@cdentyou cdentyou Jul 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch

* Testing Signal Handling and Exit Values::
* Looping Tests::
* Test Timeouts::
Expand Down Expand Up @@ -1302,19 +1304,105 @@ srunner_add_suite (sr, make_pack_suite ());
@end example

@node Selective Running of Tests, Testing Signal Handling and Exit Values, Multiple Suites in one SRunner, Advanced Features

@section Selective Running of Tests

After adding a couple of suites and some test cases in each, it is
sometimes practical to be able to run only one suite, or one specific
test case, without recompiling the test code. Check provides two ways
to accomplish this, either by specifying a suite or test case by name
or by assigning tags to test cases and specifying one or more tags to
run.

@menu
* Selecting Tests by Suite or Test Case::
* Selecting Tests Based on Arbitrary Tags::
@end menu

@node Selecting Tests by Suite or Test Case, Selecting Tests Based on Arbitrary Tags, Selective Running of Tests, Selective Running of Tests
@subsection Selecting Tests by Suite or Test Case

@vindex CK_RUN_SUITE
@vindex CK_RUN_CASE
After adding a couple of suites and some test cases in each, it is
sometimes practical to be able to run only one suite, or one
specific test case, without recompiling the test code. There are
two environment variables available that offers this ability,
@code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the value to
the name of the suite and/or test case you want to run. These
environment variables can also be a good integration tool for
running specific tests from within another tool, e.g. an IDE.


There are two environment variables available that offer this
ability, @code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the
value to the name of the suite and/or test case you want to run. These
environment variables can also be a good integration tool for running
specific tests from within another tool, e.g. an IDE.

@node Selecting Tests Based on Arbitrary Tags, ,Selecting Tests by Suite or Test Case, Selective Running of Tests
@subsection Selecting Tests Based on Arbitrary Tags

@vindex CK_INCLUDE_TAGS
@vindex CK_EXCLUDE_TAGS

It can be useful to dynamically include or exclude groups of tests to
be run based on criteria other than the suite or test case name. For
example, one or more tags can be assigned to test cases. The tags
could indicate if a test runs for a long time, so such tests could be
excluded in order to run quicker tests for a sanity
check. Alternately, tags may be used to indicate which functional
areas test cover. Tests can then be run that include all test cases
for a given set of functional areas.

In Check, a tag is a string of characters without white space. One or
more tags can be assigned to a test case by using the
@code{tcase_set_tags} function. This function accepts a string, and
multiple tags can be specified by delimiting them with spaces. For
example:

@example
@verbatim
Suite *s;

TCase *red, *blue, *purple, *yellow, *black;

s = suite_create("Check Tag Filtering");

red = tcase_create("Red");
tcase_set_tags(red, "Red");
suite_add_tcase (s, red);
tcase_add_test(red, red_test1);

blue = tcase_create("Blue");
tcase_set_tags(blue, "Blue");
suite_add_tcase (s, blue);
tcase_add_test(blue, blue_test1);

purple = tcase_create("Purple");
tcase_set_tags(purple, "Red Blue");
suite_add_tcase (s, purple);
tcase_add_test(purple, purple_test1);

@end verbatim
@end example

Once test cases are tagged they may be selectively run in one of two ways:

a) Using Environment Variables

There are two environment variables available for selecting test cases
based on tags: @code{CK_INCLUDE_TAGS} and
@code{CK_EXCLUDE_TAGS}. These can be set to a space separated list of
tag names. If @code{CK_INCLUDE_TAGS} is set then test cases which
include at least one tag in common with @code{CK_INCLUDE_TAGS} will be
run. If @code{CK_EXCLUDE_TAGS} is set then test cases with one tag in
common with @code{CK_EXCLUDE_TAGS} will not be run. In cases where
both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} match a tag for
a test case the test will be excluded.

Both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} can be
Copy link
Contributor

@brarcher brarcher Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering about this. Could you add a test which verifies this behavior?

Copy link
Author

@cdentyou cdentyou Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean one that explictly uses the env variables ? There is already an include_exclude suite that tests the correct filtering - its just that it does not use the env vars - it would certainly be easy enough to clone it to produce an env var version as well if you think that's useful. Probably should just for completeness.

Copy link
Contributor

@brarcher brarcher Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the tests in check_check_tags.c do verify that CK_INCLUDE_TAGS and CK_EXCLUDE_TAGS work as expected. This is good. I think that there should be something to check that CK_INCLUDE_TAGS and CK_EXCLUDE_TAGS work with CK_RUN_SUITE and CK_RUN_CASE. I do not think there is something for this yet.

Copy link
Author

@cdentyou cdentyou Jul 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes - true - I will add something.

specified in conjunction with @code{CK_RUN_SUITE} or even
@code{CK_RUN_CASE} in which case they will have the effect of further
narrowing the selection.

b) Programmatically

The @code{srunner_run_tagged} function allows one to specify which
tags to run or exclude from a suite runner. This can be used to
programmatically control which test cases may run.

@node Testing Signal Handling and Exit Values, Looping Tests, Selective Running of Tests, Advanced Features
@section Testing Signal Handling and Exit Values

Expand Down Expand Up @@ -1975,6 +2063,10 @@ CK_RUN_CASE: Name of a test case, runs only that test. See section @ref{Selectiv

CK_RUN_SUITE: Name of a test suite, runs only that suite. See section @ref{Selective Running of Tests}.

CK_INCLUDE_TAGS: String of space separated tags, runs only test cases associated with at least one of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.

CK_EXCLUDE_TAGS: String of space separated tags, runs only test cases not associated with any of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.

CK_VERBOSITY: How much output to emit, accepts: ``silent'', ``minimal'', ``normal'', ``subunit'', or ``verbose''. See section @ref{SRunner Output}.

CK_FORK: Set to ``no'' to disable using fork() to run unit tests in their own process. This is useful for debugging segmentation faults. See section @ref{No Fork Mode}.
Expand Down
67 changes: 66 additions & 1 deletion src/check.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ static void suite_free(Suite * s)
free(s);
}


TCase *tcase_create(const char *name)
{
char *env;
Expand Down Expand Up @@ -149,10 +150,49 @@ TCase *tcase_create(const char *name)
tc->ch_sflst = check_list_create();
tc->unch_tflst = check_list_create();
tc->ch_tflst = check_list_create();
tc->tags = check_list_create();

return tc;
}

/*
* Helper function to create a list of tags from
* a space separated string.
*/
List *tag_string_to_list(const char *tags_string)
{
List *list;
char *tags;
char *tag;

list = check_list_create();

if (NULL == tags_string)
{
return list;
}

tags = strdup(tags_string);
tag = strtok(tags, " ");
while (tag)
{
check_list_add_end(list, strdup(tag));
tag = strtok(NULL, " ");
}
free(tags);
return list;
}

void tcase_set_tags(TCase * tc, const char *tags_orig)
{
/* replace any pre-existing list */
if (tc->tags)
{
check_list_apply(tc->tags, free);
check_list_free(tc->tags);
}
tc->tags = tag_string_to_list(tags_orig);
}

static void tcase_free(TCase * tc)
{
Expand All @@ -161,15 +201,40 @@ static void tcase_free(TCase * tc)
check_list_apply(tc->ch_sflst, free);
check_list_apply(tc->unch_tflst, free);
check_list_apply(tc->ch_tflst, free);
check_list_apply(tc->tags, free);
check_list_free(tc->tflst);
check_list_free(tc->unch_sflst);
check_list_free(tc->ch_sflst);
check_list_free(tc->unch_tflst);
check_list_free(tc->ch_tflst);

check_list_free(tc->tags);
free(tc);
}

unsigned int tcase_matching_tag(TCase *tc, List *check_for)
{

if (NULL == check_for)
{
return 0;
}

for(check_list_front(check_for); !check_list_at_end(check_for);
check_list_advance(check_for))
{
for(check_list_front(tc->tags); !check_list_at_end(tc->tags);
check_list_advance(tc->tags))
{
if (0 == strcmp((const char *)check_list_val(tc->tags),
(const char *)check_list_val(check_for)))
{
return 1;
Copy link
Contributor

@brarcher brarcher Jun 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does returning here leak tags, which contains strdup'ed data?

Copy link
Author

@cdentyou cdentyou Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch ! fixed.

}
}
}
return 0;
}

void suite_add_tcase(Suite * s, TCase * tc)
{
if(s == NULL || tc == NULL || check_list_contains(s->tclst, tc))
Expand Down
75 changes: 71 additions & 4 deletions src/check.h.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,19 @@ CK_DLL_EXP void CK_EXPORT suite_add_tcase(Suite * s, TCase * tc);
* */
CK_DLL_EXP TCase *CK_EXPORT tcase_create(const char *name);

/**
* Associate a test case with certain tags.
* Replaces any existing tags with the new set.
*
* @param tc the test case
*
* @param tags string containing arbitrary tags separated by spaces.
* This will be copied. Passing NULL clears all tags.
*
* @since 0.11.0
* */
CK_DLL_EXP void CK_EXPORT tcase_set_tags(TCase * tc,
const char *tags);
/**
* Add a test function to a test case
*
Expand Down Expand Up @@ -955,8 +968,9 @@ CK_DLL_EXP void CK_EXPORT srunner_free(SRunner * sr);
* In addition to running all suites, if the suite runner has been
* configured to output to a log, that is also performed.
*
* Note that if the CK_RUN_CASE and/or CK_RUN_SUITE environment variables
* are defined, then only the named suite and/or test case is run.
* Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS and/or
* CK_EXCLUDE_TAGS environment variables are defined, then only the
* named suites or test cases will run.
*
* @param sr suite runner to run all suites from
* @param print_mode the verbosity in which to report results to stdout
Expand All @@ -974,9 +988,22 @@ CK_DLL_EXP void CK_EXPORT srunner_run_all(SRunner * sr,
* suite runner has been configured to output to a log, that is also
* performed.
*
* Note that if the sname and tcname parameters are passed as null
* then the function will fallback to using the environment variables
* CK_RUN_SUITE and CK_RUN_CASE respectively in order to select the
* suite/cases.
*
* Similarly if the CK_INCLUDE_TAGS and/or CK_EXCLUDE_TAGS environment
* variables are defined then these will further filter the test cases
* (see srunner_run_tagged, below).
*
* @param sr suite runner where the given suite or test case must be
* @param sname suite name to run. A NULL means "any suite".
* @param tcname test case name to run. A NULL means "any test case"
* @param sname suite name to run. A NULL means use the value of the
* environment variable CK_RUN_SUITE if set, otherwise run "any/every
* suite".
* @param tcname test case name to run. A NULL means use the value of
* the environment variable CK_RUN_CASE if set, otherwise run
* "any/every case".
* @param print_mode the verbosity in which to report results to stdout
*
* @since 0.9.9
Expand All @@ -986,6 +1013,46 @@ CK_DLL_EXP void CK_EXPORT srunner_run(SRunner * sr, const char *sname,
enum print_output print_mode);


/**
* Run a specific suite or test case or testcases with specific tags
Copy link
Contributor

@brarcher brarcher Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to update the comments on the srunner_run_all and srunner_run functions above to include the following:

 * Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS,
  * and/or CK_EXCLUDE_TAGS environment variables are defined, then only
  * the selected suite or test cases will run.

namely:

 /**
  * Runs a suite runner and all contained suite, printing results to
  * stdout as specified by the print_mode.
  *
  * In addition to running all suites, if the suite runner has been
  * configured to output to a log, that is also performed.
  *
  * Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS,
  * and/or CK_EXCLUDE_TAGS environment variables are defined, then only
  * the selected suite or test cases will run.
  *
  * @param sr suite runner to run all suites from
  * @param print_mode the verbosity in which to report results to stdout
  *
  * @since 0.6.0
  */
 CK_DLL_EXP void CK_EXPORT srunner_run_all(SRunner * sr,
                                           enum print_output print_mode);

 /**
  * Run a specific suite or test case from a suite runner, printing results
  * to stdout as specified by the print_mode.
  *
  * In addition to running any applicable suites or test cases, if the
  * suite runner has been configured to output to a log, that is also
  * performed.
  *
  * Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS,
  * and/or CK_EXCLUDE_TAGS environment variables are defined, then only
  * the selected suite or test cases will run.
  *
  * @param sr suite runner where the given suite or test case must be
  * @param sname suite name to run. A NULL means "any suite".
  * @param tcname test case name to run. A NULL means "any test case"
  * @param print_mode the verbosity in which to report results to stdout
  *
  * @since 0.9.9
  */
 CK_DLL_EXP void CK_EXPORT srunner_run(SRunner * sr, const char *sname,
                                       const char *tcname,
                                       enum print_output print_mode);

Copy link
Contributor

@brarcher brarcher Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the srunner_run_tagged function comment should be updated with this statement.

Copy link
Author

@cdentyou cdentyou Jul 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch.

Copy link
Author

@cdentyou cdentyou Jul 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Slightly reworked your suggested text to make the interaction between the parameters and the env variables explicit.

* from a suite runner, printing results to stdout as specified by the
* print_mode.
*
* In addition to running any applicable suites or test cases, if the
* suite runner has been configured to output to a log, that is also
* performed.
*
* Note that if sname, tcname, include_tags, exclude_tags parameters
* are passed as NULL then if the environment variables CK_RUN_SUITE,
* CK_RUN_CASE, CK_INCLUDE_TAGS, CK_EXCLUDE_TAGS are defined then these
* values will be used instead.
*
* @param sr suite runner where the given suite or test case must be
* @param sname suite name to run. A NULL means use the value of the
* environment variable CK_RUN_SUITE if set, otherwise run "any/every
* suite".
* @param tcname test case name to run. A NULL means use the value of
* the environment variable CK_RUN_CASE if set, otherwise run
* "any/every case".
* @param include_tags space separate list of tags. Only run test
* cases that share one of these tags. A NULL means use the value of
* the environment variable CK_INCLUDE_TAGS if set, otherwise run
* "any/every test case".
* @param exclude_tags space separate list of tags. Only run test
* cases that do not share one of these tags even if they are selected
* by an included tag. A NULL means use the value of the environment
* variable CK_EXCLUDE_TAGS if set, otherwise run "any/every test
* case".
* @param print_mode the verbosity in which to report results to stdout
*
* @since 0.11.0
*/
CK_DLL_EXP void CK_EXPORT srunner_run_tagged(SRunner * sr, const char *sname,
const char *tcname,
const char *include_tags,
const char *exclude_tags,
enum print_output print_mode);

/**
* Retrieve the number of failed tests executed by a suite runner.
*
Expand Down
4 changes: 4 additions & 0 deletions src/check_impl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ struct TCase
List *unch_tflst;
List *ch_sflst;
List *ch_tflst;
List *tags;
};

typedef struct TestStats
Expand Down Expand Up @@ -134,4 +135,7 @@ enum fork_status cur_fork_status(void);

clockid_t check_get_clockid(void);

unsigned int tcase_matching_tag(TCase *tc, List *check_for);
List *tag_string_to_list(const char *tags_string);

#endif /* CHECK_IMPL_H */
Loading