Support arbitrary tagging and selection of testcases.

A testcase can optionally have a list of tags associated with it.
Srunner can be run with an optional include list of tags and an optional
exclude list of tags. These will have the effect of filtering testcases
that would otherwise be run.

Author: Crispin Dent-Young

doc/check.texi

@@ -974,6 +974,8 @@ easier for the developer to write, run, and analyze tests.
 * Test Fixtures::               
 * Multiple Suites in one SRunner::  
 * Selective Running of Tests::
+* Selecting Tests by Suite or Test Case::
+* Selecting Tests Based on Arbitrary Tags::
 * Testing Signal Handling and Exit Values::  
 * Looping Tests::               
 * Test Timeouts::               
@@ -1302,19 +1304,105 @@ srunner_add_suite (sr, make_pack_suite ());
 @end example
 
 @node Selective Running of Tests, Testing Signal Handling and Exit Values, Multiple Suites in one SRunner, Advanced Features
+
 @section Selective Running of Tests
 
+After adding a couple of suites and some test cases in each, it is
+sometimes practical to be able to run only one suite, or one specific
+test case, without recompiling the test code.  Check provides two ways
+to accomplish this, either by specifying a suite or test case by name
+or by assigning tags to test cases and specifying one or more tags to
+run.
+
+@menu
+* Selecting Tests by Suite or Test Case::
+* Selecting Tests Based on Arbitrary Tags::
+@end menu
+
+@node Selecting Tests by Suite or Test Case, Selecting Tests Based on Arbitrary Tags, Selective Running of Tests, Selective Running of Tests
+@subsection Selecting Tests by Suite or Test Case
+
 @vindex CK_RUN_SUITE
 @vindex CK_RUN_CASE
-After adding a couple of suites and some test cases in each, it is
-sometimes practical to be able to run only one suite, or one
-specific test case, without recompiling the test code. There are
-two environment variables available that offers this ability,
-@code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the value to
-the name of the suite and/or test case you want to run. These
-environment variables can also be a good integration tool for
-running specific tests from within another tool, e.g. an IDE.
-  
+
+There are two environment variables available that offer this
+ability, @code{CK_RUN_SUITE} and @code{CK_RUN_CASE}. Just set the
+value to the name of the suite and/or test case you want to run. These
+environment variables can also be a good integration tool for running
+specific tests from within another tool, e.g. an IDE.
+
+@node Selecting Tests Based on Arbitrary Tags, ,Selecting Tests by Suite or Test Case, Selective Running of Tests
+@subsection Selecting Tests Based on Arbitrary Tags
+
+@vindex CK_INCLUDE_TAGS
+@vindex CK_EXCLUDE_TAGS
+
+It can be useful to dynamically include or exclude groups of tests to
+be run based on criteria other than the suite or test case name. For
+example, one or more tags can be assigned to test cases. The tags
+could indicate if a test runs for a long time, so such tests could be
+excluded in order to run quicker tests for a sanity
+check. Alternately, tags may be used to indicate which functional
+areas test cover. Tests can then be run that include all test cases
+for a given set of functional areas.
+
+In Check, a tag is a string of characters without white space. One or
+more tags can be assigned to a test case by using the
+@code{tcase_set_tags} function. This function accepts a string, and
+multiple tags can be specified by delimiting them with spaces. For
+example:
+
+@example
+@verbatim
+    Suite *s;
+
+    TCase *red, *blue, *purple, *yellow, *black;
+
+    s = suite_create("Check Tag Filtering");
+
+    red = tcase_create("Red");
+    tcase_set_tags(red, "Red");
+    suite_add_tcase (s, red);
+    tcase_add_test(red, red_test1);
+
+    blue = tcase_create("Blue");
+    tcase_set_tags(blue, "Blue");
+    suite_add_tcase (s, blue);
+    tcase_add_test(blue, blue_test1);
+
+    purple = tcase_create("Purple");
+    tcase_set_tags(purple, "Red Blue");
+    suite_add_tcase (s, purple);
+    tcase_add_test(purple, purple_test1);
+
+@end verbatim
+@end example
+
+Once test cases are tagged they may be selectively run in one of two ways:
+
+a) Using Environment Variables
+
+There are two environment variables available for selecting test cases
+based on tags: @code{CK_INCLUDE_TAGS} and
+@code{CK_EXCLUDE_TAGS}. These can be set to a space separated list of
+tag names. If @code{CK_INCLUDE_TAGS} is set then test cases which
+include at least one tag in common with @code{CK_INCLUDE_TAGS} will be
+run. If @code{CK_EXCLUDE_TAGS} is set then test cases with one tag in
+common with @code{CK_EXCLUDE_TAGS} will not be run. In cases where
+both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} match a tag for
+a test case the test will be excluded.
+
+Both @code{CK_INCLUDE_TAGS} and @code{CK_EXCLUDE_TAGS} can be
+specified in conjunction with @code{CK_RUN_SUITE} or even
+@code{CK_RUN_CASE} in which case they will have the effect of further
+narrowing the selection.
+
+b) Programmatically
+
+The @code{srunner_run_tagged} function allows one to specify which
+tags to run or exclude from a suite runner. This can be used to
+programmatically control which test cases may run.
+
 @node Testing Signal Handling and Exit Values, Looping Tests, Selective Running of Tests, Advanced Features
 @section Testing Signal Handling and Exit Values
 
@@ -1975,6 +2063,10 @@ CK_RUN_CASE: Name of a test case, runs only that test. See section @ref{Selectiv
 
 CK_RUN_SUITE: Name of a test suite, runs only that suite. See section @ref{Selective Running of Tests}.
 
+CK_INCLUDE_TAGS: String of space separated tags, runs only test cases associated with at least one of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.
+
+CK_EXCLUDE_TAGS: String of space separated tags, runs only test cases not associated with any of the tags, See section @ref{Selecting Tests Based on Arbitrary Tags}.
+ 
 CK_VERBOSITY: How much output to emit, accepts: ``silent'', ``minimal'', ``normal'', ``subunit'', or ``verbose''.  See section @ref{SRunner Output}.
 
 CK_FORK: Set to ``no'' to disable using fork() to run unit tests in their own process. This is useful for debugging segmentation faults.  See section @ref{No Fork Mode}.

src/check.c

@@ -103,6 +103,7 @@ static void suite_free(Suite * s)
     free(s);
 }
 
+
 TCase *tcase_create(const char *name)
 {
     char *env;
@@ -149,10 +150,49 @@ TCase *tcase_create(const char *name)
     tc->ch_sflst = check_list_create();
     tc->unch_tflst = check_list_create();
     tc->ch_tflst = check_list_create();
+    tc->tags = check_list_create();
 
     return tc;
 }
 
+/*
+ * Helper function to create a list of tags from
+ * a space separated string.
+ */
+List *tag_string_to_list(const char *tags_string)
+{
+    List *list;
+    char *tags;
+    char *tag;
+
+    list = check_list_create();
+
+    if (NULL == tags_string)
+    {
+	return list;
+    }
+
+    tags = strdup(tags_string);
+    tag = strtok(tags, " ");
+    while (tag)
+    {
+	check_list_add_end(list, strdup(tag));
+	tag = strtok(NULL, " ");
+    }
+    free(tags);
+    return list;
+}
+
+void tcase_set_tags(TCase * tc, const char *tags_orig)
+{
+    /* replace any pre-existing list */
+    if (tc->tags)
+    {
+	check_list_apply(tc->tags, free);
+	check_list_free(tc->tags);
+    }
+    tc->tags = tag_string_to_list(tags_orig);
+}
 
 static void tcase_free(TCase * tc)
 {
@@ -161,15 +201,40 @@ static void tcase_free(TCase * tc)
     check_list_apply(tc->ch_sflst, free);
     check_list_apply(tc->unch_tflst, free);
     check_list_apply(tc->ch_tflst, free);
+    check_list_apply(tc->tags, free);
     check_list_free(tc->tflst);
     check_list_free(tc->unch_sflst);
     check_list_free(tc->ch_sflst);
     check_list_free(tc->unch_tflst);
     check_list_free(tc->ch_tflst);
-
+    check_list_free(tc->tags);
     free(tc);
 }
 
+unsigned int tcase_matching_tag(TCase *tc, List *check_for)
+{
+
+    if (NULL == check_for)
+    {
+	return 0;
+    }
+
+    for(check_list_front(check_for); !check_list_at_end(check_for);
+        check_list_advance(check_for))
+    {
+	for(check_list_front(tc->tags); !check_list_at_end(tc->tags);
+	    check_list_advance(tc->tags))
+	{
+	    if (0 == strcmp((const char *)check_list_val(tc->tags),
+			    (const char *)check_list_val(check_for)))
+	    {
+		return 1;
+	    }
+	}
+    }
+    return 0;
+}
+
 void suite_add_tcase(Suite * s, TCase * tc)
 {
     if(s == NULL || tc == NULL || check_list_contains(s->tclst, tc))

src/check.h.in

@@ -172,6 +172,19 @@ CK_DLL_EXP void CK_EXPORT suite_add_tcase(Suite * s, TCase * tc);
  * */
 CK_DLL_EXP TCase *CK_EXPORT tcase_create(const char *name);
 
+/**
+ * Associate a test case with certain tags.
+ * Replaces any existing tags with the new set.
+ *
+ * @param tc the test case
+ *
+ * @param tags string containing arbitrary tags separated by spaces.
+ *        This will be copied. Passing NULL clears all tags.
+ *
+ * @since 0.11.0
+ * */
+CK_DLL_EXP void CK_EXPORT tcase_set_tags(TCase * tc,
+					 const char *tags);
 /**
  * Add a test function to a test case
  *
@@ -955,8 +968,9 @@ CK_DLL_EXP void CK_EXPORT srunner_free(SRunner * sr);
  * In addition to running all suites, if the suite runner has been
  * configured to output to a log, that is also performed.
  *
- * Note that if the CK_RUN_CASE and/or CK_RUN_SUITE environment variables
- * are defined, then only the named suite and/or test case is run.
+ * Note that if the CK_RUN_CASE, CK_RUN_SUITE, CK_INCLUDE_TAGS and/or
+ * CK_EXCLUDE_TAGS environment variables are defined, then only the
+ * named suites or test cases will run.
  *
  * @param sr suite runner to run all suites from
  * @param print_mode the verbosity in which to report results to stdout
@@ -974,9 +988,22 @@ CK_DLL_EXP void CK_EXPORT srunner_run_all(SRunner * sr,
  * suite runner has been configured to output to a log, that is also
  * performed.
  *
+ * Note that if the sname and tcname parameters are passed as null
+ * then the function will fallback to using the environment variables
+ * CK_RUN_SUITE and CK_RUN_CASE respectively in order to select the
+ * suite/cases.
+ * 
+ * Similarly if the CK_INCLUDE_TAGS and/or CK_EXCLUDE_TAGS environment
+ * variables are defined then these will further filter the test cases
+ * (see srunner_run_tagged, below).
+ *
  * @param sr suite runner where the given suite or test case must be
- * @param sname suite name to run. A NULL means "any suite".
- * @param tcname test case name to run. A NULL means "any test case"
+ * @param sname suite name to run. A NULL means use the value of the
+ * environment variable CK_RUN_SUITE if set, otherwise run "any/every
+ * suite".
+ * @param tcname test case name to run. A NULL means use the value of
+ * the environment variable CK_RUN_CASE if set, otherwise run
+ * "any/every case".
  * @param print_mode the verbosity in which to report results to stdout
  *
  * @since 0.9.9
@@ -986,6 +1013,46 @@ CK_DLL_EXP void CK_EXPORT srunner_run(SRunner * sr, const char *sname,
                                       enum print_output print_mode);
 
 
+/**
+ * Run a specific suite or test case or testcases with specific tags
+ * from a suite runner, printing results to stdout as specified by the
+ * print_mode.
+ *
+ * In addition to running any applicable suites or test cases, if the
+ * suite runner has been configured to output to a log, that is also
+ * performed.
+ *
+ * Note that if sname, tcname, include_tags, exclude_tags parameters
+ * are passed as NULL then if the environment variables CK_RUN_SUITE,
+ * CK_RUN_CASE, CK_INCLUDE_TAGS, CK_EXCLUDE_TAGS are defined then these
+ * values will be used instead.
+ *
+ * @param sr suite runner where the given suite or test case must be
+ * @param sname suite name to run. A NULL means use the value of the
+ * environment variable CK_RUN_SUITE if set, otherwise run "any/every
+ * suite".
+ * @param tcname test case name to run. A NULL means use the value of
+ * the environment variable CK_RUN_CASE if set, otherwise run
+ * "any/every case".
+ * @param include_tags space separate list of tags. Only run test
+ * cases that share one of these tags. A NULL means use the value of
+ * the environment variable CK_INCLUDE_TAGS if set, otherwise run
+ * "any/every test case".
+ * @param exclude_tags space separate list of tags. Only run test
+ * cases that do not share one of these tags even if they are selected
+ * by an included tag. A NULL means use the value of the environment
+ * variable CK_EXCLUDE_TAGS if set, otherwise run "any/every test
+ * case".
+ * @param print_mode the verbosity in which to report results to stdout
+ *
+ * @since 0.11.0
+ */
+CK_DLL_EXP void CK_EXPORT srunner_run_tagged(SRunner * sr, const char *sname,
+					     const char *tcname,
+					     const char *include_tags,
+					     const char *exclude_tags,
+					     enum print_output print_mode);
+
 /**
  * Retrieve the number of failed tests executed by a suite runner.
  *

src/check_impl.h

@@ -65,6 +65,7 @@ struct TCase
     List *unch_tflst;
     List *ch_sflst;
     List *ch_tflst;
+    List *tags;
 };
 
 typedef struct TestStats
@@ -134,4 +135,7 @@ enum fork_status cur_fork_status(void);
 
 clockid_t check_get_clockid(void);
 
+unsigned int tcase_matching_tag(TCase *tc, List *check_for);
+List *tag_string_to_list(const char *tags_string);
+
 #endif /* CHECK_IMPL_H */