Label sets allow for more expressive label filtering

Author: onsi

docs/index.md

@@ -2590,8 +2590,8 @@ The real power, of labels, however, is around filtering.  You can filter by labe
 - The `!` unary operator representing the NOT operation.
 - The `,` binary operator equivalent to `||`.
 - The `()` for grouping expressions.
-- All other characters will match as label literals.  Label matches are **case insensitive** and trailing and leading whitespace is trimmed.
 - Regular expressions can be provided using `/REGEXP/` notation.
+- All other characters will match as label literals.  Label matches are **case insensitive** and trailing and leading whitespace is trimmed.
 
 To build on our example above, here are some label filter queries and their behavior:
 
@@ -2602,10 +2602,63 @@ To build on our example above, here are some label filter queries and their beha
 | `ginkgo --label-filter="network && !slow"` | Run specs labelled `network` that aren't `slow` |
 | `ginkgo --label-filter=/library/` | Run specs with labels matching the regular expression `library` - this will match the three library-related specs in our example.
 
+##### Label Sets
+
+In addition to flat strings, Labels can also construct sets.  If a label has the format `KEY:VALUE` then a set with key `KEY` is created and the value `VALUE` is added to the set.  For example:
+
+```go
+Describe("The Library API", Label("API:Library"), func() {
+  It("can fetch a list of books", func() {
+    // has the labels [API:Library]
+    // API is a set with value {Library}
+  })
+  It("can fetch a list of books by shelf", Label("API:Shelf", "Readiness:Alpha"), func() {
+    // has the labels [API:Library, API:Shelf, Readiness:Alpha]
+    // API is a set with value {Library, Shelf}
+    // Readiness is a set with value {Alpha}
+
+  })
+  It("can fetch a list of books by zip code", Label("API:Geo", "Readiness:Beta"), func() {
+    // has the labels [API:Library, API:Geo, Readiness:Beta]
+    // API is a set with value {Library, Geo}
+    // Readiness is a set with value {Beta}
+  })
+})
+```
+
+Label filters can operate on sets using the notation: `KEY: SET_OPERATION <ARGUMENT>`. The following set operations are supported:
+
+| Set Operation | Argument | Description |
+| --- | --- | --- |
+| `isEmpty` | None | Matches if the set with key `KEY` is empty (i.e. no label of the form `KEY:*` exists) |
+| `containsAny` | `SINGLE_VALUE` or `{VALUE1, VALUE2, ...}` | Matches if the `KEY` set contains _any_ of the elements in `ARGUMENT` |
+| `containsAll` | `SINGLE_VALUE` or `{VALUE1, VALUE2, ...}` | Matches if the `KEY` set contains _all_ of the elements in `ARGUMENT` |
+| `consistsOf` | `SINGLE_VALUE` or `{VALUE1, VALUE2, ...}` | Matches if the `KEY` set contains _exactly_ the elements in `ARGUMENT` |
+| `isSubsetOf` | `SINGLE_VALUE` or `{VALUE1, VALUE2, ...}` | Matches if the elements in the `KEY` set are a subset of the elements in `ARGUMENT` |
+
+leading and trailing whitespace is alwasy trimmed around keys and values and comparisons are always case-insensitive.  Keys and values in the filter-language set operations are always literals; regular expressions are not supported.  A special note should be made about the behavior of `isSubsetOf`: if the `KEY` set is empty then the filter will always match.  This is because an empty set is always a subset of any other set.
+
+You can combine set operations with other label filters using the logical operators.  For example: `ginkgo --label-filter="integration && !slow && Readiness: isSubsetOf {Beta, RC}"` will run all tests that have the label `integration`, do not have the label `slow` and have a `Readiness` set that is a subset of `{Beta, RC}`.  This would exclude `Readiness:Alpha` but include specs with `Readiness:Beta` and `Readiness:RC` as well as specs with no `Readiness:*` label.
+
+Some more examples:
+
+| Query | Behavior |
+| --- | --- |
+| `ginkgo --label-filter="API: consistsOf {Library, Geo}"` | Match any specs for which the `API` set contains exactly `Library` and `Geo` |
+| `ginkgo --label-filter="API: containsAny Library"` | Match any specs for which the `API` set contains either `Library` |
+| `ginkgo --label-filter="Readiness: isEmpty"` | Match any specs for which the `Readiness` set is empty |
+| `ginkgo --label-filter="Readiness: isSubsetOf Beta && !(API: containsAny Geo)"` | Match any specs for which the `Readiness` set is a subset of `{Beta}` (or empty) and the `API` set does not contain `Geo` |
+
+Label sets are helpful for organizing and filtering large spec suites in which different specs satisfy multiple overlapping concerns.  The use of label set filters is intended to be a more powerful and expressive alterantive to the use of regular expressions.  If you find yourself using a regular expression, consider if you should be using a label set instead.
+
+##### Listing Labels
+
 You can list the labels used in a given package using the `ginkgo labels` subcommand.  This does a simple/naive scan of your test files for calls to `Label` and returns any labels it finds.
 
 You can iterate on different filters quickly with `ginkgo --dry-run -v --label-filter=FILTER`.  This will cause Ginkgo to tell you which specs it will run for a given filter without actually running anything.
 
+##### Runtime Label Evaluation
+
 If you want to have finer-grained control within a test about what code to run/not-run depending on what labels match/don't match the filter you can perform a manual check against the label-filter passed into Ginkgo like so:
 
 ```go
@@ -2620,6 +2673,8 @@ It("can save books remotely", Label("network", "slow", "library query") {
 
 here `GinkgoLabelFilter()` returns the configured label filter passed in via `--label-filter`.  With a setup like this you could run `ginkgo --label-filter="network && !performance"` - this would select the `"can save books remotely"` spec but not run the benchmarking code in the spec.  Of course, this could also have been modeled as a separate spec with the `performance` label.
 
+##### Suite-Level Labels
+
 Finally, in addition to specifying Labels on subject and container nodes you can also specify suite-wide labels by decorating the `RunSpecs` command with `Label`:
 
 ```go
@@ -2631,7 +2686,6 @@ func TestBooks(t *testing.T) {
 
 Suite-level labels apply to the entire suite making it easy to filter out entire suites using label filters.
 
-
 #### Location-Based Filtering
 
 Ginkgo allows you to filter specs based on their source code location from the command line.  You do this using the `ginkgo --focus-file` and `ginkgo --skip-file` flags.  Ginkgo will only run specs that are in files that _do_ match the `--focus-file` filter *and* _don't_ match the `--skip-file` filter.  You can provide multiple `--focus-file` and `--skip-file` flags.  The `--focus-file`s will be ORed together and the `--skip-file`s will be ORed together.

integration/_fixtures/filter_fixture/widget_b_test.go

@@ -13,11 +13,15 @@ var _ = Describe("WidgetB", func() {
 
 	})
 
+	It("fish", Label("Feature:Alpha"), func() {
+
+	})
+
 	It("cat fish", func() {
 
 	})
 
-	It("dog fish", func() {
+	It("dog fish", Label("Feature:Beta"), func() {
 
 	})
 })

integration/filter_test.go

@@ -21,7 +21,7 @@ var _ = Describe("Filter", func() {
 			"--focus-file=sprocket", "--focus-file=widget:1-24", "--focus-file=_b:24-42",
 			"--skip-file=_c",
 			"--json-report=report.json",
-			"--label-filter=TopLevelLabel && !SLOW",
+			"--label-filter=TopLevelLabel && !SLOW && !(Feature: containsAny Alpha)",
 		)
 		Eventually(session).Should(gexec.Exit(0))
 		specs := Reports(fm.LoadJSONReports("filter", "report.json")[0].SpecReports)
@@ -43,6 +43,8 @@ var _ = Describe("Filter", func() {
 			"SprocketA cat", "SprocketB cat", "WidgetA cat", "WidgetB cat", "More WidgetB cat",
 			// fish is in -focus but cat is in -skip
 			"SprocketA cat fish", "SprocketB cat fish", "WidgetA cat fish", "WidgetB cat fish", "More WidgetB cat fish",
+			// Tests with Feature:Alpha
+			"WidgetB fish",
 			// Tests labelled 'slow'
 			"WidgetB dog",
 			"SprocketB fish",
@@ -95,7 +97,7 @@ var _ = Describe("Filter", func() {
 		It("can list labels", func() {
 			session := startGinkgo(fm.TmpDir, "labels", "-r")
 			Eventually(session).Should(gexec.Exit(0))
-			Ω(session).Should(gbytes.Say(`filter: \["TopLevelLabel", "slow"\]`))
+			Ω(session).Should(gbytes.Say(`filter: \["Feature:Alpha", "Feature:Beta", "TopLevelLabel", "slow"\]`))
 			Ω(session).Should(gbytes.Say(`labels: \["beluga", "bird", "cat", "chicken", "cow", "dog", "giraffe", "koala", "monkey", "otter", "owl", "panda"\]`))
 			Ω(session).Should(gbytes.Say(`nolabels: No labels found`))
 			Ω(session).Should(gbytes.Say(`onepkg: \["beluga", "bird", "cat", "chicken", "cow", "dog", "giraffe", "koala", "monkey", "otter", "owl", "panda"\]`))

internal/focus_test.go

@@ -254,6 +254,27 @@ var _ = Describe("Focus", func() {
 			})
 		})
 
+		Context("when configured with a label set filter", func() {
+			BeforeEach(func() {
+				conf.LabelFilter = "Feature: consistsOf {A, B} || Feature: containsAny C"
+				specs = Specs{
+					S(N(ntCon, Label("Feature:A", "dog")), N(ntIt, "A", Label("fish"))),               //skip because fish no feature:B
+					S(N(ntCon, Label("Feature:A", "dog")), N(ntIt, "B", Label("apple", "Feature:B"))), //include because has Feature:A and Feature:B
+					S(N(ntCon, Label("Feature:A")), N(ntIt, "C", Label("Feature:B", "Feature:D"))),    //skip because it has Feature:D
+					S(N(ntCon, Label("Feature:C")), N(ntIt, "D", Label("fish", "Feature:D"))),         //include because it has Feature:C
+					S(N(ntCon, Label("cow")), N(ntIt, "E")),                                           //skip because no Feature:
+					S(N(ntCon, Label("Feature:A", "Feature:B")), N(ntIt, "F", Pending)),               //skip because pending
+				}
+			})
+
+			It("applies the label filters", func() {
+				specs, hasProgrammaticFocus := internal.ApplyFocusToSpecs(specs, description, suiteLabels, conf)
+				Ω(harvestSkips(specs)).Should(Equal([]bool{true, false, true, false, true, true}))
+				Ω(hasProgrammaticFocus).Should(BeFalse())
+
+			})
+		})
+
 		Context("when configured with a label filter that filters on the suite level label", func() {
 			BeforeEach(func() {
 				conf.LabelFilter = "cat && TopLevelLabel"

internal/internal_integration/labels_test.go

@@ -27,10 +27,18 @@ var _ = Describe("Labels", func() {
 						It("H", rt.T("H"), Label("fish", "chicken"))
 					})
 				})
+				Describe("feature container", Label("Feature:Beta"), func() {
+					It("I", rt.T("I"), Label("Feature: Gamma"))
+					Describe("inner container", Label(" feature : alpha "), func() {
+						It("J", rt.T("J"), Label("Feature:Alpha"))
+						It("K", rt.T("K"), Label("Feature:Delta", "Feature:Beta"))
+					})
+
+				})
 			})
 		}
 		BeforeEach(func() {
-			conf.LabelFilter = "TopLevelLabel && (dog || cow)"
+			conf.LabelFilter = "TopLevelLabel && (dog || cow) || Feature: containsAny Alpha"
 			success, hPF := RunFixture("labelled tests", fixture)
 			Ω(success).Should(BeTrue())
 			Ω(hPF).Should(BeFalse())
@@ -68,6 +76,18 @@ var _ = Describe("Labels", func() {
 			Ω(reporter.Did.Find("H").ContainerHierarchyLabels).Should(Equal([][]string{{}, {"giraffe"}, {"cow"}}))
 			Ω(reporter.Did.Find("H").LeafNodeLabels).Should(Equal([]string{"fish", "chicken"}))
 			Ω(reporter.Did.Find("H").Labels()).Should(Equal([]string{"giraffe", "cow", "fish", "chicken"}))
+
+			Ω(reporter.Did.Find("I").ContainerHierarchyLabels).Should(Equal([][]string{{}, {"Feature:Beta"}}))
+			Ω(reporter.Did.Find("I").LeafNodeLabels).Should(Equal([]string{"Feature: Gamma"}))
+			Ω(reporter.Did.Find("I").Labels()).Should(Equal([]string{"Feature:Beta", "Feature: Gamma"}))
+
+			Ω(reporter.Did.Find("J").ContainerHierarchyLabels).Should(Equal([][]string{{}, {"Feature:Beta"}, {"feature : alpha"}}))
+			Ω(reporter.Did.Find("J").LeafNodeLabels).Should(Equal([]string{"Feature:Alpha"}))
+			Ω(reporter.Did.Find("J").Labels()).Should(Equal([]string{"Feature:Beta", "feature : alpha", "Feature:Alpha"}))
+
+			Ω(reporter.Did.Find("K").ContainerHierarchyLabels).Should(Equal([][]string{{}, {"Feature:Beta"}, {"feature : alpha"}}))
+			Ω(reporter.Did.Find("K").LeafNodeLabels).Should(Equal([]string{"Feature:Delta", "Feature:Beta"}))
+			Ω(reporter.Did.Find("K").Labels()).Should(Equal([]string{"Feature:Beta", "feature : alpha", "Feature:Delta"}))
 		})
 
 		It("includes suite labels in the suite report", func() {
@@ -76,11 +96,11 @@ var _ = Describe("Labels", func() {
 		})
 
 		It("honors the LabelFilter config and skips tests appropriately", func() {
-			Ω(rt).Should(HaveTracked("B", "C", "D", "F", "H"))
-			Ω(reporter.Did.WithState(types.SpecStatePassed).Names()).Should(ConsistOf("B", "C", "D", "F", "H"))
-			Ω(reporter.Did.WithState(types.SpecStateSkipped).Names()).Should(ConsistOf("A", "E"))
+			Ω(rt).Should(HaveTracked("B", "C", "D", "F", "H", "J", "K"))
+			Ω(reporter.Did.WithState(types.SpecStatePassed).Names()).Should(ConsistOf("B", "C", "D", "F", "H", "J", "K"))
+			Ω(reporter.Did.WithState(types.SpecStateSkipped).Names()).Should(ConsistOf("A", "E", "I"))
 			Ω(reporter.Did.WithState(types.SpecStatePending).Names()).Should(ConsistOf("G"))
-			Ω(reporter.End).Should(BeASuiteSummary(true, NPassed(5), NSkipped(2), NPending(1), NSpecs(8), NWillRun(5)))
+			Ω(reporter.End).Should(BeASuiteSummary(true, NPassed(7), NSkipped(3), NPending(1), NSpecs(11), NWillRun(7)))
 		})
 	})
 

internal/node_test.go

@@ -444,9 +444,9 @@ var _ = Describe("Constructing nodes", func() {
 		})
 
 		It("validates labels", func() {
-			node, errors := internal.NewNode(dt, ntIt, "", body, cl, Label("A", "B&C", "C,D", "C,D ", "  "))
+			node, errors := internal.NewNode(dt, ntIt, "", body, cl, Label("A", "B&C", "C,D", "C,D ", "  ", ":Foo"))
 			Ω(node).Should(BeZero())
-			Ω(errors).Should(ConsistOf(types.GinkgoErrors.InvalidLabel("B&C", cl), types.GinkgoErrors.InvalidLabel("C,D", cl), types.GinkgoErrors.InvalidLabel("C,D ", cl), types.GinkgoErrors.InvalidEmptyLabel(cl)))
+			Ω(errors).Should(ConsistOf(types.GinkgoErrors.InvalidLabel("B&C", cl), types.GinkgoErrors.InvalidLabel("C,D", cl), types.GinkgoErrors.InvalidLabel("C,D ", cl), types.GinkgoErrors.InvalidEmptyLabel(cl), types.GinkgoErrors.InvalidLabel(":Foo", cl)))
 			Ω(dt.DidTrackDeprecations()).Should(BeFalse())
 		})
 	})