Index: doc/how-to-write-tests.txt |
=== added file 'doc/how-to-write-tests.txt' |
--- doc/how-to-write-tests.txt 1970-01-01 00:00:00 +0000 |
+++ doc/how-to-write-tests.txt 2014-05-08 22:32:25 +0000 |
@@ -0,0 +1,304 @@ |
+How to write tests |
+================== |
+ |
+On the whole, new or updated code will not pass review unless there are tests |
+associated with the code. For code additions, the tests should cover as much |
+of the new code as practical, and for code changes, either the tests should be |
+updated, or at least the tests that already exist that cover the refactored |
+code should be identified when requesting a review to show that there is already |
+test coverage, and that the refactoring didn't break anything. |
+ |
+ |
+go test and gocheck |
+------------------- |
+ |
+The `go test` command is used to run the tests. Juju uses the `gocheck` package |
+("launchpad.net/gocheck") to provide a checkers and assert methods for the test |
+writers. The use of gocheck replaces the standard `testing` library. |
+ |
+Across all of the tests in juju-core, the gocheck package is imported |
+with a shorter alias, because it is used a lot. |
+ |
+```go |
+import ( |
+ // system packages |
+ |
+ gc "launchpad.net/gocheck" |
+ |
+ // juju packages |
+) |
+``` |
+ |
+ |
+setting up tests for new packages |
+--------------------------------- |
+ |
+Lets say we are creating a new provider for "magic" cloud, and we have a package |
+called "magic" that lives at "launchpad.net/juju-core/provider/magic". The |
+general approach for testing in juju is to have the tests in a separate package. |
+Continuing with this example the tests would be in a package called "magic_test". |
+ |
+A common idiom that has occurred in juju is to setup to gocheck hooks in a special |
+file called `package_test.go` that would look like this: |
+ |
+ |
+```go |
+// Copyright 2014 Canonical Ltd. |
+// Licensed under the AGPLv3, see LICENCE file for details. |
+ |
+package magic_test |
+ |
+import ( |
+ "testing" |
+ |
+ gc "launchpad.net/gocheck" |
+) |
+ |
+func Test(t *testing.T) { |
+ gc.TestingT(t) |
+} |
+``` |
+ |
+or |
+ |
+```go |
+// Copyright 2014 Canonical Ltd. |
+// Licensed under the AGPLv3, see LICENCE file for details. |
+ |
+package magic_test |
+ |
+import ( |
+ stdtesting "testing" |
+ |
+ "launchpad.net/juju-core/testing" |
+) |
+ |
+func Test(t *stdtesting.T) { |
+ testing.MgoTestPackage(t) |
+} |
+``` |
+ |
+The key difference here is that the first one just hooks up `gocheck` |
+so it looks for the `gocheck` suites in the package. The second makes |
+sure that there is a mongo available for the duration of the package tests. |
+ |
+A general rule is not to setup mongo for a package unless you really |
+need to as it is extra overhead. |
+ |
+ |
+writing the test files |
+---------------------- |
+ |
+Normally there will be a test file for each file with code in the package. |
+For a file called `config.go` there should be a test file called `config_test.go`. |
+ |
+The package should in most cases be the same as the normal files with a "_test" suffix. |
+In this way, the tests are testing the same interface as any normal user of the |
+package. It is reasonably common to want to modify some internal aspect of the package |
+under test for the tests. This is normally handled by a file called `export_test.go`. |
+Even though the file ends with `_test.go`, the package definition is the same as the |
+normal source files. In this way, for the tests and only the tests, additional |
+public symbols can be defined for the package and used in the tests. |
+ |
+Here is an annotated extract from `provider/local/export_test.go` |
+ |
+```go |
+// The package is the "local" so it has access to the package symbols |
+// and not just the public ones. |
+package local |
+ |
+import ( |
+ "github.com/juju/testing" |
+ gc "launchpad.net/gocheck" |
+ |
+ "launchpad.net/juju-core/environs/config" |
+) |
+ |
+var ( |
+ // checkIfRoot is a variable of type `func() bool`, so CheckIfRoot is |
+ // a pointer to that variable so we can patch it in the tests. |
+ CheckIfRoot = &checkIfRoot |
+ // providerInstance is a pointer to an instance of a private structure. |
+ // Provider points to the same instance, so public methods on that instance |
+ // are available in the tests. |
+ Provider = providerInstance |
+) |
+ |
+// ConfigNamespace is a helper function for the test that steps through a |
+// number of private methods or variables, and is an alternative mechanism |
+// to provide functionality for the tests. |
+func ConfigNamespace(cfg *config.Config) string { |
+ env, _ := providerInstance.Open(cfg) |
+ return env.(*localEnviron).config.namespace() |
+} |
+``` |
+ |
+Suites and Juju base suites |
+--------------------------- |
+ |
+With gocheck tests are grouped into Suites. Each suite has distinct |
+set-up and tear-down logic. Suites are often composed of other suites |
+that provide specific set-up and tear-down behaviour. |
+ |
+There are three main suites: |
+ |
+ * /testing/testbase.LoggingSuite (testing/testbase/log.go) |
+ * /testing.FakeHomeSuite (testing/environ.go) |
+ * /juju/testing.JujuConnSuite (juju/testing/conn.go) |
+ |
+The second two have the LoggingSuite functionality included through |
+composition. The LoggingSuite is also composed of the LoggingSuite from |
+github.com/juju/testing, which brings in the CleanupSuite from the same. |
+The CleanupSuite has the functionality around patching environment |
+variables and normal variables for the duration of a test. It also |
+provides a clean-up stack that gets called when the test teardown happens. |
+ |
+The FakeHomeSuite creates a temporary directory and sets the HOME environment |
+variable to it. It also creates ~/.juju and a simple environments.yaml file, |
+~/.ssh with a fake id_rsa.pub key, it isolates the test from the JUJU_HOME, |
+JUJU_ENV, and JUJU_LOGGING_CONFIG environment variables. |
+ |
+The JujuConnSuite does this and more. It also sets up a state server and api |
+server. This is one problem with the JujuConnSuite, it almost always does a |
+lot more than you actually want or need. This should really be broken into |
+smaller component parts that make more sense. If you can get away with not |
+using the JujuConnSuite, you should try. |
+ |
+To create a new suite composed of one or more of the suites above, you can do |
+something like: |
+ |
+```go |
+type ToolsSuite struct { |
+ testbase.LoggingSuite |
+ dataDir string |
+} |
+ |
+var _ = gc.Suite(&ToolsSuite{}) |
+ |
+``` |
+ |
+If there is no extra setup needed, then you don't need to specify any |
+set-up or tear-down methods as the LoggingSuite has them, and they are |
+called by default. |
+ |
+If you did want to do something, say, create a directory and save it in |
+the dataDir, you would do something like this: |
+ |
+```go |
+func (t *ToolsSuite) SetUpTest(c *gc.C) { |
+ t.LoggingSuite.SetUpTest(c) |
+ t.dataDir = c.MkDir() |
+} |
+``` |
+ |
+If the test suite has multiple contained suites, please call them in the |
+order that they are defined, and make sure something that is composed from |
+the LoggingSuite is first. They should be torn down in the reverse order. |
+ |
+Even if the code that is being tested currently has no logging in it, it |
+is a good idea to use the LoggingSuite as a base for two reasons: |
+ * it brings in something composed of the CleanupSuite |
+ * if someone does add logging later, it is captured and doesn't polute |
+ the logging output |
+ |
+ |
+Patching variables and the environment |
+-------------------------------------- |
+ |
+Inside a test, and assuming that the Suite has a CleanupSuite somewhere |
+in the composition tree, there are a few very helpful functions. |
+ |
+```go |
+ |
+var foo int |
+ |
+func (s *someTest) TestFubar(c *gc.C) { |
+ // The TEST_OMG environment value will have "new value" for the duration |
+ // of the test. |
+ s.PatchEnvironment("TEST_OMG", "new value") |
+ |
+ // foo is set to the value 42 for the duration of the test |
+ s.PatchValue(&foo, 42) |
+} |
+``` |
+ |
+PatchValue works with any matching type. This includes function variables. |
+ |
+ |
+Checkers |
+-------- |
+ |
+Checkers are a core concept of `gocheck` and will feel familiar to anyone |
+who has used the python testtools. Assertions are made on the gocheck.C |
+methods. |
+ |
+```go |
+c.Check(err, gc.IsNil) |
+c.Assert(something, gc.Equals, somethingElse) |
+``` |
+ |
+The `Check` method will cause the test to fail if the checker returns |
+false, but it will continue immediately cause the test to fail and will |
+continue with the test. `Assert` if it fails will cause the test to |
+immediately stop. |
+ |
+For the purpose of further discussion, we have the following parts: |
+ |
+ `c.Assert(observed, checker, args...)` |
+ |
+The key checkers in the `gocheck` module that juju uses most frequently are: |
+ |
+ * `IsNil` - the observed value must be `nil` |
+ * `NotNil` - the observed value must not be `nil` |
+ * `Equals` - the observed value must be the same type and value as the arg, |
+ which is the expected value |
+ * `DeepEquals` - checks for equality for more complex types like slices, |
+ maps, or structures. This is DEPRECATED in favour of the DeepEquals from |
+ the `github.com/juju/testing/checkers` covered below |
+ * `ErrorMatches` - the observed value is expected to be an `error`, and |
+ the arg is a string that is a regular expression, and used to match the |
+ error string |
+ * `Matches` - a regular expression match where the observed value is a string |
+ * `HasLen` - the expected value is an integer, and works happily on nil |
+ slices or maps |
+ |
+ |
+Over time in the juju project there were repeated patterns of testing that |
+were then encoded into new and more complicated checkers. These are found |
+in `github.com/juju/testing/checkers`, and are normally imported with the |
+alias `jc`. |
+ |
+The matchers there include (not an exclusive list): |
+ |
+ * `IsTrue` - just an easier way to say `gc.Equals, true` |
+ * `IsFalse` - observed value must be false |
+ * `GreaterThan` - for integer or float types |
+ * `LessThan` - for integer or float types |
+ * `HasPrefix` - obtained is expected to be a string or a `Stringer`, and |
+ the string (or string value) must have the arg as start of the string |
+ * `HasSuffix` - the same as `HasPrefix` but checks the end of the string |
+ * `Contains` - obtained is a string or `Stringer` and expected needs to be |
+ a string. The checker passes if the expected string is a substring of the |
+ obtained value. |
+ * `DeepEquals` - works the same way as the `gocheck.DeepEquals` except |
+ gives better errors when the values do not match |
+ * `SameContents` - obtained and expected are slices of the same type, |
+ the checker makes sure that the values in one are in the other. They do |
+ not have the be in the same order. |
+ * `Satisfies` - the arg is expected to be `func(observed) bool` |
+ often used for error type checks |
+ * `IsNonEmptyFile` - obtained is a string or `Stringer` and refers to a |
+ path. The checker passes if the file exists, is a file, and is not empty |
+ * `IsDirectory` - works in a similar way to `IsNonEmptyFile` but passes if |
+ the path element is a directory |
+ * `DoesNotExist` - also works with a string or `Stringer`, and passes if |
+ the path element does not exist |
+ |
+ |
+ |
+Good tests |
+---------- |
+ |
+Good tests should be: |
+ * small and obviously correct |
+ * isolated from any system or environment values that may impact the test |