Unit tests are functions that test specific pieces of code from a program or package. The primary objective of unit tests is to check the correctness of an application, leading to better software that is more robust, has fewer bugs, and is more stable. There are many ways to test software other than unit tests. Let's look at the test pyramid.
As the test pyramid metaphor suggests, unit tests are the foundation of all tests. Unit tests are the easiest to write code and the most powerful. Unit tests are especially helpful for microservices. In microservices, each system must work independently. Therefore, unit tests that allow testing against one service while mocking another are very important.
This LFX quarter I got to get my hands on LitmusChaos, a CNCF incubating opensource project that dives deep on making cloud-native chaos-engineering accessible to multiple developer personas.
In this post, I'll discuss about the testing methodologies and strategies I took to improve the code coverage as well as modify the structure (for a Go based framework) of the litmus project to make it more robust and agile for testing. Before starting, We highly recommend watching this video.
Table of Contents
- Naming Convention
- Testing Structure
- What to test?
- Test Interface
- Mocking
- Table-Driven Test
- Solitary Tests vs Sociable Tests
- Defer vs t.Cleanup()
- Dealing with Before & After tests
- Do not assert an Error message
- Avoid Flaky Test
- Fuzz testing
- Code Coverage
- Test Coverage Report(UI)
- CI Integration
- Conclusion
Naming Convention
According to the Golang testing package, we follow these naming conventions.
func helloWorld() {} // target function
func TestHelloWorld(t *testing.T) {} // test function
type FooStruct struct {}
func (f *FooStruct) Bar() {} // target method
func TestFooStruct_Bar(t *testing.T){} // test function
Testing Structure
A good structure for all unit tests follows these,
- Set up the test data
- Call your method under the test
- Assert that the expected results are returned
These three steps are replaced with “given”, “when”, “then” in BDD. You can make unit test codes easier if you adopt this pattern.
// example of given-when-then pattern
func TestChaosHubService_DeleteChaosHub(t *testing.T) {
t.Run("success", func(t *testing.T) {
// given
findResult := bson.D{
{"project_id", "1"},
{"hub_name", "hub1"},
{"hub_id", "1"},
}
// when
_, err := mockService.DeleteChaosHub(context.Background(), "1", "1")
// then
assert.NoError(t, err)
})
}
What to test?
Tests that are too close to the production code are not recommended. As soon as you fix your production code, You need to change the test code too(the test code will be broken)! You rather test for observable behavior. Here’s what Martin Fowler’s Blog suggests.
Think about if I enter values x and y, will the result be z? instead of If I enter x and y, will the method call class A first, then call class B and then return the result of class A plus the result of class B?
We can accomplish this by subtests in the Golang testing package. We don’t have to write separate functions. Instead, use t.Run() so that we can verify the result by various inputs in one function.
// example of subtests
func TestChaosHubService_UpdateChaosHub(t *testing.T) {
t.Run("cannot find same project_id hub", func(t *testing.T) {
// given codes
// when codes
// then codes
})
t.Run("success : updated hub type is remote", func(t *testing.T) {
// given codes
// when codes
// then codes
})
t.Run("success : updated hub type is not remote", func(t *testing.T) {
// given codes
// when codes
// then codes
})
t.Run("success : updated hub type is not remote, not changed data", func(t *testing.T) {
// given codes
// when codes
// then codes
})
}
Test Interface
As previously mentioned, We need to test functions' desirable results, not all lines of production code. With subtests, Interface can help what you focus on.
The interface is like a contract that expresses desired behavior. For example, the Service interface has an AddChaosHub function.
type Service interface {
AddChaosHub(chaosHub CreateChaosHubRequest) (*model.ChaosHub, error)
}
We have not implemented the interface yet. But We can write test code. This method is the method for adding a ChaosHub. If the request parameter is valid, Method success creates a chaoshub object and return object. If not, return the error. According to these instructions, We can write test code like below.
// example of unit test of AddChaosHub function
func TestChaosHubService_AddChaosHub(t *testing.T) {
// given
newHub := model.CreateChaosHubRequest{
ProjectID: "4",
HubName: "Litmus ChaosHub",
}
t.Run("already existed hub name", func(t *testing.T) {
// given
findResult := []interface{}{
bson.D{{"project_id", "3"}, {"hub_name", "Litmus ChaosHub"}},
}
cursor, _ := mongo.NewCursorFromDocuments(findResult, nil, nil)
mongoOperator.On(
"List", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
).Return(cursor, nil).Once()
// when
_, err := mockService.AddChaosHub(context.Background(), newHub)
// then
assert.Error(t, err)
})
t.Run("success", func(t *testing.T) {
// given
findResult := []interface{}{
bson.D{{"project_id", "1"}, {"hub_name", "hub1"}},
}
cursor, _ := mongo.NewCursorFromDocuments(findResult, nil, nil)
mongoOperator.On(
"List", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
).Return(cursor, nil).Once()
mongoOperator.On(
"Create", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
).Return(nil).Once()
// when
t.Cleanup(func() { clearCloneRepository(newHub.ProjectID, newHub.HubName) })
target, err := mockService.AddChaosHub(context.Background(), newHub)
// then
assert.NoError(t, err)
assert.Equal(t, newHub.HubName, target.HubName)
})
}
Mocking
LitmusChaos project adopted layered architecture.
By applying this architecture, we can see the effect of low coupling and high cohesion. Changes to GraphQL logic only require modifications to the resolver layer, and changes to business logic only require modifications to the service layer. Changes to MongoDB logic only require changes to the operator layer. Since we will be testing on all layers, there is no need to test the sub-layers of each layer, so we mock the sub-layers. In LitmusChaos, we used a library called testify for mocking.
Here’s an example. In graphql-server, ChaosHubService needs a MongoOperator to interact with MongoDB. But in unit tests, We don’t have to use real databases, We mocked MongoOperator.
// example of MockOperator (mongoDB)
type MongoOperator struct {
mock.Mock
}
// we don't have to write real logic. Mock object's method will
// be replaced at test function.
func (m MongoOperator) Get(ctx context.Context, collectionType int, query bson.D) (*mongo.SingleResult, error) {
args := m.Called(ctx, collectionType, query)
return args.Get(0).(*mongo.SingleResult), args.Error(1)
}
func (m MongoOperator) Update(ctx context.Context, collectionType int, query, update bson.D, opts ...*options.UpdateOptions) (*mongo.UpdateResult, error) {
args := m.Called(ctx, collectionType, query, update, opts)
return args.Get(0).(*mongo.UpdateResult), args.Error(1)
}
// chaoshub_test package
// Mock object is injected instead of real object.
var (
mongoOperator = new(mocks.MongoOperator)
mockOperator = dbSchemaChaosHub.NewChaosHubOperator(mongoOperator)
mockService = chaoshub.NewService(mockOperator)
)
func TestChaosHubService_DeleteChaosHub(t *testing.T) {
t.Run("cannot find same project_id hub", func(t *testing.T) {
// given
// setup expectation by using On() function.
mongoOperator.On(
"Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
).Return(&mongo.SingleResult{}, errors.New("")).Once()
// when
_, err := mockService.DeleteChaosHub(context.Background(), "1", "1")
// then
assert.Error(t, err)
})
t.Run("success", func(t *testing.T) {
// given
findResult := bson.D{
{"project_id", "1"}, {"hub_name", "hub1"}, {"hub_id", "1"},
}
singleResult := mongo.NewSingleResultFromDocument(findResult, nil, nil)
mongoOperator.On(
"Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything,
).Return(singleResult, nil).Once()
mongoOperator.On(
"Update", mock.Anything, mongodb.ChaosHubCollection, mock.Anything, mock.Anything, mock.Anything,
).Return(&mongo.UpdateResult{MatchedCount: 1}, nil).Once()
// when
_, err := mockService.DeleteChaosHub(context.Background(), "1", "1")
// then
assert.NoError(t, err)
})
}
Table-Driven Test
You can check the basics of table-driven tests here. By adopting a table-driven test approach, We can reduce the amount of repetitive code compared to repeating the same code for each test and make it straightforward to add more test cases. More details on the Golang dev blog.
// example of table-driven Test
func TestChaosHubService_UpdateChaosHub(t *testing.T) {
// given
utils.Config.RemoteHubMaxSize = "1000000000"
testCases := []struct {
name string
hub model.UpdateChaosHubRequest
got bson.D
isError bool
}{
{
name: "cannot find same project_id hub",
hub: model.UpdateChaosHubRequest{
ProjectID: "1",
HubName: "updated name",
},
isError: true,
},
{
name: "success : updated hub type is remote",
hub: model.UpdateChaosHubRequest{
ProjectID: "1",
HubName: "updated name",
RepoURL: "https://github.com/litmuschaos/chaos-charts/archive/refs/heads/master.zip",
},
got: bson.D{{"project_id", "1"}, {"hub_name", "hub1"}, {"hub_type", "REMOTE"}},
isError: false,
},
{
name: "success : updated hub type is not remote",
hub: model.UpdateChaosHubRequest{
ProjectID: "1",
HubName: "updated name",
RepoURL: "https://github.com/litmuschaos/chaos-charts",
RepoBranch: "master",
IsPrivate: false,
},
got: bson.D{{"project_id", "1"}, {"hub_name", "hub1"}},
isError: false,
},
{
name: "success : updated hub type is not remote, not changed data",
hub: model.UpdateChaosHubRequest{
ProjectID: "1",
HubName: "updated name",
RepoURL: "https://github.com/litmuschaos/chaos-charts",
RepoBranch: "master",
IsPrivate: false,
},
got: bson.D{{"project_id", "1"}, {"hub_name", "updated name"}, {"repo_url", "https://github.com/litmuschaos/chaos-charts"}, {"repo_branch", "master"}, {"is_private", false}},
isError: false,
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
// given
if tc.isError {
// given
mongoOperator.On("Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything).Return(&mongo.SingleResult{}, errors.New("")).Once()
// when
_, err := mockService.UpdateChaosHub(context.Background(), tc.hub)
// then
assert.Error(t, err)
} else {
singleResult := mongo.NewSingleResultFromDocument(tc.got, nil, nil)
mongoOperator.On("Get", mock.Anything, mongodb.ChaosHubCollection, mock.Anything).Return(singleResult, nil).Once()
mongoOperator.On("Update", mock.Anything, mongodb.ChaosHubCollection, mock.Anything, mock.Anything, mock.Anything).Return(&mongo.UpdateResult{MatchedCount: 1}, nil).Once()
// when
t.Cleanup(func() { clearCloneRepository(tc.hub.ProjectID, tc.hub.HubName) })
target, err := mockService.UpdateChaosHub(context.Background(), tc.hub)
// then
assert.NoError(t, err)
assert.Equal(t, tc.hub.HubName, target.HubName)
}
})
}
}
Solitary Tests vs Sociable Tests
There are two terms in the unit test world, Sociable Tests and Solitary Tests. See the illustration below for an explanation of the two terms.
Previously, We talked about Mocking. With Mocking, We can make all unit test codes Solitary Tests. However, if you want your code to behave based on the results of actual actions in the lower layers, you should use Sociable Tests.
For example, In graphql-server’s ChaosHub package, ChaosHubService uses chaosHubOps.GitClone() in the AddChaosHub method. The AddChaosHub method performs the git clone through a real ChaosHub url, which means that if you mock the git clone part, you need additional logic to determine if the url is valid. Also, the GetExperiment method performs file I/O operations based on the cloned repository. For these cases, Sociable Tests, which do not mock chaosHubOps.GitClone(), is more appropriate.
Defer vs t.Cleanup()
If you need to clean up the resources used by your test, use t.Cleanup(). Unlike the defer function, this function also works fine in the event of a panic. You can check the details in this link.
// Example of t.Cleanup() for cleanup resources
func TestChaosHubService_AddChaosHub(t *testing.T) {
t.Run("success", func(t *testing.T) {
// given codes ...
// when : called t.Cleanup() functions before when codes
t.Cleanup(func(){clearCloneRepository(newHub.ProjectID, newHub.HubName)})
target, err := mockService.AddChaosHub(context.Background(), newHub)
// then codes
})
}
Dealing with Before & After tests
Sometimes, We need to add additional tasks before or after tests. The Golang testing package gave us a solution. TestMain function can be declared per package. You can add additional processes like below.
func TestMain(m *testing.M) {
// pre-process
os.Exit(m.Run())
// post-process
}
You can use the init() function. But init() function cannot be used in After logic. So I recommend using the TestMain function rather than the init function.
Do not assert an Error message
The Error message is only for human consumption. That means, It can easily change. So, Rather than using an Error message, you can just check if the error is not nil. More details are in the following conversations.
Avoid Flaky Test
Test functions needed to be deterministic. Do not make Flaky Tests. Here are common causes of flakiness include:
- Poorly written tests.
- Async wait
- Test order dependency
- Concurrency
More details are in this link.
Fuzz testing
The unit test has limitations in that unit tests’ input must be added by the developer. Fuzz testing can test many edge cases like coding interviews so that we can prevent SQL injection, buffer overflow, and more. Fuzz testing involves injecting random data with your original test cases. You can make the Fuzz test function like below.
func Reverse(s string) string {
// function to reverse a string
}
func FuzzReverse(f *testing.F) {
// table-driven fuzzing
testcases := []string{"word1", "word2", "word3"}
for _, tc := range testcases {
f.Add(tc) // Use f.Add to provide a seed corpus
}
f.Fuzz(
func(t *testing.T, a string) { // Value of a will be auto generated and passed
// Assert that the length of the reversed string is the same as the original
},
)
}
And you can run unit tests with Fuzz like below.
go test -fuzz=Fuzz
More details are in this link.
Code Coverage
We use the Golang command tool to check test coverage. It's simple but really powerful. You don't need any extra tools if you already installed Go.
Golang command tool automatically installs, builds, and tests Go programs using nothing more than the source code as the build specification.
# Check Specific package's code coverage
# in the package root
go test -cover ./...
# Check the entire backend code coverage
# in the backend module root (in my case, graphql-server)
go test --coverpkg ./... -coverprofile cover.out ./... ; \
echo -n "total: " ; \
go tool cover -func=cover.out | tail -1 | awk '{print $NF}'
Test Coverage Report(UI)
The Golang command tool also gave us to check coverage by HTML UI. Official guide is here. Once you execute the entire backend unit tests, You can find cover.out file. If you executed the below code, now you can see a beautiful UI that can check coverage. the green color of codes is covered by your unit tests code and the red one is not.
# Check the entire backend code coverage
# in the project root
go test --coverpkg ./... -coverprofile cover.out ./... ; \
echo -n "total: " ; \
go tool cover -func=cover.out | tail -1 | awk '{print $NF}'
# cover.out to cover.html
go tool cover -html=cover.out -o cover.html
CI Integration
If you adopt CI / CD pipeline to your current project, you can integrate unit testing jobs to CI pipeline. CI jobs in the GitHub actions will execute unit tests so developers do not have to run them locally. Here's a sample.
name: build-pipeline
on:
pull_request: # this example CI job runs when you raise PR
branches:
- master
env:
DOCKER_BUILDKIT: 1
jobs:
changes:
runs-on: ubuntu-latest
backend-unit-tests:
runs-on: ubuntu-latest
needs:
- changes
steps:
- name: Checkout repository
uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: "1.16"
- name: Backend unit tests
shell: bash
run: |
# cd to the backend directory
# run your test here!
go test -cover ./...
docker-build-backend-server:
runs-on: ubuntu-latest
needs:
- changes
- backend-unit-tests
steps:
- name: Build backend server docker image
shell: bash
run: |
# run docker build job after all the tests are passed
As you seen, you can run unit tests before build docker image. If unit tests failed, no further jobs will be executed.
Conclusion
Through this post, We discussed several tips of how to write efficient unit test codes. If you have any questions or suggestions, please comment below or send me an email(lak9348@gmail.com)
As I mentioned before, I worked on writing unit tests in LitmusChaos. LitmusChaos can help you find weaknesses in your project by injecting chaos.
If you are interested in LitmusChaos, Join community! You can join the LitmusChaos community on GitHub and Slack.
Thank you for reading 🙏