Getting our automation to 100%

One of the challenges we have taken on since last year has been driving our test passes to be 100% automated.  Historically, a test pass has had a manual testing component - a human sits in front of a computer that is running a build of Tableau that we intend to ship to customers and validates the responses to test cases are the results we expect.  This can be a challenge because, as often happens, the manual testing phase can take more than a day.  If we get a new build of Tableau each day - and we can get them more often than that - it is possible that we can never finish a manual test pass. 

It also can become very difficult to predict when a manual test pass will conclude.  If there are changes, test will need to validate if the change was expected or not, and while that may be obvious in some cases, it is not always the case.  Imagine that a new piece of hardware, such as a very high resolution monitor, comes out the day before the test pass.  It could be the case that no one on the team knew this would happen and no one has planned for how Tableau should work in that case.  This can slow down the evaluation of the results of the test pass.

To get around this, we embarked on getting enough automation in place that we can avoid all manual testing when a new build of Tableau is released for testing.  For us, this just means looking at automation results and verifying that no tests are failing.  That is my definition of 100% - that we can make our decision based on automation results.  That was a lot of work to get here, and even more work to remain at this level, but it does make testing much more reliable and predictable as time goes by. 

This also speeds up the new monitor test example I gave above.  At this point, all the test results are available so the extent of the test pass is done "all at once."  There is no  back and forth with trying each feature one at a time on the new monitor - all the automation runs and gives a report to the humans that can now start making decisions.  Without the automation, the decision process is slowed down by data gathering, and this is where automation can really speed up the cycle.

We did some tasks other than simply adding automation to help us get here and I'll cover those next.

Questions, comments, concerns and criticisms always welcome,

John