Friday, March 16, 2018

A quick jaunt to Palo Alto

The bigger organization within Tableau of which I am a part has a rather large team at our office in Palo Alto.  We had an All Hands meeting on Monday this week and I was able to attend in person.  I always like visiting Silicon Valley and decided I would ramble a bit about why this week.

  1. The number of tech companies there is astounding.  Everywhere I look I see Microsoft, Amazon, Apple, iAm, Sophos, Salesforce and so on.  It just brings out the geek in me.
  2. The shuttle we used was a Tesla.  Repeat: the shuttle was a Tesla.
  3. The weather there is typically nicer than Seattle.  One exception: I was in San Francisco once and went from hail in the morning to 95 degrees in the afternoon.  Cue the Mark Twain quote.
  4. It's always nice to step away from my desk now and then to get some new thoughts going.
  5. Meeting the entire team in Palo Alto lets me a face with a name.

I also talked with a few folks that are facing some of the same challenges our team is facing.  Maybe we can collaborate to face them and not waste time duplicating work - nothing to do with Palo Alto, just good business.

All in all, an enjoyable trip worth the time investment.

Questions, comments, concerns and criticisms always welcome,

Wednesday, March 7, 2018

A code coverage analogy

One of the tasks I have on my plate is code coverage tracking.  I've written about this in the past and today I want to go over an analogy on the usefulness of this statistic.

First I can point out it is relatively easy to track.  We have plenty of tools to do this for us, get reports, etc…  But just because it is easy to track does not mean it is all that useful.  I can track the contents of my glove compartment each day, but I think we can all agree that this would not be a useful piece of data to have.  There is not much I would do differently on a day to day basis based on the contents of the glove compartment.

Code coverage, on the other hand, can be useful.  It tells me how much code I have to validate that is not covered by automated tests - it tells me a risk level.  By itself, it is not enough (look up Mars ClimateExplorer as a terrific case study).  It does help make a decision, though. 

Here's my analogy:  code coverage is to a test engineer as a patient's temperature is to a doctor.  It is one useful metric that can help isolate risk.  By itself, a 98.6F temperature does not tell a doctor that a patient is healthy.  It only can help eliminate some possible causes of illness - certain stages of the flu, for instance.  No doctor could reasonably base a diagnosis on one temperature measurement.  It is one statistic among many others such as blood pressure, heart rate, etc.. that can help lead to a more accurate diagnosis.

Likewise, just because I have a 98.6% rate of code coverage, I cannot pronounce a feature risk free and ready to ship.  I also have to look at scenario pass rates, international/localization status, accessibility, performance goals and so on. 

Too often we get trapped in a mindset of "this statistics by itself is useless so let's not bother tracking it."  While the first half of that sentence may be true (and is, for code coverage), the second half of that sentence does not follow.  If given a choice of having a patient's temperature included in a checkup or not having the temperature included, doctors will always want to track that statistic.  It helps give part of a bigger picture.

And that is exactly what code coverage does.

Questions, comments, concerns and criticisms always welcome,

Thursday, March 1, 2018

Amazing amount of mileage out of this one test file

One of the accumulations that happens to testers over time is the build up of a staggering array of test files.  Files that use "odd" characters in Japanese, a Finnish test script, an absolutely huge Excel file, etc…  I tend to keep any file that ever exposed a bug around, use it in automation and use it for ad hoc testing as time goes by.

I also have one very simple file that I use quite often.  Very basic, but very good for "smoke testing" : simply ensuring a new feature doesn't crash right out of the gate.

It's just a csv file with some simple numbers and text in it:


Super basic, but fast to load.  It also has a mix of integers and floating point numbers, named columns, some text values and multiple rows of data.  I use it at least a half dozen times per week and I wanted to share this since it makes a good point that sometimes a simple test case is enough to provide some value.  If my test - whatever it is - fails with this file, I know I am blocked and I can file a bug and move on.

And if it passes, then I know I can start to dive further into the test matrix I have.  There's no guarantee that any of those other tests will pass.  But since I love analogies, I see it like this.

This simple test matrix is the equivalent of starting a car.  If it fails, I know I won't be testing the brakes or driving on the highway.  But just because the car starts I still can't tell if it will work when put in gear.  But at least I know the basic test - the engine works, in this analogy - is passing.

Questions, comments, concerns and criticisms always welcome,

Wednesday, February 21, 2018

Integration Testing, Part 7 The value of integration tests

I was reading this blog post this morning (Arlo works here, by the way) in which the author gives some advantages of a good test suite.  It is a great read and I want to use it as a springboard to summing up the benefits of integration testing.

In our example, we had one integration test that compared the written specification of a proposed muffler combined with a proposed engine.  If the engineers want to use a new muffler, we can immediately tell if it can handle the exhaust from the engine currently being used.  We don't need to pay to order a test muffler, pay to remove the old one, install the new one and then finally find out it doesn't work.  (In this example, we would actually damage the engine or some other physical component.  That seldom happens in the software world - just restore the previous build and keep going).  This test can save thousands of dollars and several days worth of work if the muffler + engine combination are not compatible.

The actual hardware test will also pay off.  If the specification is not right, or if the engine is operating outside of specifications, we can find that out before the car is sold.  To say the least, that would benefit customer satisfaction since the engine will work properly always.  You can imagine the downside to shipping a defective car - especially if good testing would have alerted us to the defect early enough to address it.

Thanks for following along with the notional integration testing thought experiment.  I'm going to get back to doing that right now!

Questions, comments, concerns and criticisms always welcome,

Monday, February 12, 2018

Tangential (get it?) post: I am digging into this precision based article on Code Project

I've been a loyal follower of the Code Project for quite some time and even written a few articles for them a while back.

It's no secret that my team here at Tableau takes precision and accuracy very seriously and I just discovered this article about that:

Overflow and underflow are pretty well covered, but there are aspects of precision that are very hard to control.  I've been reading through this and thought I would share.  I'll finish up integration testing next time - this article was simply a very timely happenstance that I wanted to share.

Questions, comments, concerns and criticisms always welcome,

Wednesday, February 7, 2018

Integration Testing, part 6: Updating requirement based on integration testing

So now we have 2 types of integration tests giving us information on our muffler + engine system.  We are checking both the documentation (data sheets) of each component to ensure they will work together.  We also test the physical system itself to validate the real world behavior.  This way we can ensure the car will actually work after we build it.

There is one last aspect of this I want to cover.  There is a possibility that we actually decide to change engines or mufflers at the midpoint of the design process.  One reason for this could be the result of our testing.  If we notice that, at maximum engine RPM the muffler is at 100% capacity we may decide we need a larger muffler if the engine has a possibility of over-revving.

In this case, we need to back up a few steps and update our documentation.  If we move from Muffler Model 100 to Model 200, we need to update our specifications that we use in our computer automated test.  We have to take into account the different capacity of the new muffler - we clearly want to ensure the model of muffler we are committed to using has valid statistics.

We also will need to update our physical test as well, and this may be a little easier to understand.  If the muffler has a different size, we may need to move our sensors, the mounting bracket for the sensors, and so on.  We may need to change our tolerance levels as well.   If we had set a warning that the engine was exceeding the capacity of the old muffler, we will need to change that setting to reflect the capacity of the new muffler, for instance.

At this point we now have updated specifications, updated tests and can quickly validate how well the muffler + engine work together.  Had we missed either step in updating the changing requirements, we run the serious risk of not being able to tell that the engine + muffler would not work. 

I'll cover the costs of finding these errors when I sum up next week.

Questions, comments, concerns and criticisms always welcome,

Monday, January 29, 2018

Integration Testing, part 5 More than one test may be needed

In my last post I mentioned a test that we can run on the specifications for the engine and muffler.  This is notionally a check we write a script for the computer to run and validate that the components will work with each other.

That is probably not enough, though.  Just checking the data sheets is no guarantee the actual muffler and engine will work together.  Imagine a scenario in which the redline (the top speed) of the engine is given.  Then imagine that, for whatever reason, the engine we are using exceeds that speed.  In this case, the engine will start to output more exhaust than the ratings sheet indicates and our muffler may not be able to handle that much extra, unexpected exhaust.

One possibility for this is an emergency.  Suppose the driver needs to get to a hospital and doesn't care about the damage the car may take.  In this case, we need to verify the behavior of the engine + muffler even when it goes out of specification.

The testing here is mechanical in nature.  We create a fake engine of some sort that outputs more than we expect the real engine to produce and test with the muffler.  At this point, we document the behavior of the muffler.  Some reasonable expectations are:
  1. The muffler fails immediately and simply does not process the exhaust.  Instead, it simply passes through with no catalytic converter and no sound reduction.
  2. The muffler fails more catastrophically.  It could break, overheat or worse, even explode.
  3. There is also a case that the muffler designers built a safety margin into their specification and did not document it.  In this case, the muffler may work, perhaps only for a short duration.

We don't know what we should do if the muffler + engine do not work together in this scenario.  At this point, the testing organization is in exploratory testing mode and simply needs to determine the behavior.  Once we have a clear understanding of what is likely to occur we can apply that knowledge to making a decision. 

I'll cover that next.

Questions, comments, concerns and criticisms always welcome,