A nice blog post from Andy Kriebel provides a nice break from TC preparations

As you can tell from my dearth of recent posts, I have been incredibly busy with getting prepared for Tableau Conference.  Hard to believe it is coming up next week!

While hammering out the details of the Hands On Session I will be co-presenting, I saw that Andy Kriebel has just uploaded his 1000th blog post over at vizwiz.com.

Andy, who I have not met, is one of our Zen masters and has been a tremendous advocate for Tableau for years.  This post is a sincere homage to Tableau Public (free!) and I look forward to the next 1000 posts Andy writes.

Back to TC prep.

Questions, comments, concerns and criticisms always welcome,
John

Matlab + Tableau - making math much easier!

A quick note that Tableau can be set to use Matlab as an external server - like R or Python.

There is an article from Mathworks here: https://www.mathworks.com/products/reference-architectures/tableau.html

Since my team also owns this feature I figured I would point it out.  I've always liked Matlab - it is very easy to look at some equations and then "just type them into Matlab."  Thanks for helping us out along the way, Mathworks!

Questions, comments, concerns and criticisms always welcome,
John

TC is getting closer

I mentioned I would be co-hosting a Hands On Training session at Tableau Conference this year.  That has been keeping me very busy this past month. 

The session I am co - presenting is Intermediate Statistics.  It features many of the areas my team here owns.  To that end, I'm gathering data sources to provide some meaningful training on Trend Lines, Forecasting, Clustering and correlation.  <Insert "correlation is not causation" warning here>.

Surprisingly, there are not many ready made data sets that focus on intermediate level analysis.  Most are either introductory and too basic for this type of hands on training, or are very large datasets that can be overwhelming.

I don't want to tip my hand here yet since I am still working on getting a few data sets nailed down, but I think I have the general idea in place.  It's simply a time consuming process to get all the details hammered out in advance.

Then on to practicing my speaking!

Questions, comments, concerns and criticisms always welcome,
John

Getting caught up here and in my day job!

I just noticed that I haven't met my self imposed goal on updating this blog every week.  I've been locked down with so much day to day work this simply fell off the radar.  Instead of any insight (that's coming up :)  ) I'll share what I have been doing.

1. Filling holes in test coverage.  I had time to look at our code coverage numbers and found some holes in our coverage.  I have added some tests to validate the code continues to do exactly what it was designed to do.
2. Identifying holes in coverage.  As we get ready to send features out to our customers, we review the test plan we have in place to ensure we did not let any test cases slip through the cracks.  We find some very small gaps in our coverage when we did this exercise last time, so I added some tasks to our to do list and have been helping to get them done.
3. Had some good internal training on hiring practices and other activities like that.
4. Many (many!) discussions with partner teams about adding cross team features in the future.
5. Preparing for Tableau conference (I wrote about this last time).
6. And then simply doing my "regular" job has been very time consuming recently.

Looking ahead, I have been thinking about this situation.  If you give an artist a rake as the only implement to use to make some art, you had better expect to see some leaves.  I'll dive into that comment next up.

Questions, comments, concerns and criticisms always welcome,

John

I'll be speaking at Tableau Conference this year

Just got my confirmation that I will be co-speaking a Hands On Training for Basic Statistics in Tableau.  This should be a great experience - our team owns all the code that drives the features we will be teaching.  And since I am a tester, if there are any glitches, I guess I can only look in the mirror for who to contact about them (heh).

More about Tableau Conference here: https://tc18.tableau.com/

Now I need to go prepare the topics which we will present!

Questions, comments, concerns and criticisms always welcome,

John

The Guin Constant

I was reading James Apnes Notes on Discrete Mathematics  this last week and came across this statement in section 2.1.5:

"The natural numbers N. These are defined using the Peano axioms,
and if all you want to do is count, add, and multiply, you don’t need
much else. (If you want to subtract, things get messy.)"

Since the book is aimed at Computer Science majors at Yale, I was bothered by this statement right off the bat.  Computers are pretty good at some aspects of mathematics, but not all.

In this case, I can't use a computer to count.  Eventually, my computer will run out of memory. When I try to add one more to the current number, my computer will crash.  I know what you are thinking: "I'll just add more memory!"  (It's turtles all the way down).  That won't work either since memory is made out of matter, and the universe in which we live has some finite amount of matter.

So suppose we used each bit of matter (atoms, leptons, whatever smaller building block we discover next) 100% efficiently to store the biggest number we possibly can store in this universe.

I call that the Guin Constant.   It is the biggest number this universe can hold precisely using all matter to store it.

And please don't add one to it.  I'd hate to crash this universe.

Questions, comments, concerns and criticisms always welcome,
John

PS: apologies to mathematicians if this constant already exists.  It's hard to find information like this with internet searches and/or the math books within arms reach of me right now.

Moving day for tests

One of the tasks on my plate is breaking up our old (old!) test suite.  We have a set of tests that live in this suite and without getting into the gory details, they need to be moved out of that project.  In short, that project is old and needs everything to be built to be used for testing.  Since unit tests are designed NOT to need everything built - just the bit of code I am trying to test - this is not a good paradigm to follow long term.

So we have started to break up these old tests and move them to the best location (module) into which they belong.  It's going well so far  but as some tests are really old they become troublesome and time consuming to move.

In the process, I have learned more about Cmake than I ever wanted to know!  As you can imagine, if a test is moved from its original location and into a new location, the file that directed it to be built in the old location needs to have that entry removed.  And the file that instructs the compiler for the new location needs to be updated to include the new file as well.

So in the best case, a single move has to update three files.  I haven't hit that best case yet - there are always more updates  needed - but I am getting better at it each day.  If you are interested, let me know and I can post more details.

Questions, comments, concerns and criticisms always welcome,

John

This week's extremely annoying and catastrophic bug

Somehow - and I have no idea how it happened - my Windows PATH environment variable got completely cleared.

The PATH variable is a list of folders on the hard drive that are accessed, in order, when I type a command at a command line (among other things).  So if I have a command window open and type "python" to start python, Windows will look for a program named python.exe (or .bat, or .cmd) in my current location.  If it finds one, it starts it.  If it does not find python there, it looks in the first folder of that PATH variable.  If Windows finds a python program there, it runs it.  If not, it looks in the next folder in that list, and so on.

I have probably a dozen commercial tools like python installed that use this.  I also have a couple of dozen batch and command files I have written to do repetitive tasks that I put in c:\windows\system32 (since that is always in the PATH) so they are always accessible.

But since that list got wiped, this all broke.  And the only way I have to restore it is to find broken applications, one at a time, that have dependencies on that PATH variable to work right.  When they fail, I have to either repair the application, reinstall it or manually figure out what went wrong and fix it myself.

Frankly, this has taken a tremendous amount of my time this week and I honestly think I might have been better just reinstalling everything on my C: drive from the ground up.  I keep all my data on a different drive so I wouldn't be that far behind the curve when done.  But at this point I have most of my key applications fixed so I think I will stay on my current path.

But this really messed up my week.  Sigh.

Questions, comments, concerns and criticisms always welcome,
John

Workaround for the Tabpy bug

I've spent the last couple of weeks heads down on the Tabpy bug reported over on our site: https://github.com/tableau/TabPy/issues/106

Essentially, we broke some customers when we rolled out 2018.1 servers.

It took awhile to narrow down the defect.  After a lot of hard debugging, we isolated the cause to servers that had never set a timeout value. The timeout controls how long Tableau will wait for a server response.  Anyway, if that value is not set at all then the server will default to a 0 time.  This causes it to wait no time at all for a response, and then give up when the server does not respond in zero time.

Embarrassing and annoying for sure.  But as we mention on the site, there is a workaround.  Run the tabadmin tool to set a value (directions are in the thread above).

Now that we know the cause of the error, we can start working on a fix.  Stay tuned for that.

Since this has taken almost all my time for the  past two weeks, I wanted to share an update on where we are.

Questions, comments, concerns and criticisms always welcome,
John

Tableau is aquiring Empirical Systems

Almost missed this big news today.  Tableau is purchasing/acquiring Empirical Systems:

https://techcrunch.com/2018/06/13/tableau-gets-ai-shot-in-the-arm-with-empirical-systems-acquisition/

Again, this is good news.  Check it out and feel encouraged to let me know what you think.

Questions, comments, concerns and criticisms always welcome,

John

Stepping back to look at a partner team here at Tableau

One of the teams at Tableau is called the Extensions team internally.  They are releasing a new feature to connect data to Tableau and are in the 2018.2 Beta 2 release of Tableau.

I listened to their online training on this and they admitted to a bug they had with their original beta release.

The original manifest XML we posted on github for our fiirst beta had a typo in it. 

Check out this one line:

<manifest xmlns="http://wwww.tableau.com/xml/extension_manifest" manifest-version="0.1">

Do you see the typo?  There are 4 ws in the url.

It should have been

<manifest xmlns="http://www.tableau.com/xml/extension_manifest" manifest-version="0.1">

When troubleshooting, this was pretty easy to find (but embarrassing).  It is also easy to fix - just delete that extra w.  And anyone writing code to use extensions could presumably make that fix as well for the beta releases.  Clearly, we are fixing this for final release.

I appreciate this bug, though, since it is a great example of the need to be meticulous with checking text.  That file is not that large, but reading XML can be tedious and typos can creep in.  It is also an example of NOT changing strings right before you release a piece of software. 

Anyway, I thought I would share this bug and fix.  Extensions are pretty terrific - check them out if you have a preview account at https://prerelease.tableau.com/

Questions, comments, concerns and criticisms always welcome,

John

Another busy week in Tableau Test

As usual, there is plenty of activity on the test team this week here at Tableau.

A random sample of activities, all of which are typical:

1. Test pass week!  We ship every quarter (from the test point of view, this happens MUCH more frequently -I wrote about this 2 weeks ago)
2. Finding and filling holes in test coverage.  With a product as agile as Tableau, the current state is that we have some source code files without adequate automated test coverage.  One of the test teams' tasks is to find these files and add tests to validate the functionality of that file.  This is a fairly straightforward task to explain, but if (and when!) the code is complex, adding tests can be a time consuming event.
3. One of our testers is adding a rolling log handler to tabpy.  Hop over to github to see her proposed changes.
4. Oddly, we are updating our Windows machines to the latest version.  That takes a half day - I wish I would have known that BEFORE I clicked the button Monday morning.  I would have waited until the end of the day.
5. Many server setups and configurations centered around R and Python external servers.  We're getting ready to test the secure capability we are adding to those services.  Again, see the tabpy site for details.
6. Automation analysis results, find/file/regress bugs, etc…  The standard test workload is always ongoing.

So we are set for another full week of activity here.  Nothing too special about this week so I am just sharing a "day in the life" type of update.

Questions, comments, concerns and criticisms always welcome,

John

Distracted by some fun math

For many reasons, I have spent a greater amount of time this week simply building code over (and over) again.  While the code is building, I use that time to get caught up on email, online conversations, documentation and so on.

This week I was catching up on RSS feeds and stumbled on this site: http://somethingorotherwhatever.com/sum-of-3-palindromes/  I have spent just enough time there to convince myself the proof is true (heh) and to be impressed by the fellow - Christian Lawson-Perfect -  that built the site.  It was hard to tear myself away, but testing calls…

I never knew that every number greater than one could be expressed as the sum of three palindromes.  A high level view is here.  And thanks to Evelyn Jamb for pointing me to this

On that final note, I subscribed to Evelyn's email against my better judgment.  Normally, email newsletters get bloated, multiply, and generally go off topic or get to the point that I otherwise unsubscribe.  Evelyn's tiny letter has not gone off course - I am glad I subscribed.  If you want to give it a shot, please do!

Questions, comments, concerns and criticisms always welcome,

John

I had not heard the term "rolling log" before but now Tabpy has one

From the "you learn something new every day" category, I saw that we had a proposed enhancement to Tabpy to support a rolling log.  I had not heard that term before but the concept is fairly simple.

Suppose your application creates a log whenever it gets started.  That log is named "My Application Log.txt" and is saved when the application exits.  Now you have a dilemma.  Suppose I start the application, run it for a bit then exit.  I have my log file.  Then I start it again.

What do I do with the previous log file?

Some obvious options:

1. I could have the application delete it and start a new one.
2. I could just keep adding to the existing log
3. I could create a new log with a different name

Each one of these has pros and cons.  Idea 1 is simple - easy to understand, but you lose the history of what had happened last time the application ran.  That information is often useful if you need to troubleshoot

2 and 3 both keep all that history, but the hard drive will eventually get full.

Option 4 is the Rolling Log.  It is like 3, but deletes all except the last few logs.  I can set it to keep the 5 most recent, 2 most recent, 10 most recent, etc…  Then I retain some amount of history but avoid filling up the hard drive over time.

Check out the proposed change we want to take for Tabpy over at https://github.com/tableau/TabPy/.  The pull request is the current request as I write this but that will change over time.

Questions, comments, concerns and criticisms always welcome,

John

How often we ship Tableau, a test perspective

If you look at Tableau closely, it's obvious - and we make the claim - we ship a new version of Tableau every quarter.  But from the test point of view, we ship much more often.  Here's what is looks like from our point of view.

Imagine you own a textbook publishing company. Each quarter, you release a new textbook covering some new topic you have not covered before.  An example might be Astronomy: The Mountains of Mars from Winter 2017 and  Egyptian History out in Spring 2018.

At the same time you are ready to ship the Egyptian History book, though, the Mars explorer finds enough data to cause you to need to update one of the chapters of the Mars Mountain book.  So for Spring 2018, you have 2 books you need to send out the door: the new Egypt book and an updated version of the Mars book. 

Your proofreaders will need to focus most of their time on the new book but still must devote some amount of time to validating the text and layout of the updated chapter of the Mars book.  Additionally, the new chapter might change the page count of the Mars book.  If so, you might need to bind the book differently.  If there are new photos, you may want to update the cover or back of the book.  The table of contents might change, and the index will likely need to be updated. 

A test case for the index might be to validate the previous contents are intact after the new chapter index is inserted.  If the size of the index no longer fits on the current set of pages, you will need to rebind the book, shrink the index or otherwise resolve this dilemma.  And the proofreaders, who might have naively thought they needed to verify only the new chapter contents, potentially have to validate the entire Mars book.

Testing is in the same position.  While our focus is typically on the new versions of Tableau we release every quarter, we also continue to support the last few years' worth of releases. That means in addition to testing the "major" release of Tableau, we have to test the updates we consistently ship as well.  So from my point of view, always have multiple releases we have to validate.  And that means that we ship far more often than once per quarter.

Questions, comments, concerns and criticisms always welcome,

John

Constantly learning

One of the requirements that comes along with working in the tech industry (or any industry, really) is to adopt a notion of constant learning.  A new computer science major today will know more than those that graduated ten years ago, and will know less than students hired ten years from now.

In order to stay current with the industry, I have found several opportunities to keep learning.  One of my favorites is online classes (MOOCs, or massive open online courses).  The University of Washington has a Data Science program certificate class starting soon and it looks like there are enough of us around here to develop a study group.  For many folks, having that level of interaction is a necessity for getting the most out of the class.  The environment it creates -  a useful forum for discussing the techniques being taught - really helps cement the  lesson at hand.

I'm not sure what the emphasis of this class will be, though.  I hope it is more along the lines of implementing a few ML routines as opposed to using "off the shelf" solutions (which are never 100% complete - you always need to write code at some point) but it is definitely on my radar.  Let me know if you sign up for any classes in this series (the audit track is free) and maybe we can "attend" together!

Questions, comments, concerns and criticisms always welcome,

John

Lots of Tabpy activity

Looking back over the last few weeks I have had a ton of meetings.  One of the products my team owns is Tabpy and I have spent a good amount of time over there.

Specifically, I have been performing some code reviews coming in from the community (thanks gang!) and even had a call with one of the developers.  I also checked in some documentation changes over there to update some error messages to help make setup errors a bit more clear.

Also on the Tabpy front, we have had some team wide customer calls about how companies are using Python integration and how we can help them meet their goals.  Behind the scenes, we are taking notes, designing stories and entering a set of items to track this work in our backlog.  And yes, we are working on these items already, but I (obviously) can't share specifics.  That is simply a frustrating aspect of blogging about testing.

Questions, comments, concerns and criticisms always welcome,

John

Filling holes in test coverage

One of the tasks in my plate is to find and fill holes in test coverage.  A good question - which I am going to avoid for now - is how features ship with holes in coverage.  Instead, I want to focus on holes that get created over time.

For instance, my team owns the Trend Lines feature in Tableau.  Here is a sample image we have used to test for years:

You can see the multiple trend lines drawn, each with a confidence band around it.  We validate that we do not alter this expected visualization with each new version of Tableau using a simple image comparison.  We also validate the math behind this feature but I want to focus on the image for now.  This test has been running and passing for years.

So how can debt accumulate with this scenario?  One way it cropped up was adding a version of Tableau that works on the Mac.  Initially, when this test was created, only a Windows version of Tableau existed.  Since that point, we have added Tableau for the Mac, Linux (via server) and different flavors of server (Windows and Linux).  So now we have to verify we draw correctly on each new platform.

The hole gets created when some new supported configuration is added.  When we added Mac support, we obviously ran all our tests.  In this case, the Windows test did not break since we changed nothing for Windows.  And because we did not add a test for the Mac, the non-existent test did not fail (again, obviously).  But we never added that test so a hole existed.  We are filling that gap now, as well as Linux tests.

Questions, comments, concerns and criticisms always welcome,

John

Updating the tasks I posted from 2 weeks ago

Two weeks ago I posted a laundry list of to do items.  We are an agile team with a  2 week sprint cycle, so it seems natural to follow up on these.

Again, in no particular order:

1. We are getting some beta feedback on our features and are working on fixes for those items.  --- I contacted the person that submitted a key piece of feedback and showed him a proposed change.  He thought it was much better, so remain tuned.

2. Some of our unit tests use integration points to set up before the test runs. 
1. This is ongoing work.  In the past 2 weeks, I have refactored 4 tests.
2. I also had to update a new test we added because the path to one of the files "is too long."  Sigh.

3. Spent 2 hours performing backlog cleanup.  Consolidating related items, assigning tasks to stories, etc…

4. I actually tested Tableau :)

5. I also had time to jump over to Tabpy on github and perform a code review and merge there.

FWIW, 2 and 4 will be focus areas for the next sprint.  Let me know if you find these updates interesting.

Questions, comments, concerns and criticisms always welcome,

John

Top 3 Outlook tips to help with day to day life

Outlook has SO much functionality built into it that it can get overwhelming at times - meetings come in, reminders fire, emai pops up, etc.. etc… etc..

But there are some actions you can take to make your life much easier.  I share these with each new team I go to and want to put them here as well.  For Outlook "old-timers" (to which I belong since I was there for Outlook 97) these may seem obvious.  If you have never heard of these before, these may really help.

1. Create  a "cooling off" rule.  I have a rule set to delay all mail I send by 2 minutes.  This helps me a few times per year when I notice a typo at the last second, see that someone else replied while I was pressing send, or otherwise make some goof I need to correct.  Here's how



1. Open the Home tab | Rules | Manage Rules and Alerts…
2. Select Apply rule on messages I send | Next
3. 
   
4. Conditions is the next tab.  I leave this as is since I want this to apply to all my mail, so I click Next.  I get a popup that says this will apply to every message I send, and since that is exactly what I want, I click Yes.
5. Now I get the action I want Outlook to take.  I select Defer delivery by a number of minutes and change the link to 2:
6. 
7. Now all mail will wait in the Outbox for 2 minutes after I press send.
8. Exceptions can be useful.  You may want the rule to be Delay all mail by 2 minutes unless is it High Importance, so you can set exceptions like that here.
9. Click Finish and the rule is saved.
2. Second tip is helpful when an email thread is spiraling out of control.  Right click it and select "Ignore".  You will get an alert reminding you that this item and all replies, etc… will be moved to Deleted Items.  Nifty.
3.  Finally, I work remotely now and then and it is helpful to force Outlook to sync.  Yes, it syncs automatically, but if I know I am about to lose my wifi connection, I can force it to sync by hitting F9.  This is more useful in soothing my conscience, but it is a habit that mentally helps me so I keep doing it.

1 and 2 are the biggies.  They don't help every single day, but when they do help, they make a HUGE difference.

Enjoy!
 
Questions, comments, concerns and criticisms always welcome,
John

Back to work

Last week I had a trip to Palo Alto and this week I had a few days of internal training (which was quite good).  Now it is back to work.

And by work, this week I wanted to simply post a "laundry list" of tasks on my plate.

So, in no particular order:

1. We are getting some beta feedback on our features and are working on fixes for those items.  Stay tuned for more details on this as we get the work completed.
2. Some of our unit tests use integration points to set up before the test runs.  Refactor those to be closer to unit tests.  This gives a few benefits:
1. Slightly faster build times.  This work generally results in less code which generally builds faster.
2. Easier to know what happened when a test fails.  If a test has to complete three setup items before running the test (item four), then a failure reported could be one of the 3 setup items or a failure of the code you wanted to test.  In fact, the odds are 3/4 that your code is working fine, but something else in setup failed.  Less tasks to do make it easier to identify the source of a reported failure.
3. Our team had sprint planning this week.  I think I mentioned we do 2 week sprints.
4. We scrubbed our backlog to keep it current and reprioritized items as needed (see item #1 above for an example of what might cause planned work to be re-prioritized).
5. Two days of training (Changing your Viewpoint types of things)
6. I actually tested Tableau :)

Questions, comments, concerns and criticisms always welcome,

John

A quick jaunt to Palo Alto

The bigger organization within Tableau of which I am a part has a rather large team at our office in Palo Alto.  We had an All Hands meeting on Monday this week and I was able to attend in person.  I always like visiting Silicon Valley and decided I would ramble a bit about why this week.

1. The number of tech companies there is astounding.  Everywhere I look I see Microsoft, Amazon, Apple, iAm, Sophos, Salesforce and so on.  It just brings out the geek in me.
2. The shuttle we used was a Tesla.  Repeat: the shuttle was a Tesla.
3. The weather there is typically nicer than Seattle.  One exception: I was in San Francisco once and went from hail in the morning to 95 degrees in the afternoon.  Cue the Mark Twain quote.
4. It's always nice to step away from my desk now and then to get some new thoughts going.
5. Meeting the entire team in Palo Alto lets me a face with a name.

I also talked with a few folks that are facing some of the same challenges our team is facing.  Maybe we can collaborate to face them and not waste time duplicating work - nothing to do with Palo Alto, just good business.

All in all, an enjoyable trip worth the time investment.

Questions, comments, concerns and criticisms always welcome,

John

A code coverage analogy

One of the tasks I have on my plate is code coverage tracking.  I've written about this in the past and today I want to go over an analogy on the usefulness of this statistic.

First I can point out it is relatively easy to track.  We have plenty of tools to do this for us, get reports, etc…  But just because it is easy to track does not mean it is all that useful.  I can track the contents of my glove compartment each day, but I think we can all agree that this would not be a useful piece of data to have.  There is not much I would do differently on a day to day basis based on the contents of the glove compartment.

Code coverage, on the other hand, can be useful.  It tells me how much code I have to validate that is not covered by automated tests - it tells me a risk level.  By itself, it is not enough (look up Mars ClimateExplorer as a terrific case study).  It does help make a decision, though. 

Here's my analogy:  code coverage is to a test engineer as a patient's temperature is to a doctor.  It is one useful metric that can help isolate risk.  By itself, a 98.6F temperature does not tell a doctor that a patient is healthy.  It only can help eliminate some possible causes of illness - certain stages of the flu, for instance.  No doctor could reasonably base a diagnosis on one temperature measurement.  It is one statistic among many others such as blood pressure, heart rate, etc.. that can help lead to a more accurate diagnosis.

Likewise, just because I have a 98.6% rate of code coverage, I cannot pronounce a feature risk free and ready to ship.  I also have to look at scenario pass rates, international/localization status, accessibility, performance goals and so on. 

Too often we get trapped in a mindset of "this statistics by itself is useless so let's not bother tracking it."  While the first half of that sentence may be true (and is, for code coverage), the second half of that sentence does not follow.  If given a choice of having a patient's temperature included in a checkup or not having the temperature included, doctors will always want to track that statistic.  It helps give part of a bigger picture.

And that is exactly what code coverage does.

Questions, comments, concerns and criticisms always welcome,

John

Amazing amount of mileage out of this one test file

One of the accumulations that happens to testers over time is the build up of a staggering array of test files.  Files that use "odd" characters in Japanese, a Finnish test script, an absolutely huge Excel file, etc…  I tend to keep any file that ever exposed a bug around, use it in automation and use it for ad hoc testing as time goes by.

I also have one very simple file that I use quite often.  Very basic, but very good for "smoke testing" : simply ensuring a new feature doesn't crash right out of the gate.

It's just a csv file with some simple numbers and text in it:

x	y	z	name
2	-1	4	ant
4	0	-2	bat
6	3.1	1.5	cat
-1	0.2	3	dog

Super basic, but fast to load.  It also has a mix of integers and floating point numbers, named columns, some text values and multiple rows of data.  I use it at least a half dozen times per week and I wanted to share this since it makes a good point that sometimes a simple test case is enough to provide some value.  If my test - whatever it is - fails with this file, I know I am blocked and I can file a bug and move on.

And if it passes, then I know I can start to dive further into the test matrix I have.  There's no guarantee that any of those other tests will pass.  But since I love analogies, I see it like this.

This simple test matrix is the equivalent of starting a car.  If it fails, I know I won't be testing the brakes or driving on the highway.  But just because the car starts I still can't tell if it will work when put in gear.  But at least I know the basic test - the engine works, in this analogy - is passing.

Questions, comments, concerns and criticisms always welcome,

John

Integration Testing, Part 7 The value of integration tests

I was reading this blog post this morning (Arlo works here, by the way) in which the author gives some advantages of a good test suite.  It is a great read and I want to use it as a springboard to summing up the benefits of integration testing.

In our example, we had one integration test that compared the written specification of a proposed muffler combined with a proposed engine.  If the engineers want to use a new muffler, we can immediately tell if it can handle the exhaust from the engine currently being used.  We don't need to pay to order a test muffler, pay to remove the old one, install the new one and then finally find out it doesn't work.  (In this example, we would actually damage the engine or some other physical component.  That seldom happens in the software world - just restore the previous build and keep going).  This test can save thousands of dollars and several days worth of work if the muffler + engine combination are not compatible.

The actual hardware test will also pay off.  If the specification is not right, or if the engine is operating outside of specifications, we can find that out before the car is sold.  To say the least, that would benefit customer satisfaction since the engine will work properly always.  You can imagine the downside to shipping a defective car - especially if good testing would have alerted us to the defect early enough to address it.

Thanks for following along with the notional integration testing thought experiment.  I'm going to get back to doing that right now!

Questions, comments, concerns and criticisms always welcome,

John

Tangential (get it?) post: I am digging into this precision based article on Code Project

I've been a loyal follower of the Code Project for quite some time and even written a few articles for them a while back.

It's no secret that my team here at Tableau takes precision and accuracy very seriously and I just discovered this article about that: https://www.codeproject.com/Articles/25294/Avoiding-Overflow-Underflow-and-Loss-of-Precision

Overflow and underflow are pretty well covered, but there are aspects of precision that are very hard to control.  I've been reading through this and thought I would share.  I'll finish up integration testing next time - this article was simply a very timely happenstance that I wanted to share.

Questions, comments, concerns and criticisms always welcome,
John

Integration Testing, part 6: Updating requirement based on integration testing

So now we have 2 types of integration tests giving us information on our muffler + engine system.  We are checking both the documentation (data sheets) of each component to ensure they will work together.  We also test the physical system itself to validate the real world behavior.  This way we can ensure the car will actually work after we build it.

There is one last aspect of this I want to cover.  There is a possibility that we actually decide to change engines or mufflers at the midpoint of the design process.  One reason for this could be the result of our testing.  If we notice that, at maximum engine RPM the muffler is at 100% capacity we may decide we need a larger muffler if the engine has a possibility of over-revving.

In this case, we need to back up a few steps and update our documentation.  If we move from Muffler Model 100 to Model 200, we need to update our specifications that we use in our computer automated test.  We have to take into account the different capacity of the new muffler - we clearly want to ensure the model of muffler we are committed to using has valid statistics.

We also will need to update our physical test as well, and this may be a little easier to understand.  If the muffler has a different size, we may need to move our sensors, the mounting bracket for the sensors, and so on.  We may need to change our tolerance levels as well.   If we had set a warning that the engine was exceeding the capacity of the old muffler, we will need to change that setting to reflect the capacity of the new muffler, for instance.

At this point we now have updated specifications, updated tests and can quickly validate how well the muffler + engine work together.  Had we missed either step in updating the changing requirements, we run the serious risk of not being able to tell that the engine + muffler would not work. 

I'll cover the costs of finding these errors when I sum up next week.

Questions, comments, concerns and criticisms always welcome,

John

Integration Testing, part 5 More than one test may be needed

In my last post I mentioned a test that we can run on the specifications for the engine and muffler.  This is notionally a check we write a script for the computer to run and validate that the components will work with each other.

That is probably not enough, though.  Just checking the data sheets is no guarantee the actual muffler and engine will work together.  Imagine a scenario in which the redline (the top speed) of the engine is given.  Then imagine that, for whatever reason, the engine we are using exceeds that speed.  In this case, the engine will start to output more exhaust than the ratings sheet indicates and our muffler may not be able to handle that much extra, unexpected exhaust.

One possibility for this is an emergency.  Suppose the driver needs to get to a hospital and doesn't care about the damage the car may take.  In this case, we need to verify the behavior of the engine + muffler even when it goes out of specification.

The testing here is mechanical in nature.  We create a fake engine of some sort that outputs more than we expect the real engine to produce and test with the muffler.  At this point, we document the behavior of the muffler.  Some reasonable expectations are:

1. The muffler fails immediately and simply does not process the exhaust.  Instead, it simply passes through with no catalytic converter and no sound reduction.
2. The muffler fails more catastrophically.  It could break, overheat or worse, even explode.
3. There is also a case that the muffler designers built a safety margin into their specification and did not document it.  In this case, the muffler may work, perhaps only for a short duration.

We don't know what we should do if the muffler + engine do not work together in this scenario.  At this point, the testing organization is in exploratory testing mode and simply needs to determine the behavior.  Once we have a clear understanding of what is likely to occur we can apply that knowledge to making a decision. 

I'll cover that next.

Questions, comments, concerns and criticisms always welcome,

John

Integration testing, part 4. The type of test to create

Now that I have generally covered the task at hand - ensuring that our engine and muffler will work together - I can finally start covering how to test that they work together.

When I last mentioned the testing challenge of validating the engine output can always we handled by the muffer, I mentioned 2 different test methods we could employ.

The first was an automated test, running on a computer, which would compare the data sheet of the engine to the data sheet of the muffler.  Specifically, it would look at the volume and pressure output of the  engine and validate that all these numbers were less than the maximum capacity of the muffler.  If we assume that we can examine the data sheets and that they are up to date then this is a terrific solution.  Once we alter the document, we find out instantly if the numbers will work for us.  If we were sitting around a design room, we could quickly look at multiple different engine and muffler combinations and rule out sets that obviously would not work.  No need to weld the engine into place or anything heavy like that.  This type of check can prevent errors down the road (so to speak), is relatively cheap to implement and gives very speedy feedback.

This is a solution I would push to implement to help with this problem.

Since I am a tester by nature, though, this is not the only check I would want to use.  We've all heard the old sayings of not putting all our eggs in one basket and I would also want a backup system in place to help me understand the operation of the engine and muffler system.

More on that next time!

Questions, comments, concerns and criticisms always welcome,

John