On Saturday, I attended the Continuous Integration and Testing Conference (CITCON) here in Chicago (The home of Web 2.0!).

CITCON was planned using the OpenSpaces format, so it was 99% discussion based, with very few canned presentations, and even those were just introductions to a larger discussion. CITCON was very well run and I hope to attend it next year when it comes back to Chicago. There are rumors that the next CITCON will take place in London in a few months (late Summer/early Fall). (Heads up, Sam!)

I suggested and lead several session topics (more on that later). The session called “The Language of Testing” was my favorite. (Derek Wade from 3back.com was kind enough to jump in and “co-lead” the session with me.) In the Language of Testing, we spent an hour getting down on paper all the different types of testing that people actually use in their day-to-day software work. We got many terms down on paper, but we only got a chance to actually flesh out and define 3 types of testing. (I’ll post more on that later.)

On the CITCON mailing list after the conference, Kumar McMillan made this comment:

In one of the 3 o’clock seminars, “Language of Testing”, we ran out of time but were about to define the term “Testing Tool.” The rules were simply not to use any words from the term itself. Here is my stab at it: “Something you can use to prove the functionality of software.” … but everyone knows that, right ?

Python fans should also read the first part of his post, especially this:

It’s somewhat of a coincidence but over the past couple days I have been putting the final touches on a python module called “testtools,” getting it ready for open source. And … here it is : http://testtools.python-hosting.com/

Being both a Pythonista and a connoisseur of testing tools, I’ll definitely be checking out his new module. But in this post, I’m going to explore his definition of “Testing Tool”. Here goes:

I had a major “AHA! Eureka!” moment after this conference and especially because of the “Language of Testing” session.

Software Testing is a one-part “Social Science“, and one-part “Hard Science“.

Kumar’s definition of a testing tool is “[S]omething you can use to prove the functionality of software”. I think that is missing a few distinctions.

In electronic circuit design, there are testing tools like PSpice, that you can use to mathematically prove that your design is correct. By “correct”, I mean that your circuit doesn’t blow up in smoke because of a short-circuit or incorrectly wired component.

In software testing, since at one level it is just software, and software is just a bunch of NAND gates dynamically put together on a massive scale– yes there is a “mathematical certainty” aspect of proving its functionality. Tools can be used for things like load testing or syntax validation that can give you some mathematical certainty of how your app will work in the real world.

But the other “half” of the problem is the customer or user of your software, which may or may not be a living, breathing human being. (The notion of “web services” generally assumes that a machine, not a human being is the true “user” of the software.) If a human being makes the final decision on whether some bit of software is functioning correctly or not, then it is very, very hard to have mathematical certainty that your software is functioning properly from their perspective. To get this kind of mathematical certainty, you need to jump into the realm of the social sciences: Psychology, Social Psychology, Economics, Marketing, etc… In the world of the social sciences, things are rarely ‘black and white’. Instead, there are Bell Curves, Fuzzy Math, Heuristics, Standard Deviations, and Confidence Intervals, lots of graphs and lots of theories… “Hard Scientists” sometimes look down upon the practitioners of “Social Sciences” for their fuzzy theories and conclusions and their lack of 100% repeatable and testable solutions. But when it comes to explaining and predicting the behavior of the human mind, “<fuzzynumber> +- <asmallerfuzzynumber> %” is the best you can hope for.

In short, software testing is both “precise” and “fuzzy”. Ignore either at your peril.

To bring this back to Kumar’s original definition of “Testing Tool”… It depends– if you’re testing the “digital” aspect of software, yes, you can “prove” that it works… But in the “analog” (aka human) aspect, you can’t “prove” that it works. You can only make “very educated guesses”.