The (Some) Fallacies of Test First Development
Written on November 5, 2009 – 7:06 pm | by admin
I should first clarify that I would never come out against the idea of writing tests for your code. I find testing to be a wonderful way to aid refactoring, fight regression, and build developmental confidence in a piece of software. Lately I have tried to provide significant test coverage for projects I work on. My problem is with buzz word fanatics, suits, zealous developers, and tangentially related technical staff who question myself or my team in our decision to not require test first methods as part of our normal development procedure.
Fallacy #1: Test first development is proper unit testing.
Unit testing serves two primary purposes. It allows you to have a fast and efficient way to verify that code you have written is working properly and it provides a mechanism to guard against regression. Bugs that are found are easier to keep fixed and business requirements are easier to maintain as a project ages and grows. There is nothing in either of these purposes that dictate when a test should be written. If anything writing tests after you have your implementation in place fits just as well if not better with these two goals.
Fallacy #2: Test first development will improve your work flow and make it easier for you to focus on the task at hand, providing you with a more efficient and effective coding session.
I tend to write my code more like Bob Ross will paint a portrait. I know what I want to end up with and I have a palette of the tools I need to get there. I would love to say that I always have a proof of concept and that write-one-to-throw-away is the only way to go but the reality of deadlines and business requirements means that this is not always possible. When I sit down to code I am not sure that method 'a' is going to return 'x' and method 'b' is going to return 'y'. All I know is that I have a large module that needs to do something and I have an idea how to get it there. In order to get there I need to put some code on the page and work with it and modern languages afford me that luxury. We are long past the era of punch cards and 2 am university time shares with one chance to get your code correct. Writing unit tests at this point is potentially a waste of time because I have not yet decided how all of the units fit together. You can argue that I should know these items or that I should use my tests to figure it out before I start coding but I write software because I enjoy it and the act of development is a creative and evolving process. Testing first has shown itself (to me) to be detrimental to this part of the process. For me it causes hyper focus on trivial components and interrupts my thought process when an implementation change causes major test refactoring. Test first also causes me to write tests for features that do not make it in to the final implementation because I thought I might need them when I started. I can see where forcing test first may help inexperienced or junior level developers but I do not think it should be considered a process requirement. In the end a good, experienced coder should decide what works best for them. In this case I choose general regression focused testing over test first development.
Fallacy #3: You should always shoot for 100% code coverage and this is a good reason to test first because you won't write any untested code.
We should be smarter than this. A good developer should be able to decide which areas of the code are likely to be fragile and what should be heavily tested. In fact, by definition, 100% coverage does not actually mean what it seems to imply. 100% coverage only means that a test suite encounters every line of code in your application. It should be obvious that this is a pretty useless metric. If an editor reads every line of a novel does that mean that the editor made sure all of the i's were dotted? So what we really want is 100% functionality coverage and this means that you are at the mercy of how far ahead of the process you can think, how many edge cases you can imagine and what bugs you have seen in the past. Unless you are willing to spend a considerable amount of time dreaming up things to test you will not likely be anywhere near 100% of "useful" coverage. I'd further argue that striving for 100% coverage (by its definition) is a waste of time. I find that the 100% coverage mentality tends to lead to insignificant and even useless tests so using it as an argument for test first development is tenuous at best.
Fallacy #4: If you are having a hard time writing tests first than you are not doing it right.
I am not going to even justify this overtly arrogant statement with an extended argument. The fact of the matter is that the "right" way to do test first may not work with a person's development flow, style, or approach to coding. If their personal approach to writing code provides solid and relatively error free code with a useful test suite than who are you to tell them they are doing it wrong?
The bottom line is that "when the tests get written" is a matter of personal preference. An intelligent and experienced coder should decide for himself what works for the given requirements. At no point should a business requirement dictate that all programmers write their tests first. Sometimes I write my tests first and sometimes I write them last, I do not suffer from delusions of grandeur but I am a good programmer and the answer to "when do I write my tests?" does not change that. So write your tests first, second, third or parallel if that works for you and stop giving me the gas face every time I state that I do not always do things your way.
10 Responses to “The (Some) Fallacies of Test First Development”
By wayneeseguin on Nov 6, 2009 | Reply
Great points Bill. I agree.
By asd on Nov 8, 2009 | Reply
"Unit testing serves two primary purposes."
Three, actually. It can drive good API design, which is also the primary reason for writing tests first.
By Mazen Harake on Nov 8, 2009 | Reply
ASD has a point but I still agree that Tests-First approach can suck time like a black hole. This is why you use psuedo-tests (or BDD). Funny enough I just wrote a blog post inline with yours. Check it out on my website link. Good arguments and points, more structured then mine are
By tjstankus on Nov 8, 2009 | Reply
I actually think regression and verification are secondary purposes of test-driving code, a nice side-effect if you will. The primary purpose is producing more modular, understandable code. It's actually pretty tough to write a big pile of junk when you're being vigilant about test-driving. But I agree with you that TDD/BDD often adds unnecessary overhead when you're writing a prototype or proof-of-concept.
TDD/BDD is a skill that takes work, like any other. It's not like you start doing it and suddenly it's all rainbows and puppies. It's painful and often slower, especially at first. All that said, I'm anti-dogma of any kind. Do what works for you. But know that TDD/BDD is not a magic light switch. It's gonna take time and work to get into the groove of it and become test-infected.
By noob on Nov 8, 2009 | Reply
@asd
No. Experience and competency in design skill drive good API design, unless you are suggesting that TDD replaces design skills.
By james on Nov 8, 2009 | Reply
> TDD/BDD is a skill that takes work, like any other. It's not like you start doing it and suddenly it's all rainbows and puppies.
Why don't you spend that time on honing your design, algorithm, and problem solving skills? How is TDD/BDD going to help you when you have too many data to store in memory? How is TDD/BDD gonna help you when your ranking algorithm is taking 20 years to run?
The last 30 years of software engineering people have been building databases, kernels, web-browsers, search engines, word processors fine without the TDD/BDD. What large and complex software system have been done using TDD lately in your lifetime?
By Paul Keeble on Nov 8, 2009 | Reply
The very fact your writing unit tests at all is a good sign your on your way to being test infected. But the whole point of Unit testing is not to achieve coverage of the code or to functionally test it, that is a side effect of the approach. Its a useful side effect as it contributes to maintenance but doing activities that may/should save you time in the future is a recipe for not doing those things.
TDD on the other hand is a design process. By focussing on external factors for a classes/modules we can assign clear responsibilities. Writing the test first allows a free form way to produce a clean API with clear and obvious responsibilities for methods, and to signify the relationships between them. Its quick to prototype the look of various possible approaches and see how it looks in practice and any edge cases that look bad. In the process you produce some executing tests and can use them to confirm the behaviour. You don't get complete functional or line coverage, but what you do get is a more obvious design. How far you go with that initial step of tests to determine the functional needs of the system is up to experience but its always useful to do it.
TDD is about design, not about tests. Its an approach to producing an executable design. Everything else is a side effect with benefits.
By Chris on Nov 8, 2009 | Reply
If you write tests to verify that your code works properly, and to guard against regression, then perhaps you would benefit by not writing unit tests at all? Regression tests / functional tests do a much better job of demonstrating that your code actually does what it is supposed to do. In an integrated, end to end way, your unit tests give very little guarantees or confidence. If unit tests are not driving your design, because you don't write code that way, then why create such fine grained tests at all?
I am one of those zealots who does the whole BDD thing, because I find that when I write code that way, it tends to evolve from understanding what the interface should be before thinking about how the code should solve the problem. I tend to work on applications that grow and change regularly in production, and have found that when I write code using TDD, that I find it easier to keep the code clean and easy to maintain. If your mileage varies because you have superior design skills, that's okay. Though it may make it harder for others working on the same code base who don't share the same talent.
Even with TDD, I find that I still need to add separate acceptance tests, often after I've written the code, because unit tests don't prove that your application works. That's not what they are for. You can have a high percentage of code coverage, with all your unit tests passing, but still have an application that won't even start.
By Christian on Nov 12, 2009 | Reply
Most of this sounds like a strawman to me. I suppose if someone is perpetuating these fallacies, they really don’t know what they are talking about.
As mentioned by a few previous commenters, TDD is not about Unit Testing, or code coverage at all. It’s purely a design process, that makes it much easier to create a loosely coupled, highly cohesive design. Do you think TDD practioners never write unit tests afterwards?
I am really not sure what you hope to accomplish with a post like this. Are you purposefully misrepresenting TDD/BDD to be contrarian? If you think your representation of TDD is fair, what design process do you suggest developers follow to accomplish, high cohesion, and low coupling?
By admin on Nov 13, 2009 | Reply
The post was a originally a response to several recent situations where I had to deal with people who are so dogmatic about their way of doing things that its almost useless to discuss it with them. In this case I understand testing well and I am not here to debate my abilities or understanding of TDD. I never said that TDD is ineffective when applied correctly and if you read the article I never personally said TDD had anything to do with unit testing or code coverage, I said that people have used those items as arguments in favor of TDD. The fact of the matter is that it doesn’t work for everybody or every team. I write my tests first often, sometimes it works for me. But I refuse to require it of my team or of myself as it is not always the right way to approach a problem.
As far as how I hope to accomplish low coupling and high cohesion in code design and development. I find that good, experienced developers (of whom there may be shortage of in this industry) do not have a problem achieving these items without TDD and that inexperienced developers struggle with it whether they are are testing first or not. There is plenty of good code that was written before everyone jumped on the TDD bandwagon.
Testing first in and of itself does nothing to ensure that your code is decoupled, no matter what it is still all about how well you can model the process mentally. Some are better than others at that process and TDD will help many people but that doesn’t mean it helps everybody.
Your point here seems to be in the same vein as quetions like “How did anyone ever keep in touch before email?” . Sure it was harder to maintain communication for some people but others managed to do it just fine.
Thank you for taking the time to leave a comment here. I have had an overwhelmingly positive response to this post from others and two thirds of my reddit votes were positive. In this case I’m quite sure the naysayers are in the minority but this does not mean that I give less value to a contrary opinion.
Debating testing methodologies seems akin to debating religion these days and there really isn’t a tangible best answer. All that matters is what works for you and your team. I had some things to say about testing and a colleague who agreed with my points thought I should put it on Reddit, it was my first submission there and I had no idea I’d get so many responses. I have learned a lot from those responses and I guess that was the real point of the post.