Giant Robots: A critical look at the current state of Ruby testing
Jaime Bellmyer
· 1 year ago
I certainly don’t pretend to be in the same class as a lot of gurus that have added their two cents to this post. That being said, I have yet to hear the magic word, “moderation”.
Contexts don’t make you nest 10 layers deep. You do. I’d rather have the ability to nest a little, because it’s one of the things I love about Shoulda and RSpec. Same goes for one assertion per test. Alex nailed it – tests should be atomic. Test as narrow a case as possible, but use all the assertions required to ensure that test passes.
As contractors, we have a myopic view. Of course we’re stuck learning multiple frameworks. We have to be able to jump into someone else’s code, and that means maintaining a wider skill set than those who work with the same applications long term.
So if that’s part of the argument for Test::Unit, then factor it out. Because it’s not a barrier to entry to any person, team or company looking to adopt Rails. I’d say ALWAYS start with Test::Unit, because it’s the foundation before you decide you need something fancier.
Perhaps the greatest time-waster for me personally has been attempting to keep up with what we “should” be doing each week :) Maybe the great lesson in this post transcends testing: find what works, be aware of what’s new, and only adopt what improves your process.
Wow, I’m one preachy bastard.
Sean
· 1 year ago
This makes a lot of sense. I like it, but I’m torn.
I love the ease of shoulda, but the overhead of creating all of those records for every context is killing me. Short of disconnecting the database and using unitrecord, I had to speed up our test run. I was able to drop it from 260s to 100s by switching to Ruby Enterprise Edition.
So, we’re good with everything for now, but we’ve really just postponed dealing with the core issue.
Though, really, the problem is using shoulda in combination with a data factory (we use factory_girl, but that’s neither here nor there). In trying to obliterate fixtures from our app, I converted a couple of test files over to factory_girl (using .build where I could), and the test time more than doubled in some cases. Fixtures are brittle (in a future-tense way), but when you’re setting up a record in a setup method, they blow everything else away for speed. (We still went with the factories, but we have a couple of fixtures sitting around, waiting for conversion.)
Dan Pickett
· 1 year ago
People want to get crazy with mocking and stubbing everything in controller contexts. That just makes a mess imo. But, eliminating the persistence layer from functional tests still remains a problem.
Sometimes, I’m really uncomfortable about macros. They’re very convenient and easy on the eyes, but often times they are pretty inefficient. I’ve been trying to mitigate some of it with nested contexts but that makes the tests harder on the eyes.
I think it’s a hard balance between these things. You want tests to be easy to write, but you also want them to run quickly. You want them to be expressive, but honestly do you every show clients test output? You want to make tests so they aren’t brittle, but you also want them to be isolated and expressive of your intent. All these things I believe are a problem with TDD and not Ruby itself, but I believe that Ruby developers are doing the most innovative things in attempting to solve these issues and achieve true TATFT qualm.
Great post.
Jeremy McAnally
· 1 year ago
Let me voice my strong disagreement.
First, that’s a very poor example of why context blocks are bad. That’s just a crappy way to write those tests, not the library’s fault. Deciding to use macros like that are your own fault. You could just as easily write the same code in Test::Unit as context (or probably Shoulda) and avoid the problem you point out. And context has before(:all), so you can fix the larger problems here very easily.
Second, your criticism of how Shoulda macros should work is weird. I don’t know about Shoulda, but in context you can just write those test methods and call them from your context generated tests just like you point out. Problem solved.
Thirdly, hey! That’s cool that you like Rails’ way of doing things because that’s where I got the idea and it’s what my implementation is based on! :)
This really seems like a criticism of how Shoulda and perhaps RSpec are implemented. I invite you to try context; I think you’ll find that most of the issues are moot due to the way it’s implemented.
Josh Nichols
· 1 year ago
@Jeremy,
To your first point about it being a crappy way of writing tests…
I’ve definitely written similar tests in my time. There’s definitely a code smell in there. How might you suggest writing it instead?
Dan Croak
· 1 year ago
@Jeremy,
Thanks for the comment. I was hoping to hear from you and folks in the RSpec world.
I second Josh’s request to see a good version of the “why context blocks are bad” test.
After reading your comment, I guess my main point from the post still remains. If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?
The only thing potentially missing from Test::Unit is the ActiveSupport::TestCase.test as I wrote it above (and how it is written in context), with no aliases, and nothing further. It belongs IN Test::Unit, however.
We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.
Also, RSpec and Shoulda both have similar ways of doing before(:all). I look at that as another layer of indirection that is necessary when you start introducing contexts, especially nested contexts. It’s solving a problem that shouldn’t be there in the first place.
James Golick
· 1 year ago
I’m of two minds on this issue. Shoulda’s implementation really sucks; that’s a big part of the problem. Most Shoulda macros are completely unreadable, and full of code smells – bad for tests. Shoulda’s implementation of contexts is also somewhat weak.
I got really excited about context when I saw it, because I thought that it was the better-implemented solution I had been looking for. I agree with most of Jeremy’s arguments above.
But, there’s still a big problem with context in my mind, and that’s the dissonance between the names of the tests as they appear in the contexts and should/it blocks, and the way that they actually get defined as test methods.
When a test fails in context, I see test_person_who_is_admin_should_have_many_posts. In order to actually locate that code in the person_test file, I have to search around for a set of contexts (potentially deeply nested) and should blocks that might generate such a test name. Failing tests should be really easy to find immediately, even when you’re new to the code, IMO at least.
Also, and this is obviously subjective, I’m not crazy about looking at deeply indented code. One of the alleged benefits of contexts is that they can be nested. I find that ends up creating test code that’s hard to follow.
Finally, Jay Fields’ expectations is still the most pleasant testing library I’ve ever used. It says: forget about writing tons of strings in your tests – test names are comments, and comments are code smells. Instead, write your tests expressively, and use the assertion as the test name (i.e. expect(5) { 3 + 2 }). It forces you in to one assertion per test without making it hard to work with. I’d really encourage people to give it a try. I can’t say enough good things about it.
Since there’s no way to use expectations in rails, though, there’s a good chance I’ll start my next project on test/unit.
Alex Young
· 1 year ago
One assertion per test doesn’t make sense to me either. Tests have to be atomic in some sense, perhaps conceptually (i.e. this test represents a particular edge case found by a customer). This probably got out of hand at some point with the quest for finding problems where they don’t exist.
I don’t really agree with “all of these testing tools are solutions to non-existant problems” —Sometimes it’s good to switch testing approaches to gain a different perspective. I default to Test::Unit, but sometimes do supporting tests in RSpec to force myself to think differently about my code’s design and quality.
Jeremy
· 1 year ago
You wrote the better way to write that test yourself. :) That or use a before(:all) so that the setup stuff is only run once.
And as to “why,” I think we’ve argued BDD vs. TDD vs. WTFDD to death. If you dig Test::Unit, go for it. If you like Shoulda or context or whatever-lib, then use it. If you like RSpec, go for it. It’s about focusing on the right things.
I forgot to address your comments on being APi compatible with RSpec and Shoulda. It’s so transitioning to context doesn’t require you to rewrite all of your tests. You can move Test::Unit, Shoulda, or RSpec (using matchy) with very minimal changing of your test code.
Giles Bowkett
· 1 year ago
Jeremy McNally’s new context library is tiny, yet it is trying to be API-compatible with RSpec, shoulda, etc. This blows my mind. I’ve talked to Jeremy and I think he’s a great guy but I don’t understand this as a design goal at all.
I think the design goal is obvious: universality. Highly compatibility means high availability.
Daniel Lyons
· 1 year ago
High compatibility with what? Many of these libraries are themselves changing from minor release to minor release.
Jeremy
· 1 year ago
@Daniel: I don’t plan on 100% compatibility with anything. I hit about 90% compatibility with RSpec et. al. which is all most people use. It’s not like I’m trying to replace RSpec or something, but if I can add compatibility with a few aliases, then why not?
And more to the point, the fact that these libs change so much is one of the reasons I wrote context in the first place. Bugs in one version are fixed in the next, but surprise! The API’s changed so now you’ve got even more failures. </soap_box> :)
Justin
· 1 year ago
Personal mantras I do not recommend, just my personal observations/test cases.
1. “Testing” is not “Specing” 2. “Specify” a truthful statement over “Should” expect a return value. 3. I’d rather macros over meta. 4. I’d rather DSLs with organization over no DSLs. 5. I’d rather huge long specs with details, then tests that try to be DRY
I’m sorry but this argument/article is completely moot (2 years old?), with terrible examples that back up worthless commentary. Find what works at your company, with your teams, or build another Rspec clone. ;)
Secondly, this sounded like a reply to ENTP’s moaning. Nobody cares if ENTP moves off Rspec. Nobody cares that their Rake files were busted. The only thing I learned from Courtney’s article was the comments from Rspec people having to ask THEM for bug reports. Disturbingly, this was the third time this week I read that on a so called “Ruby community leader’s” blog.
Paul Barry
· 1 year ago
Great article. I’m questioning the validity of one assertion per test now. But one thing I do like about contexts is nested context. That allows me to define some things that need to be setup for a set of tests, and then more things for the tests in the inner contexts. Maybe the same kind of thing can be achieved with protected helper methods.
From RSpec specifically, I would also miss pending specs, does test/unit have that?
David James
· 2 months ago
test-unit 2 (the gem) has pend() that does pending
I’m so happy to see something like this. I’ve been happily using test::unit for three years and recently switched to RSpec. I love the reporting it provides me, but I don’t like much of the rest of it.
@Dan said in the comments: “We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.”
I agree wholeheartedly. In fact, in my talk at a local code camp in 10 minutes, I’m going to use test::unit because it’s right there, waiting to be used.
Morgan Roderick
· 1 year ago
A refreshing read, and so many interesting comments!!!
I don’t really see the splintering… what I do see is that there are a lot of dedicated people trying to solve the same overlapping issues. Some of these solutions get’s to be used in ruby-based frameworks, and eventually might also be included into ruby itself.
Sure, there are problems with Shoulda, rSpece, etc… but they do help us write better tests, and like this post shows, also provide excellent basis for discussion and improving.
Personally, I enjoy working with Shoulda, and do write my own macros … it helps keep my tests easy to understand and easy to update…. which I believe was the purpose of Shoulda to begin with: “Making Tests Easy on the Fingers and Eyes”.
Shoulda, rSpec, Test::Unit are not the holy grail of (ruby) testing, they’re stepping stones … and we, the community, will have to test each of them out in all ways possiblle to find they best way forwards.
...
Jay Fields is not always right, but he certainly manages to get people talking. Personally, I find the whole “one assertion per test” to be somewhat stupid … experience has repeatedly shown, that you cannot distill our collective wisdom into one-sentence-one-size-fits-all statements.
Dan: Thanks for sharing your thoughts (and frustrations) :-)
Felipe Giotto
· 1 year ago
Wow! You said everything I wanted to say about one assertion per test! I understand the benefits of one assertion per test, but my tests became too slow. Recently, I changed most of them to the good, old style of one GET and many assertions!
Ken Collins
· 1 year ago
I’ve been deep in AR tests a lot lately and they do have “context” blocks (after a fashion) when you consider that they stuff multiple test classes into a single file. It breaks the use of autotest to be able to automatically map a class to a filename since there will be 3 to 4 (maybe more) in one file.
I think finding a context to put your test in is dirt easy when you use your tools. For instance, TextMate has keyboard commands for “Toggle Foldings At Level” that can go as high/deep to 10 contexts. One keyboard command to see all the contexts from the top level… then drill down. Takes only a few seconds. IMHO, a much much better way to organize code. If tests are flat and let’s say you have 3-4 tests for a single #instance_method, then those should be in a context, else entropy and un-manageability will set in. BTW… thanks for Shoulda :)
nicholas a. evans
· 1 year ago
With respect to: “How will my current Java or C# skills translate?”
I learned rspec back when it was far more magical than it is now (0.5-0.6), and I knew very little ruby at the time. It didn’t take me weeks to learn. It took me minutes. Everything seemed far more obvious to me than it ever did for any of the xUnit toolkits I had used, and they had a nice chart to help people make the transition easy: http://rspec.info/documentation/test_unit.html</p>
Methinks the “rspec is too hard to learn” debate is rather silly… because the people making the argument often say “not for me… but you know… for others it would be.” http://weblog.raganwald.com/2008/02/lets-make-t...>
The only thing I missed from C#/nUnit was the VS.Net/Resharper code completion and IDE support. But I got over that quickly enough.
Jay Levitt
· 1 year ago
“If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?”
+1! This is why I still code everything in machine language. It’s all Turing-complete anyway.
Brian
· 1 year ago
Cucumber has definitely come in handy in our team’s project, although I don’t think we use it in the same way as others. We don’t use it as acceptance testing as much as very high level integration testing. Our app has a lot of various (extremely critical) edge cases that need to function in a specific way. We’ve had controller specs and model specs pass, but catch problems with Cucumber, such as certain input causing problems, or not being presented with a certain form at a specific point.
What our team did a while back was nuke the view specs folder and replace them with cucumber. It makes far more sense. There doesn’t seem to me to be much value in speccing a million expectations about the structure of the page. They’re very brittle and don’t help. It makes more sense to say that this specific sort of user should be able to do this specific activity and not worry about which controller, models, and views are being used and what the markup looks like. This, to me, is where Cucumber shines.
Dan Hensgen
· 1 year ago
I like how expressive one-assertion-per-test is, but bummed with the performance.
I’ve been making use of the optional description parameter to all assert methods. It’s not quite as pretty in the source code, but it lets me describe what I’m asserting and still performs well. Plus, it’s built right in to Test::Unit.
Yossef
· 1 year ago
Jaime really hit the nail on the head with moderation.
I love contexts, and I love the more natural feel and flow of “BDD tools” like RSpec. I like typing in natural sentences instead of using underscores. And I want to break something every time I have to type “assert_equal”. I switched to Shoulda from the standard Ruby Test::Unit because of some of these things, and though I was still stuck with assertions, nested contexts were a joy.
It was during the Charlotte RubyConf that we (OGC) talked to David Chelimsky about nested contexts in RSpec. He seemed unsure because some people might go too far and some people might be confused. We told him that it wasn’t worth keeping a useful feature out because of concerns that some won’t use it well. I can’t say it’s all because of us, but it wasn’t long after that RSpec had nested contexts. And once they were available, I switched and stayed there.
Recently, I’ve been looking at other choices. I’m starting to play with bacon now. RSpec has grown and become magical to the point I’m not sure about it, and I want something small and simple to understand. But I want it to be a spec DSL. I want my executable specification.
Why should we stop building libraries and tools and build apps instead? That’s a false dichotomy. Apps are built on libraries and using tools.
I have a lot more to say because I care a lot about testing, but I’ll stop for now. Moderation, after all.
Paul
· 1 year ago
I personally have had a very good ROI on my time learning RSpec.
Scott Becker
· 1 year ago
@Dan – “We need to stop making new libraries and new tools.”
Yeah! Lets get back to Java, or ASP, or COBOL. It worked well enough! ....umm, what?!
I get what you mean, you don’t want to have to keep learning new tools. But then why even go through the effort of learning Ruby or Rails, or etc etc. At some point you had to stop going down the previously comfortable road and learn those.
For me, BDD has obvious benefits. Clarity of intent. Readability. This speeds up maintenance down the road.
As to one-assertion-per-test (or spec), thats just a design goal, not a requirement. But I’ve seen what packing multiple assertions into one test looks like, and it ain’t pretty. You get assertions that have nothing to do with the name of the test, and later on you come back and wonder why, but too bad, because you can’t go back and ask either yourself or your co-worker or whomever it was what they were thinking 6 months ago. Make your test do one granular thing, and you can easily understand why. Especially if you write the test first. This is do-able in Test::Unit, but I think the nested contexts make this a lot easier and more DRY.
Good post though, it’s good to occasionally step back and consider why.
Adam Williams
· 1 year ago
I agree that the one assertion per test is worth considering, but not to be idolized. I agree that we should use all things in moderation. I agree that RSpec internals used to be insane, but the conceptual weight of using the thing has never been a challenge. I’ve decided I like RSpec just fine. Great post. Got me thinking again. As for cost of loading data, check out http://github.com/aiwilliams/dataset. I’d love some feedback. Be gentle.
staffingpowernurse
· 2 weeks ago
The information for RSpec is very nice. Thank for you.
All Comments
I certainly don’t pretend to be in the same class as a lot of gurus that have added their two cents to this post. That being said, I have yet to hear the magic word, “moderation”.
Contexts don’t make you nest 10 layers deep. You do. I’d rather have the ability to nest a little, because it’s one of the things I love about Shoulda and RSpec. Same goes for one assertion per test. Alex nailed it – tests should be atomic. Test as narrow a case as possible, but use all the assertions required to ensure that test passes.
As contractors, we have a myopic view. Of course we’re stuck learning multiple frameworks. We have to be able to jump into someone else’s code, and that means maintaining a wider skill set than those who work with the same applications long term.
So if that’s part of the argument for Test::Unit, then factor it out. Because it’s not a barrier to entry to any person, team or company looking to adopt Rails. I’d say ALWAYS start with Test::Unit, because it’s the foundation before you decide you need something fancier.
Perhaps the greatest time-waster for me personally has been attempting to keep up with what we “should” be doing each week :) Maybe the great lesson in this post transcends testing: find what works, be aware of what’s new, and only adopt what improves your process.
Wow, I’m one preachy bastard.
This makes a lot of sense. I like it, but I’m torn.
I love the ease of shoulda, but the overhead of creating all of those records for every context is killing me. Short of disconnecting the database and using unitrecord, I had to speed up our test run. I was able to drop it from 260s to 100s by switching to Ruby Enterprise Edition.
So, we’re good with everything for now, but we’ve really just postponed dealing with the core issue.
Though, really, the problem is using shoulda in combination with a data factory (we use factory_girl, but that’s neither here nor there). In trying to obliterate fixtures from our app, I converted a couple of test files over to factory_girl (using .build where I could), and the test time more than doubled in some cases. Fixtures are brittle (in a future-tense way), but when you’re setting up a record in a setup method, they blow everything else away for speed. (We still went with the factories, but we have a couple of fixtures sitting around, waiting for conversion.)
People want to get crazy with mocking and stubbing everything in controller contexts. That just makes a mess imo. But, eliminating the persistence layer from functional tests still remains a problem.
Sometimes, I’m really uncomfortable about macros. They’re very convenient and easy on the eyes, but often times they are pretty inefficient. I’ve been trying to mitigate some of it with nested contexts but that makes the tests harder on the eyes.
I think it’s a hard balance between these things. You want tests to be easy to write, but you also want them to run quickly. You want them to be expressive, but honestly do you every show clients test output? You want to make tests so they aren’t brittle, but you also want them to be isolated and expressive of your intent. All these things I believe are a problem with TDD and not Ruby itself, but I believe that Ruby developers are doing the most innovative things in attempting to solve these issues and achieve true TATFT qualm.
Great post.
Let me voice my strong disagreement.
First, that’s a very poor example of why context blocks are bad. That’s just a crappy way to write those tests, not the library’s fault. Deciding to use macros like that are your own fault. You could just as easily write the same code in Test::Unit as context (or probably Shoulda) and avoid the problem you point out. And context has before(:all), so you can fix the larger problems here very easily.
Second, your criticism of how Shoulda macros should work is weird. I don’t know about Shoulda, but in context you can just write those test methods and call them from your context generated tests just like you point out. Problem solved.
Thirdly, hey! That’s cool that you like Rails’ way of doing things because that’s where I got the idea and it’s what my implementation is based on! :)
This really seems like a criticism of how Shoulda and perhaps RSpec are implemented. I invite you to try context; I think you’ll find that most of the issues are moot due to the way it’s implemented.
@Jeremy,
To your first point about it being a crappy way of writing tests…
I’ve definitely written similar tests in my time. There’s definitely a code smell in there. How might you suggest writing it instead?
@Jeremy,
Thanks for the comment. I was hoping to hear from you and folks in the RSpec world.
I second Josh’s request to see a good version of the “why context blocks are bad” test.
After reading your comment, I guess my main point from the post still remains. If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?
The only thing potentially missing from Test::Unit is the ActiveSupport::TestCase.test as I wrote it above (and how it is written in context), with no aliases, and nothing further. It belongs IN Test::Unit, however.
We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.
Also, RSpec and Shoulda both have similar ways of doing before(:all). I look at that as another layer of indirection that is necessary when you start introducing contexts, especially nested contexts. It’s solving a problem that shouldn’t be there in the first place.
I’m of two minds on this issue. Shoulda’s implementation really sucks; that’s a big part of the problem. Most Shoulda macros are completely unreadable, and full of code smells – bad for tests. Shoulda’s implementation of contexts is also somewhat weak.
I got really excited about context when I saw it, because I thought that it was the better-implemented solution I had been looking for. I agree with most of Jeremy’s arguments above.
But, there’s still a big problem with context in my mind, and that’s the dissonance between the names of the tests as they appear in the contexts and should/it blocks, and the way that they actually get defined as test methods.
When a test fails in context, I see test_person_who_is_admin_should_have_many_posts. In order to actually locate that code in the person_test file, I have to search around for a set of contexts (potentially deeply nested) and should blocks that might generate such a test name. Failing tests should be really easy to find immediately, even when you’re new to the code, IMO at least.
Also, and this is obviously subjective, I’m not crazy about looking at deeply indented code. One of the alleged benefits of contexts is that they can be nested. I find that ends up creating test code that’s hard to follow.
Finally, Jay Fields’ expectations is still the most pleasant testing library I’ve ever used. It says: forget about writing tons of strings in your tests – test names are comments, and comments are code smells. Instead, write your tests expressively, and use the assertion as the test name (i.e. expect(5) { 3 + 2 }). It forces you in to one assertion per test without making it hard to work with. I’d really encourage people to give it a try. I can’t say enough good things about it.
Since there’s no way to use expectations in rails, though, there’s a good chance I’ll start my next project on test/unit.
One assertion per test doesn’t make sense to me either. Tests have to be atomic in some sense, perhaps conceptually (i.e. this test represents a particular edge case found by a customer). This probably got out of hand at some point with the quest for finding problems where they don’t exist.
I don’t really agree with “all of these testing tools are solutions to non-existant problems” —Sometimes it’s good to switch testing approaches to gain a different perspective. I default to Test::Unit, but sometimes do supporting tests in RSpec to force myself to think differently about my code’s design and quality.
You wrote the better way to write that test yourself. :) That or use a before(:all) so that the setup stuff is only run once.
And as to “why,” I think we’ve argued BDD vs. TDD vs. WTFDD to death. If you dig Test::Unit, go for it. If you like Shoulda or context or whatever-lib, then use it. If you like RSpec, go for it. It’s about focusing on the right things.
I forgot to address your comments on being APi compatible with RSpec and Shoulda. It’s so transitioning to context doesn’t require you to rewrite all of your tests. You can move Test::Unit, Shoulda, or RSpec (using matchy) with very minimal changing of your test code.
Jeremy McNally’s new context library is tiny, yet it is trying to be API-compatible with RSpec, shoulda, etc. This blows my mind. I’ve talked to Jeremy and I think he’s a great guy but I don’t understand this as a design goal at all.
I think the design goal is obvious: universality. Highly compatibility means high availability.
High compatibility with what? Many of these libraries are themselves changing from minor release to minor release.
@Daniel: I don’t plan on 100% compatibility with anything. I hit about 90% compatibility with RSpec et. al. which is all most people use. It’s not like I’m trying to replace RSpec or something, but if I can add compatibility with a few aliases, then why not?
And more to the point, the fact that these libs change so much is one of the reasons I wrote context in the first place. Bugs in one version are fixed in the next, but surprise! The API’s changed so now you’ve got even more failures. </soap_box> :)
Personal mantras I do not recommend, just my personal observations/test cases.
1. “Testing” is not “Specing”
2. “Specify” a truthful statement over “Should” expect a return value.
3. I’d rather macros over meta.
4. I’d rather DSLs with organization over no DSLs.
5. I’d rather huge long specs with details, then tests that try to be DRY
I’m sorry but this argument/article is completely moot (2 years old?), with terrible examples that back up worthless commentary. Find what works at your company, with your teams, or build another Rspec clone. ;)
Secondly, this sounded like a reply to ENTP’s moaning. Nobody cares if ENTP moves off Rspec. Nobody cares that their Rake files were busted. The only thing I learned from Courtney’s article was the comments from Rspec people having to ask THEM for bug reports. Disturbingly, this was the third time this week I read that on a so called “Ruby community leader’s” blog.
Great article. I’m questioning the validity of one assertion per test now. But one thing I do like about contexts is nested context. That allows me to define some things that need to be setup for a set of tests, and then more things for the tests in the inner contexts. Maybe the same kind of thing can be achieved with protected helper methods.
From RSpec specifically, I would also miss pending specs, does test/unit have that?
http://test-unit.rubyforge.org/test-unit/
I’m so happy to see something like this. I’ve been happily using test::unit for three years and recently switched to RSpec. I love the reporting it provides me, but I don’t like much of the rest of it.
@Dan said in the comments: “We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.”
I agree wholeheartedly. In fact, in my talk at a local code camp in 10 minutes, I’m going to use test::unit because it’s right there, waiting to be used.
A refreshing read, and so many interesting comments!!!
I don’t really see the splintering… what I do see is that there are a lot of dedicated people trying to solve the same overlapping issues. Some of these solutions get’s to be used in ruby-based frameworks, and eventually might also be included into ruby itself.
Sure, there are problems with Shoulda, rSpece, etc… but they do help us write better tests, and like this post shows, also provide excellent basis for discussion and improving.
Personally, I enjoy working with Shoulda, and do write my own macros … it helps keep my tests easy to understand and easy to update…. which I believe was the purpose of Shoulda to begin with: “Making Tests Easy on the Fingers and Eyes”.
Shoulda, rSpec, Test::Unit are not the holy grail of (ruby) testing, they’re stepping stones … and we, the community, will have to test each of them out in all ways possiblle to find they best way forwards.
...
Jay Fields is not always right, but he certainly manages to get people talking. Personally, I find the whole “one assertion per test” to be somewhat stupid … experience has repeatedly shown, that you cannot distill our collective wisdom into one-sentence-one-size-fits-all statements.
Dan: Thanks for sharing your thoughts (and frustrations) :-)
Wow! You said everything I wanted to say about one assertion per test! I understand the benefits of one assertion per test, but my tests became too slow. Recently, I changed most of them to the good, old style of one GET and many assertions!
I’ve been deep in AR tests a lot lately and they do have “context” blocks (after a fashion) when you consider that they stuff multiple test classes into a single file. It breaks the use of autotest to be able to automatically map a class to a filename since there will be 3 to 4 (maybe more) in one file.
I think finding a context to put your test in is dirt easy when you use your tools. For instance, TextMate has keyboard commands for “Toggle Foldings At Level” that can go as high/deep to 10 contexts. One keyboard command to see all the contexts from the top level… then drill down. Takes only a few seconds. IMHO, a much much better way to organize code. If tests are flat and let’s say you have 3-4 tests for a single #instance_method, then those should be in a context, else entropy and un-manageability will set in. BTW… thanks for Shoulda :)
With respect to: “How will my current Java or C# skills translate?”
I learned rspec back when it was far more magical than it is now (0.5-0.6), and I knew very little ruby at the time. It didn’t take me weeks to learn. It took me minutes. Everything seemed far more obvious to me than it ever did for any of the xUnit toolkits I had used, and they had a nice chart to help people make the transition easy: http://rspec.info/documentation/test_unit.html</p>
Methinks the “rspec is too hard to learn” debate is rather silly… because the people making the argument often say “not for me… but you know… for others it would be.” http://weblog.raganwald.com/2008/02/lets-make-t...>
The only thing I missed from C#/nUnit was the VS.Net/Resharper code completion and IDE support. But I got over that quickly enough.
“If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?”
+1! This is why I still code everything in machine language. It’s all Turing-complete anyway.
Cucumber has definitely come in handy in our team’s project, although I don’t think we use it in the same way as others. We don’t use it as acceptance testing as much as very high level integration testing. Our app has a lot of various (extremely critical) edge cases that need to function in a specific way. We’ve had controller specs and model specs pass, but catch problems with Cucumber, such as certain input causing problems, or not being presented with a certain form at a specific point.
What our team did a while back was nuke the view specs folder and replace them with cucumber. It makes far more sense. There doesn’t seem to me to be much value in speccing a million expectations about the structure of the page. They’re very brittle and don’t help. It makes more sense to say that this specific sort of user should be able to do this specific activity and not worry about which controller, models, and views are being used and what the markup looks like. This, to me, is where Cucumber shines.
I like how expressive one-assertion-per-test is, but bummed with the performance.
I’ve been making use of the optional description parameter to all assert methods. It’s not quite as pretty in the source code, but it lets me describe what I’m asserting and still performs well. Plus, it’s built right in to Test::Unit.
Jaime really hit the nail on the head with moderation.
I love contexts, and I love the more natural feel and flow of “BDD tools” like RSpec. I like typing in natural sentences instead of using underscores. And I want to break something every time I have to type “assert_equal”. I switched to Shoulda from the standard Ruby Test::Unit because of some of these things, and though I was still stuck with assertions, nested contexts were a joy.
It was during the Charlotte RubyConf that we (OGC) talked to David Chelimsky about nested contexts in RSpec. He seemed unsure because some people might go too far and some people might be confused. We told him that it wasn’t worth keeping a useful feature out because of concerns that some won’t use it well. I can’t say it’s all because of us, but it wasn’t long after that RSpec had nested contexts. And once they were available, I switched and stayed there.
Recently, I’ve been looking at other choices. I’m starting to play with bacon now. RSpec has grown and become magical to the point I’m not sure about it, and I want something small and simple to understand. But I want it to be a spec DSL. I want my executable specification.
Why should we stop building libraries and tools and build apps instead? That’s a false dichotomy. Apps are built on libraries and using tools.
I have a lot more to say because I care a lot about testing, but I’ll stop for now. Moderation, after all.
I personally have had a very good ROI on my time learning RSpec.
@Dan – “We need to stop making new libraries and new tools.”
Yeah! Lets get back to Java, or ASP, or COBOL. It worked well enough! ....umm, what?!
I get what you mean, you don’t want to have to keep learning new tools. But then why even go through the effort of learning Ruby or Rails, or etc etc. At some point you had to stop going down the previously comfortable road and learn those.
For me, BDD has obvious benefits. Clarity of intent. Readability. This speeds up maintenance down the road.
As to one-assertion-per-test (or spec), thats just a design goal, not a requirement. But I’ve seen what packing multiple assertions into one test looks like, and it ain’t pretty. You get assertions that have nothing to do with the name of the test, and later on you come back and wonder why, but too bad, because you can’t go back and ask either yourself or your co-worker or whomever it was what they were thinking 6 months ago. Make your test do one granular thing, and you can easily understand why. Especially if you write the test first. This is do-able in Test::Unit, but I think the nested contexts make this a lot easier and more DRY.
Good post though, it’s good to occasionally step back and consider why.
I agree that the one assertion per test is worth considering, but not to be idolized. I agree that we should use all things in moderation. I agree that RSpec internals used to be insane, but the conceptual weight of using the thing has never been a challenge. I’ve decided I like RSpec just fine. Great post. Got me thinking again. As for cost of loading data, check out http://github.com/aiwilliams/dataset. I’d love some feedback. Be gentle.