Giant Robots - Latest Comments in A critical look at the current state of Ruby testing

Re: A critical look at the current state of Ruby testing

Brad — Sat, 08 May 2010 14:23:03 -0000

Ah! Thanks--much better now.

Re: A critical look at the current state of Ruby testing

Brad — Fri, 07 May 2010 19:02:27 -0000

Meant 'confused' not 'confusing'... I wonder if it's because they are sorted by 'date' and all the dates (except the one 'reply to' a specific comment six months later) all now (> 1 year later) have a date of "1 year ago" and so it's sorting based on some other criteria than date/time? Anyway, still confused, so at least with my reply I now know the most recent 'start' at the top... ;-}

Re: A critical look at the current state of Ruby testing

Matt Jankowski — Fri, 07 May 2010 19:02:03 -0000

Brad - that's been driving me mad for a while - thanks for asking about it.

Turns out there's a disqus setting of "sort by popular" instead of "oldest first". I just set it back to be oldest first....because that's actually possible to follow a discussion with.

Thanks!

Re: A critical look at the current state of Ruby testing

Brad — Fri, 07 May 2010 18:58:07 -0000

I cannot for the life of me follow the comments below (28, at current count). There are replies below *and* above. No indenting. I simply cannot follow the 'discussion'--is anyone else who comes across this interesting post and follow-up discussion confusing like me?

Re: A critical look at the current state of Ruby testing

staffingpowernurse — Fri, 04 Dec 2009 00:20:59 -0000

The information for RSpec is very nice. Thank for you.

Re: A critical look at the current state of Ruby testing

David James — Sun, 18 Oct 2009 13:57:02 -0000

test-unit 2 (the gem) has pend() that does pending

http://test-unit.rubyforge....

Re: A critical look at the current state of Ruby testing

Adam Williams — Fri, 14 Nov 2008 09:19:00 -0000

I agree that the one assertion per test is worth considering, but not to be idolized. I agree that we should use all things in moderation. I agree that RSpec internals used to be insane, but the conceptual weight of using the thing has never been a challenge. I’ve decided I like RSpec just fine. Great post. Got me thinking again. As for cost of loading data, check out http://github.com/aiwilliam.... I’d love some feedback. Be gentle.

Re: A critical look at the current state of Ruby testing

Scott Becker — Thu, 13 Nov 2008 14:17:00 -0000

@Dan – “We need to stop making new libraries and new tools.”

Yeah! Lets get back to Java, or ASP, or COBOL. It worked well enough! ....umm, what?!

I get what you mean, you don’t want to have to keep learning new tools. But then why even go through the effort of learning Ruby or Rails, or etc etc. At some point you had to stop going down the previously comfortable road and learn those.

For me, BDD has obvious benefits. Clarity of intent. Readability. This speeds up maintenance down the road.

As to one-assertion-per-test (or spec), thats just a design goal, not a requirement. But I’ve seen what packing multiple assertions into one test looks like, and it ain’t pretty. You get assertions that have nothing to do with the name of the test, and later on you come back and wonder why, but too bad, because you can’t go back and ask either yourself or your co-worker or whomever it was what they were thinking 6 months ago. Make your test do one granular thing, and you can easily understand why. Especially if you write the test first. This is do-able in Test::Unit, but I think the nested contexts make this a lot easier and more DRY.

Good post though, it’s good to occasionally step back and consider why.

Re: A critical look at the current state of Ruby testing

Paul — Wed, 12 Nov 2008 18:05:00 -0000

I personally have had a very good ROI on my time learning RSpec.

Re: A critical look at the current state of Ruby testing

Yossef — Tue, 11 Nov 2008 14:54:00 -0000

Jaime really hit the nail on the head with moderation.

I love contexts, and I love the more natural feel and flow of “BDD tools” like RSpec. I like typing in natural sentences instead of using underscores. And I want to break something every time I have to type “assert_equal”. I switched to Shoulda from the standard Ruby Test::Unit because of some of these things, and though I was still stuck with assertions, nested contexts were a joy.

It was during the Charlotte RubyConf that we (OGC) talked to David Chelimsky about nested contexts in RSpec. He seemed unsure because some people might go too far and some people might be confused. We told him that it wasn’t worth keeping a useful feature out because of concerns that some won’t use it well. I can’t say it’s all because of us, but it wasn’t long after that RSpec had nested contexts. And once they were available, I switched and stayed there.

Recently, I’ve been looking at other choices. I’m starting to play with bacon now. RSpec has grown and become magical to the point I’m not sure about it, and I want something small and simple to understand. But I want it to be a spec DSL. I want my executable specification.

Why should we stop building libraries and tools and build apps instead? That’s a false dichotomy. Apps are built on libraries and using tools.

I have a lot more to say because I care a lot about testing, but I’ll stop for now. Moderation, after all.

Re: A critical look at the current state of Ruby testing

Dan Hensgen — Tue, 11 Nov 2008 14:26:00 -0000

I like how expressive one-assertion-per-test is, but bummed with the performance.

I’ve been making use of the optional description parameter to all assert methods. It’s not quite as pretty in the source code, but it lets me describe what I’m asserting and still performs well. Plus, it’s built right in to Test::Unit.

Re: A critical look at the current state of Ruby testing

Jaime Bellmyer — Mon, 10 Nov 2008 21:11:00 -0000

I certainly don’t pretend to be in the same class as a lot of gurus that have added their two cents to this post. That being said, I have yet to hear the magic word, “moderation”.

Contexts don’t make you nest 10 layers deep. You do. I’d rather have the ability to nest a little, because it’s one of the things I love about Shoulda and RSpec. Same goes for one assertion per test. Alex nailed it – tests should be atomic. Test as narrow a case as possible, but use all the assertions required to ensure that test passes.

As contractors, we have a myopic view. Of course we’re stuck learning multiple frameworks. We have to be able to jump into someone else’s code, and that means maintaining a wider skill set than those who work with the same applications long term.

So if that’s part of the argument for Test::Unit, then factor it out. Because it’s not a barrier to entry to any person, team or company looking to adopt Rails. I’d say ALWAYS start with Test::Unit, because it’s the foundation before you decide you need something fancier.

Perhaps the greatest time-waster for me personally has been attempting to keep up with what we “should” be doing each week :) Maybe the great lesson in this post transcends testing: find what works, be aware of what’s new, and only adopt what improves your process.

Wow, I’m one preachy bastard.

Re: A critical look at the current state of Ruby testing

Brian — Mon, 10 Nov 2008 18:40:00 -0000

Cucumber has definitely come in handy in our team’s project, although I don’t think we use it in the same way as others. We don’t use it as acceptance testing as much as very high level integration testing. Our app has a lot of various (extremely critical) edge cases that need to function in a specific way. We’ve had controller specs and model specs pass, but catch problems with Cucumber, such as certain input causing problems, or not being presented with a certain form at a specific point.

What our team did a while back was nuke the view specs folder and replace them with cucumber. It makes far more sense. There doesn’t seem to me to be much value in speccing a million expectations about the structure of the page. They’re very brittle and don’t help. It makes more sense to say that this specific sort of user should be able to do this specific activity and not worry about which controller, models, and views are being used and what the markup looks like. This, to me, is where Cucumber shines.

Re: A critical look at the current state of Ruby testing

Jay Levitt — Mon, 10 Nov 2008 15:21:00 -0000

“If I can do the same thing in context as Test::Unit (and Rails), what compelling reason is there to use context or any other tool that comes along?”

+1! This is why I still code everything in machine language. It’s all Turing-complete anyway.

Re: A critical look at the current state of Ruby testing

nicholas a. evans — Mon, 10 Nov 2008 12:21:00 -0000

With respect to: “How will my current Java or C# skills translate?”

I learned rspec back when it was far more magical than it is now (0.5-0.6), and I knew very little ruby at the time. It didn’t take me weeks to learn. It took me minutes. Everything seemed far more obvious to me than it ever did for any of the xUnit toolkits I had used, and they had a nice chart to help people make the transition easy: http://rspec.info/documenta...

Methinks the “rspec is too hard to learn” debate is rather silly… because the people making the argument often say “not for me… but you know… for others it would be.” http://weblog.raganwald.com...

The only thing I missed from C#/nUnit was the VS.Net/Resharper code completion and IDE support. But I got over that quickly enough.

Re: A critical look at the current state of Ruby testing

Ken Collins — Mon, 10 Nov 2008 11:07:00 -0000

I’ve been deep in AR tests a lot lately and they do have “context” blocks (after a fashion) when you consider that they stuff multiple test classes into a single file. It breaks the use of autotest to be able to automatically map a class to a filename since there will be 3 to 4 (maybe more) in one file.

I think finding a context to put your test in is dirt easy when you use your tools. For instance, TextMate has keyboard commands for “Toggle Foldings At Level” that can go as high/deep to 10 contexts. One keyboard command to see all the contexts from the top level… then drill down. Takes only a few seconds. IMHO, a much much better way to organize code. If tests are flat and let’s say you have 3-4 tests for a single #instance_method, then those should be in a context, else entropy and un-manageability will set in. BTW… thanks for Shoulda :)

Re: A critical look at the current state of Ruby testing

Felipe Giotto — Mon, 10 Nov 2008 06:57:00 -0000

Wow! You said everything I wanted to say about one assertion per test! I understand the benefits of one assertion per test, but my tests became too slow. Recently, I changed most of them to the good, old style of one GET and many assertions!

Re: A critical look at the current state of Ruby testing

Morgan Roderick — Sun, 09 Nov 2008 05:42:00 -0000

A refreshing read, and so many interesting comments!!!

I don’t really see the splintering… what I do see is that there are a lot of dedicated people trying to solve the same overlapping issues. Some of these solutions get’s to be used in ruby-based frameworks, and eventually might also be included into ruby itself.

Sure, there are problems with Shoulda, rSpece, etc… but they do help us write better tests, and like this post shows, also provide excellent basis for discussion and improving.

Personally, I enjoy working with Shoulda, and do write my own macros … it helps keep my tests easy to understand and easy to update…. which I believe was the purpose of Shoulda to begin with: “Making Tests Easy on the Fingers and Eyes”.

Shoulda, rSpec, Test::Unit are not the holy grail of (ruby) testing, they’re stepping stones … and we, the community, will have to test each of them out in all ways possiblle to find they best way forwards.

...

Jay Fields is not always right, but he certainly manages to get people talking. Personally, I find the whole “one assertion per test” to be somewhat stupid … experience has repeatedly shown, that you cannot distill our collective wisdom into one-sentence-one-size-fits-all statements.

Dan: Thanks for sharing your thoughts (and frustrations) :-)

Re: A critical look at the current state of Ruby testing

Brian Hogan — Sat, 08 Nov 2008 16:19:00 -0000

I’m so happy to see something like this. I’ve been happily using test::unit for three years and recently switched to RSpec. I love the reporting it provides me, but I don’t like much of the rest of it.

@Dan said in the comments: “We need to stop making new libraries and new tools. How about we focus on making apps? I’m guilty of this, too.”

I agree wholeheartedly. In fact, in my talk at a local code camp in 10 minutes, I’m going to use test::unit because it’s right there, waiting to be used.

Re: A critical look at the current state of Ruby testing

Paul Barry — Sat, 08 Nov 2008 11:04:00 -0000

Great article. I’m questioning the validity of one assertion per test now. But one thing I do like about contexts is nested context. That allows me to define some things that need to be setup for a set of tests, and then more things for the tests in the inner contexts. Maybe the same kind of thing can be achieved with protected helper methods.

From RSpec specifically, I would also miss pending specs, does test/unit have that?

Re: A critical look at the current state of Ruby testing

Justin — Sat, 08 Nov 2008 01:58:00 -0000

Personal mantras I do not recommend, just my personal observations/test cases.

1. “Testing” is not “Specing”
2. “Specify” a truthful statement over “Should” expect a return value.
3. I’d rather macros over meta.
4. I’d rather DSLs with organization over no DSLs.
5. I’d rather huge long specs with details, then tests that try to be DRY

I’m sorry but this argument/article is completely moot (2 years old?), with terrible examples that back up worthless commentary. Find what works at your company, with your teams, or build another Rspec clone. ;)

Secondly, this sounded like a reply to ENTP’s moaning. Nobody cares if ENTP moves off Rspec. Nobody cares that their Rake files were busted. The only thing I learned from Courtney’s article was the comments from Rspec people having to ask THEM for bug reports. Disturbingly, this was the third time this week I read that on a so called “Ruby community leader’s” blog.

Re: A critical look at the current state of Ruby testing

Jeremy — Fri, 07 Nov 2008 23:58:00 -0000

@Daniel: I don’t plan on 100% compatibility with anything. I hit about 90% compatibility with RSpec et. al. which is all most people use. It’s not like I’m trying to replace RSpec or something, but if I can add compatibility with a few aliases, then why not?

And more to the point, the fact that these libs change so much is one of the reasons I wrote context in the first place. Bugs in one version are fixed in the next, but surprise! The API’s changed so now you’ve got even more failures. </soap_box> :)

Re: A critical look at the current state of Ruby testing

Daniel Lyons — Fri, 07 Nov 2008 23:52:00 -0000

High compatibility with what? Many of these libraries are themselves changing from minor release to minor release.

Re: A critical look at the current state of Ruby testing

Giles Bowkett — Fri, 07 Nov 2008 19:07:00 -0000

Jeremy McNally’s new context library is tiny, yet it is trying to be API-compatible with RSpec, shoulda, etc. This blows my mind. I’ve talked to Jeremy and I think he’s a great guy but I don’t understand this as a design goal at all.

I think the design goal is obvious: universality. Highly compatibility means high availability.

Re: A critical look at the current state of Ruby testing

Jeremy — Fri, 07 Nov 2008 19:06:00 -0000

You wrote the better way to write that test yourself. :) That or use a before(:all) so that the setup stuff is only run once.

And as to “why,” I think we’ve argued BDD vs. TDD vs. WTFDD to death. If you dig Test::Unit, go for it. If you like Shoulda or context or whatever-lib, then use it. If you like RSpec, go for it. It’s about focusing on the right things.

I forgot to address your comments on being APi compatible with RSpec and Shoulda. It’s so transitioning to context doesn’t require you to rewrite all of your tests. You can move Test::Unit, Shoulda, or RSpec (using matchy) with very minimal changing of your test code.