I was in a discussion the other week about choosing a language for your acceptance tests. The general consensus was that you should mirror your other code and choose the same language. For example, if you’re using a .NET stack then choosing SpecFlow with your step definitions and DSL written in C# would be the best way forward. As I said in the discussion, whilst I agree that this *can* be the right option, I certainly wouldn’t forgo asking myself and my team a couple of important questions before jointly making a decision.

Who will own the test framework?

This is the big one. This is, far and away, the most important question to ask. The answer will depend on your organisation, the make up of your team and the ability of those writing tests. I think everyone on any team is probably capable of writingacceptance tests in plain English. You know the kind, gherkin scenarios in Given…When…Then… or a similar format. Lots of frameworks, most of which are part of the Cucumber family, can have your tests written in an a way that anyone can read them and understand what’s going on. What I’m talking about is what happens behind the scenes. Those gherkin scenarios and examples will need mapping to some proper code that actually does stuff (for example, replicating a user on a web page or mobile app). Who owns that part of the test framework?

Who owns this part should influence your decision to choose a language. If ownership lies with those who have experience with writing code (for example, professional software developers or perhaps SDETs) then you’ll have more options than if you want to pass the baton over to your manual testers. I’ve worked with companies who want their testers to transition from being purely manual to being automation specialists. If this is you, then you may want to pick a language that’s easy to pick up. I’ve worked with teams who’ve instructed me to write test frameworks in C# because that’s that their developers use. They figure that it’ll be easier for them to teach the testers a language that they know inside out. It’s never worked that well. There are always issues with Visual Studio and it takes a lot longer for testers to learn the C# syntax than other, more forgiving languages.

which_language-1The obvious choice in this scenario is Ruby. You can write Ruby on any machine, with any text editor. You don’t need a powerful computer or an expensive IDE. Your code doesn’t need compiling and the language itself is easy to read *and* write. This is something that’s often overlooked as a requirement for a test language. Many talk about being able to read code & step definitions and that’s important, for sure, but so is being able to write them. Sure, you can create yourself a DSL (Domain Specific Language) that makes it easy to read fluent step definitions, but being easy to read does not necessarily mean easy to write. I’ve seen lots of DSLs where beginners can pick out the English words and work out what’s going on, but writing such steps is harder as the syntax isn’t that obvious. Languages like Ruby are a lot more lightweight and, as such, I’ve had a lot more success teaching beginners to write tests using Cucumber than I have using SpecFlow.

With beginners, there’s also the issue of running into problems. Ideally, you’d want to be able to fix any problems yourself. You may even find yourself making changes to any packages you’re utilising and submitting pull requests so that the rest of the community can use them. But with beginners, this won’t be the case. Beginners need to be able to type their problem into Stack Overflow and be able to find solutions. You’re much more likely to find a solution to a problem you’re having with SpecFlow than you are with StoryQ, for example.

which_language-2

Is there any risk of the tests being too tightly coupled with production code?

This is probably a very low risk, but you should be able to completely change or refactor your production code without it affecting your tests. There’s an argument for deliberately choosing a different language to everything else to ensure this is the case. I can, of course, see other factors overruling this approach. For example, if you’re coding in a Ruby on Rails stack then it would probably be madness to use a language like C# or Java for your tests. Now I love those languages, but they’re probably overkill if you’re just going to be automating a webpage with them.

What are you going to test?

This is one of those questions that might seem important but, after thinking about it briefly, you could be forgiven for coming to the conclusion that it doesn’t matter what you’re testing and that you should pick your language around other, more important factors. Now, all things being equal, I’d agree. But, depending on what you’re testing, things probably won’t be equal at all. For example, if you’re testing a website or a mobile app, things like waits, retries and timeouts will cause you issues if they’re not handled properly. Now of course you could write lots of boilerplate code yourself to handle all that, but many languages have existing packages you can use to make your life easier. If you’re web testing, Ruby has a great gem called Capybara that handles all this. C# has a Nuget package called Coypu, which is also fantastic. I’ve saved myself so much time in the past by using these packages and if I were testing a website again, I’d need to have a very good reason not to use one of them.

Anything else to consider?

There are a few things. What’s the aim of the test framework? Is it purely to test your code? Or will it be used partially as an exercise to help train your testers? How scalable does it have to be? Is it just for your project? Or are you writing a framework for your entire company to utilise in the future with many other projects? What are you actually going to be testing? If you just want a few quick tests for a prototype that may or may not go into production eventually, you may pick something that’s quick & easy to set up, even though you may sacrifice scalability. You can always change later on.

Tl;dr

This post isn’t about trying to lead you in any one direction with your test frameworks. My aim here is to encourage you to think before making a decision. It’s not uncommon for some testers to always pick the same language regardless of what they’re testing. There are others who always pick the same language that the rest of the code is written in. Both of these approaches can work, but you can run into problems with them both, too. The next time you start a project and have to decide on a test framework, don’t just blindly pick the same language you’ve always used and don’t just mirror the development stack. Ask yourself a few questions before making a decision. It may save you time in the long run.