We LOVE open source. So much so that we created Open Sauce, to give open source projects the ability to test their projects for free on our cloud. And in just a year, we now have over 800 OSS projects testing on Sauce! To celebrate, we'll be exploring some of the different projects test on Sauce and what makes them awesome. Naturally, the first project we wanted to cover was The Selenium Project. You know you can run your Selenium tests on the Sauce cloud, but beyond that, the Selenium Project and Sauce have a long history of working together, and they were one of the first OSS projects to ever test on Sauce. So where did it all begin?
The history
Years ago, the Selenium Project was using Bamboo for CI on one machine, with a VM for Windows and a VM for Mac. The project wanted to move toward releasing weekly updates to Selenium. For them, it meant that they could be more iterative and only needed to fix a week's worth of problems at a time, rather than months worth of issues. But before being able to release, the project had to vet each release for different browser/OS versions. But maintaining all the supported browser/OS versions for testing was a pain. A classic problem if there ever was one! The project also wanted to get a better CI process into place and integrate reliable tests into their process. They began by switching from Bamboo to Jenkins with Google Compute Engine to see what they could get running. Then, they began to work to get their tests running on Sauce. On the Sauce side, Santi made some modifications to the Sauce cloud that would allow the Selenium Project to test a custom version of Selenium, since they were testing unreleased versions. The project modified their tests to be able to run in parallel. And, with that, the Selenium Project had instant access to all the browser/OS versions they needed, whenever they needed them. Magic!
Hooked on Sauce
Fast forward to today, and the Selenium Project now tests about 30 different browser/OS configurations in parallel with every commit. According to the project, testing that took a week now takes an hour, and weekly releases have become easy. They also run JS unit tests on Sauce, and have much less manual testing to do now that much of the testing is automated. The ability to consistently run tests is the greatest benefit Sauce has given the project, as well as having access to log files and videos that are shareable to different contributors. As with most people, they deal with some flaky tests, but rarely have a Sauce-related issue.
Selenium wisdom
Even if you're the Selenium Project, you can almost always improve the way you're testing, so we asked the project to give us tips on automated testing. Here's what they had to say:
Try to keep your testing hermetic - for example, have a bunch of test pages, and serve them individually for every test run that happens
Don't rely on other services for your tests. The Facebook API or Twitter API may be down, which can cause unnecessary failures and cause you to lose the connection between the code you're testing and what you see.
Be smart and think through the browsers you test on. Have the right conversations about what needs to be tested.
Don't test on every browser, only the necessary ones. The Selenium Project only tests on 1 version of Chrome; running in 10 different versions is wasteful. Tests will always have some amount of flakiness, and if you test unneeded browsers you might get a lot more failures.
Do test functionally different browser versions. The project tests both IE 8 and 9 because there are major differences between them.
Finally, try not to need Selenium tests. Sound crazy coming from the Selenium Project, doesn't it? But the key here is that the more you try to be efficient, economical, and thoughtful about your tests, the better off your remaining tests will be. You'll probably never be able to not test at all, but it's a good exercise in restraint. Make sure there's good reasoning behind every test in your suite. Otherwise, it's easy for your test suite to balloon until its unmanageable.
Want to get involved with the Selenium Project?
According to the project, the best way to get involved is to find something you want to fix. There are hundreds of open issues, find one that gets you fired up. Another great way to get involved is to look through the wiki and how-to-build code and write a small failing unit test for the thing you want to improve. That'll motivate people to get involved. Finally, hang out in the IRC channel: #selenium.
What's next for the project?
There are some great things for the Selenium Project on the horizon, including the release of Selenium 3, RCA APIs being ripped out, and a W3C spec. Learn more on Selenium HQ.
