Back to blogs

Agile Testing: We Do It Wrong

Jun 10, 2019

By Katy Sherman

Director of Software Engineering at Premier Inc.

Last year I made a discovery. I realized I had misunderstood testing. I’d thought to test something meant to make sure it worked as expected. Matched the acceptance criteria. Did what it was supposed to be doing.

I was wrong

If you have ever been to IKEA furniture store you might have seen a robot performing a durability test on a chair. In the middle of the store, inside a glass box, the pressure is applied to the famous Poang model 20 times per minute to simulate a 250 lb person sitting in the chair 20 times per minute.

Impressive, right? I have actually owned this chair for over 10 years and it’s still good as new, except for a few scratches coming from my cats.

But if I tried to test this chair using my original definition of a test as “make sure it works”, I would probably just sit in it one time and then declare the test successful! After all, it worked for me.

Would it be enough?

See, I am an average size human, maybe on a smaller side, and I sit in chairs in a very graceful way. At least I hope so.

But what about other potential chair owners? Who might be bigger, smaller, who might want to lean, rock, jump, turn, and swing, while drinking tea and chewing on a sandwich? Those who will use the chair not one time, but for many years?

Now, the idea of testing is shifting from “checking that it works” to “finding creative ways to break it before the users do”.

Break the system before the users break it.

Software users will put your application to a real test, and they also come in different shapes. Some will try random workflows because they don’t know the product very well. The power users will want to dig deeper. And finally, there will be people who will actually want to break into your system with a malicious intent.

So, instead of our typical linear progression from coding to validation, we need to get into a cycle of building the application and breaking it, fixing it and breaking again. We should continue through the cycle of building and breaking until the product becomes resilient.

To break like a user, we need to think like a user.

The real test planning is not about capturing step by step instructions for a test case. It’s about creative imagination at  work trying to tackle the system from many different angles to find its weaknesses.

Invalid or unexpected input goes a long way in getting insight into what the system will do when the user makes a mistake or intentionally feeds it incorrect data. Testing on a large data set, or a small one, standalone or integrated into multiple systems will widen your horizons and knowledge about the product behavior. Trying to act as different users or user personas might lead to unexpected findings. Putting the system under stress as many users try to access it will help to find the system’s limitations.

And if you tested and didn’t find any defects, congratulations, your software is perfect!

Except, I haven’t seen one yet, so it’s probably too early to celebrate. It is possible that you are not looking in the right place.

You never know if the system is free of defects, or if they are hiding and you can’t find them.

Do not despair. The testing work is rewarding in itself, because it creates knowledge. The more experience you are getting about your product and different ways to break it, the more you learn about users and how they break it, the closer you are getting to making your application indestructible, like that plywood IKEA chair my cats like so much.

Build and share knowledge, learn from users, become better at breaking things.

And most importantly, do it in every sprint. That’s the heart of Quality in Agile. We are moving forward so fast, we have to be confident about our product. And the only way to be confident, is to do our best to test it, which means – to break it.

About the Author:

Katy is passionate about:

• Leadership and vision

• Innovation, technical excellence and highest quality standards

• Agility achieved through teamwork, Agile, Scrum, Kanban, TDD, CI/CD, DevOps and automation

• Breaking silos and promoting collaboration of Development, Testing and Operations under cross-functional umbrella of Software Engineering

• Diversity of personalities, experiences and opinions

Things Katy does to spread the word:

•  Speak at Technology conferences (including as an invited and key-note speaker)

•  Blog and participate in group discussions

•  Collaborate with schools, universities and clubs

•  Empower girls and women, help them learn about Technology and become engineers.