Could AI test itself? (Part 2)

We might not have solid strategies for testing AI systems, but there are already established ways we can use AI  to assist with testing solutions. AI can be used to assist with testing activities throughout the test process. From planning to completion, there are many points in the test process where AI can streamline or lend a helping hand. Using AI to increase productivity in the testing process will undoubtedly save time and money. However, it is not a silver bullet to solve all testing problems. In fact, there are new issues with utilising AI in these processes that people should be aware of when putting them into practice.

AI can be used to extract information quickly and summarise it into a desired format. This can help generate basic test scenarios from acceptance criteria. We leverage this at Razor to accelerate the creation of test scenarios, a notoriously time-consuming task. While AI-generated scenarios provide a solid foundation, human oversight is essential to ensure that all critical scenarios are covered. Tools exist already that integrate this into the test process, helping cut down the time testers use to generate test scenarios. 

From the challenges outlined already, it is clear that testing AI is a time-intensive process in order to achieve a quality output. Since utilising AI can save us so much time, you might be wondering why couldn’t AI just test itself? 

Well, an example of this is to use AI to rank the outputs given by an AI system in response to input from a user. This could save lots of admin time in testing. However, this still poses the same challenges as testing AI systems themselves. It is error-prone. 

It is not a new idea that AI can assist with writing code. Copilot has become an established developer tool which does just this. It can be used to assist in writing automated tests. This is something which of course is a helpful asset to the test process but a more recent development is the creation of self healing tests.

Self-healing tests automatically adjust to intentional code changes, preventing false test failures and in turn wasting time investigating them. While this is beneficial, relying solely on AI to identify intentional code modifications versus bugs remains a challenge.

AI systems only generate data from what they are given or the data that is available to them from the past. This means that they are unable to create new ideas from nothing or think outside the box. This can create biases towards the status quo that exists at the time. This in turn means that the results given will never change until new human-generated data and ideas are added. This is specifically important in testing. If a user is solely reliant on AI to generate test scenarios or write automated tests, the AI will tend to write the same tests repeatedly. Other novel types of defects will not be covered by the tests and potentially be missed in testing altogether. This is referred to as the Pesticide Paradox.

In both testing AI and utilising AI in testing it is clear that a human element will always be required.  AI can greatly improve productivity and is more suited to assist in improving the efficiency of mundane tasks that are not an effective use of time. This gives more time back to testers for tasks that require creativity and logic. Although there is a lot to be gained from embracing AI, it is important to use it appropriately and to fully understand its drawbacks and limitations.