TESTING, FUTURE

The Future of Testing, Part 3

This is a three part blog post, where I will focus on first part summarizing a panel discussion about the future of testing. On the second part I will mention what is already out there. On the third and final part I will share what other people from software testing and software development community think about this topic.

In this third and final part of my series of blog posts about future of testing, I will share what many of the people I respect thought about this topic.

It’s good to mention though that I guided their responses a bit. While giving them also an option to approach it in the way that they personally consider the best for them. In order to give you enough context, I’ve copied below the email I sent to everyone (there were slight differences - because later I added the comment about having the freedom of responding to questions or whatever way (s)he prefers). Then you will have better understanding what people knew when they responded.

The content of the email was:

I’m participating 25th of May (my birthday actually) to a conference in Helsinki and joining a panel discussion there, where we will be discussing:

  • What will software development look like in 1, 3 or 5 years
  • How will that impact testing approaches

Generally to get an understanding what’s going on now and in future and how that will shape the testing approaches we have.

I planned to make a blog post about that discussion. It occurred to me though that it would be great to get couple comments about the subject from people I respect and feel have something to add to this subject. It would be even better if you would be able to share your thoughts and allow me to put them to my blog post later (mentioning your name of course - https://aleksistulonen.com) 

That’s it. Thanks!

Before I will reveal the responses, I’ve mentioned the people here in alphabetical order. Responses are also in similar order. There were also couple people who haven’t yet answered, so I might add them later.

Let’s cut the crap and get to what you came for.

James Bach ([@jamesmarcusbach](https://twitter.com/jamesmarcusbach))

Software development is a big world. Some part of it will look exactly like it does today. Some part of it will look like it did 20 years ago. And some part will look a little different, but not much different.

One thing I’ve found in 30 years is that deep changes are slow in coming.

Context-Driven Testing seems to be gaining greater acceptance (I am very busy the past few years) but it will always be a small part of the pie. It’s too easy to sell dumb ideas from 40 years ago, and it’s too hard to sell skilled testing.

James Coplien ([@jcoplien](https://twitter.com/jcoplien))

Things move slowly and I doubt we’ll be able to notice any trends in a year or three, and maybe not in five. What we do know is that children are starting to be taught programming in schools in a formal (i.e., they get a grade) way and on an unprecedented scale. I suspect they are being taught badly (anyone who knows enough about software to teach it at the elementary level is probably programming kids to have all the same mistakes that your generation has, and they probably didn’t get to be elementary teachers by being great programmers or testers). And while they may not be on the streets in five years, there will in any case be more and more autodidactic programmers — I was seeing this already 12 years ago. There will be more and more programmers developing more and more code. There will be more and more emphasis on basic programming (because it’s fun) and less and less on testing (because it’s not fun, and most grade school assignments will be, by most measures, too trivial to be worth any savvy testing effort). The average number of developers working on a given product will drop, though the standard deviation will greatly increase at the same time. More and more software will be developed by individuals (a rough study of GitHub found that about half the projects there have a truck factor of 1, another 25% of 2).

So I think we’ll see more and more software developed with a cowboy heritage. How much of that affects the commercial sector is uncertain, but I haven’t seen any mass software vendor take quality seriously. Some of the EDA, telecom and aerospace people do, but I suspect we’ll see slow erosion in those areas as real testing domain expertise retires, never to be backfilled by the university-educiated (or grade-school-educated) new hires. This domain expertise grew as the first and second generations of programmers bootstrapped themselves into the workable techniques — techniques arcane enough that they never made it into academia. The next generation of programmers cut their teeth on code before faced with the grim reality of real customers, let alone real customers who needed the reliability of a financial system, an aerospace system, or a system supporting social infrastructure.

If there are still testers, and if testers are still a profession, they’ll become increasingly removed from the autodidact-driven efforts and will also, as a community, suffer from the brain drain at the hand of the retiring pioneers. The one-person dev shops won’t want to double their costs by hiring a good tester in the local market. Software quality broadly will suffer. Companies will become embarrassed, and the go-for-the-quarterly-result effect will cause companies to throw more and more outsourced “monkey testers” at software. Knowledge of scripting tools and management tools will be what defines the tester of the future; the current dearth of statistics mastery and economics knowledge among testers will only grow worse from its current sorry state.

Geez, maybe we’re already in the future…

Lisa Crispin ([@lisacrispin](https://twitter.com/lisacrispin))

What will s/w development look like in 1-5 years?

Continued innovations in languages (elm, for example) and tools will bring change. It seems to me that there is some focus in using tools and languages like these to help prevent defects and build quality in.

I hope I see more and more cross-functional teams collaborating together, with skills and capabilities trumping roles. I hope people will be less locked into a role. For example, if a tester pairs on writing some production code, or a programmer does some exploratory testing, nobody should think that is unusual.

How will that impact testing approaches?

I think we’ll continue to see delivery team members shift to a more agile mindset and get better at collaborating across the team and with the business to build quality into their products. I think (or hope) delivery team members will value learning domain knowledge more. I hope more teams will adopt techniques such as story mapping to build shared understanding of their product and features, and identify the valuable slices to build first. I see more adoption of practices such as ‘Power of Three” or “Three (or four or more) Amigos” approaches, using techniques such as example mapping, and I think this will continue. 

I hope we will see more companies valuing a learning culture and allowing teams to self-organize. I’m seeing more diversity on my own team, and I hope we will see more of that in our profession in general. These things are all critical to innovation.

Amazing automation of CI and deployment and practices such as github flow to allow teams to manage delivery is a reality already. As a tester, this takes away so much time I used to waste either deploying things to a test environment that I had to maintain myself, or putting my detective hat on to see if the CI passed for a particular commit and what environment it might have gotten deployed to. 

On my own team, developers are doing better and better at TDD and BDD, they are starting to do effective exploratory testing, they have such good clean coding practices that other testers and I have a lot of time for exploratory testing. We do so much as a group, for example, doing exploratory testing charters as a group on a new feature before we deliver it to beta. Mob programming and mob testing are catching on. 

Testers will continue to add value as “big picture” people, helping to build the shared understanding among the amigos before we write code, helping to explore and learn more about the features as they are delivered, helping to get fast feedback from our customers and translate that into new features and new tests.

I see a future of lots of collaboration, all of us getting to learn in many areas, and being proud of the valuable software that we deliver frequently at a sustainable pace! (to borrow from Elisabeth Hendrickson a bit).

Janet Gregory ([@janetgregoryca](https://twitter.com/janetgregoryca))

A couple of my ideas on the future of testing.

 

First, I hear the term “shift left” a lot, especially in Europe.  I have a  blog post coming with more detail, but my opinion about it is that it is too linear. I know it is meant to get testing earlier in the process, but what I’m afraid it might generate is more “big up-front” design.  Where is the stopping point?  Instead, let’s think of testing early and often, and keep it more of an infinite loop of feedback vs a linear line.

 

I also do not think testing is a service – anymore than programming is.  It is, and should be treated as part of the development process.

I don’t like that teams are looking only for tester’s with programming skills – it shows me that those organizations still don’t value testing skills. That said, there are contexts where it does make sense. For example, if your team was creating an application that was going to be delivered to another development team – such as a web service, or message handling system. I do think that testers need technical awareness though. They cannot live in a bubble without being collaborate with programmers. Lisa Crisping and I wrote a whole chapter on this in “More Agile Testing”, and it is available as an ebook from Test Huddle.

Applications are changing so rapidly, that testers need to keep up to date. I think there will likely be much more specializations – such as testing mobile apps, or concentration on security. We will learn to work closer with programmers to consider performance and load and scalability within the applications themselves rather than thinking about that as a separate and late testing activity.

Anders Dinsen ([@andersdinsen](https://twitter.com/andersdinsen))

I’d like to introduce two metaphors for types of testing: Supermarket testing and hospital testing.

In many projects we test things that are built and implemented rapidly, developed incrementally and very often in direct response to the needs of users. Thanks to better and richer platforms, software increasingly does not need to be developed from the ground up. Instead standard solutions are adapted and customized to fit to the needs of the customer. Small teams can now build systems that were previously very difficult to implement and required thousands of people. Further, the borders between what used to be complex and is now simple are continuously moved. Testing in such projects is often agile, rapid, informal, and highly value and user driven. 

The type of testing taking place there is what I call “supermarket testing”: Because what we test is simple, comes in a limited number of categories, skilled testers can rapidly and efficiently test almost anything without much knowledge. Testing performance depends on individuals’ performance.  Hospital testing is at the other end of the spectrum and I call it so because hospitals are very complex organizations. They involve a great deal of logistics, and the organization is both very hierarchical, processes oriented and formal - and at the same time a knowledge organization. This apparent paradox creates tension that keeps the hospital in an excited state. It is moving. Always awake. A lot of testing is taking place in similarly large, complex organizations, where products are developed both by vendors and in-house. Products may be simple or are very complex, but they are nearly always very mission critical. For this reason processes, standards, metrics and measurements are used extensively in these organizations to manage and control processes in the organization. Including testing. The performance of the hospital is based on the organization. Not the individual.

Hospital testing is in a crisis and needs to develop. 

Despite processes, standards, metrics and measurements, hospital testing does not find and report the important problems in the projects. We experience frequest delays and losses, even scandals. People are loosing confidence in what large solutions we can build today.

Supermarket testing may seem to be the solution. It seems not not to be in crisis, but the truth is that it fails to help solving the problems of many organizations: For despite the rapid, agile and value and user driven testing taking place in super market testing, this type of testing does not deal with the complex systemic risks that haunt the large organizations and their critical systems. 

Further more, being based on the performance of individuals’ and small teams, it scales horribly.

In this perspective, testing is not living on a burning platform. Testing is on multiple burning platforms and things need to change.

The future of testing will address this problem. We will see new structured testing approaches emerge which take the systemic risks seriously and qualifies them. In the future, testers will no longer look for root causes of problems, but look for properties of the components for the systems being tested that indicate that something, somewhere can potentially develop out of control. We will test to look for fragility and appreciate when systems (including the humans being part of them) have anti-fragile properties.

Only that way can we keep helping build a safer, more sustainable and better world.

Karen Johnson ([@karennjohnson](https://twitter.com/karennjohnson))

The field continues to evolve and will, of course, continue to do so. If I look at software development and testing in the past three decades, I see an emerging field that practically needed “all hands on deck.” By this “all hands on deck” expression, I mean that many people joined the computer field whether they had a background or education in the field, and the industry quickly consumed (and needed) those people. Many people in the field (myself included) were self-taught.

 

Now as the computer industry matures – as well as the programming languages, tools, and practices – the field is changing. Software development in the next 3-5 years may increase in some areas such as mobile development but may slow in other areas such as website development. Also, where people are physically employed is shifting from North America to other countries.

 

Overall, I don’t see the field “booming” as the field did in the beginning. As part of that slowdown, facing a future in computers may become more difficult to achieve without the computer science education or background.  No longer does the field look for “all hands on deck” as much as getting the “right hands at the best price.” (And of course sometimes those two goals are in conflict.) This means opportunities may slim, education and background may be required; and the majority of the work may take place in countries outside of the United States.

 

A current hot topic is test automation and whether the future holds a place for manual testing. This topic has been a hot topic for some years but has surfaced again with more strength and more argument. With the increased use of Agile development and automated testing, there will be less opportunities for manual testing. That does not mean that there is no space for a self-taught manual tester,  but those opportunities will be less in abundance than in the past. In discussing this topic at several testing conferences domestically (in the United States) as well as internationally, I’ve said that if you want to be a manual tester in the future, the pressure to be a “rock star” tester is increasing. Simply put, the positions will be fewer and the need to be excellent at the craft will be essential.

 

If I look at the industrial revolution – and certainly this period will be seen as the computer revolution – there were many jobs in the beginning ,but as the field matured, the number of jobs decreased. The same will be true in testing – the number of jobs will decrease. For some companies, particularly those companies that don’t value testing, they will continue to look for the cheapest labor available and may forever fail to realize that skilled testing cannot always be automated and that cheaper is not always better.

Alan Page ([@alanpage](https://twitter.com/alanpage))

What will software development look like in 1, 3 or 5 years How will that impact testing approaches

Nothing super-different from me here. I expect that more companies – even those not shipping weekly or more often, will still move toward more agile-ish engineering approaches (smaller iterations, smaller batches, build->measure->learn). Data collection and analysis will also continue to grow in importance and use.

 

To me, it means more teams forget about what to call people and integrated / unified engineering teams become more of the norm, and dedicated test teams become rarer and rarer. The good testers will do fine on these teams, as I expect that as more teams move in this direction that they recognize the need for a quality specialist among an “engineering” team.

 

I don’t know that any of this changes testing approaches. Software developers are certainly capable of writing good unit, functional, integration tests and more – and they’ll be doing a lot more of this. The testers who do well on these teams will concentrate on risks to customer experience, and let the developers take care of functional correctness

Amy Phillips ([@itjustbroke](https://twitter.com/itjustbroke))

What will software development look like in 1, 3 or 5 years

I don’t think software development is going to change too much over the next few years. Agile will continue to be around, and continue to add value to teams who succeed in opting it. Others will continue with Waterfall/Agile attempts and continue to fail, or succeed. I hope to see more teams adopting DevOps approaches (Get Ops teams collaborating with Dev teams), and with this maybe making releases becomes less problematic and maybe more frequent (for some teams). 

I think the biggest change is going to be in the type of software development taking place. Mobile will continue to grow, and APIs will continue to lead to connected applications. I hope we see a rise is AI, blockchain, and IOT. 

How will that impact testing approaches

Testers are going to need to become more comfortable with new technology. Frequent releases and improved automation tools are make web easier to release and therefore patch fix. As this continues the need for thorough testing will reduce (for most projects). That doesn’t mean testing is dead, it just means we’ll need to focus on the things that are higher risk/more unknown. For example Testing self-driving cars is going to be a huge, and largely unknown challenge.

On a very personal note I hope testing becomes more of a collaborative role. I would love to see an end to the us vs. them conversations in testing and development. I hope that as more teams see the value DevOps brings to developers and ops people we’ll start to talk more about joining testers with developers. 

Other things that are going on

Awesome testing communities are emerging (thanks to people like Rosie Sherry, Tony Bruce, AST, Let’s Test, etc). I hope this will encourage testers to reach out to others, learn new skills, and stay enthusiastic about the craft.

Maaret Pyhäjärvi ([@maaretp](https://twitter.com/maaretp))

I don’t think much will change yet in 1, 3 or 5 years, other than that our approaches continue to diversify: some companies will embrace the devops-style development and continuous testing, while others figure ways for releasing in small batches that get tested, and others do larger and larger batches. Esp. in the bespoke software sector, the forces of how to make money are so much in favor of waterfall that it takes decades for that to change.

But if there is a trend I’d like to see in testing, it’s towards the assumption of skill in testing. Kind of like this: 

- black-hat hackers are people who learn to exploit software for malice.

- white-hat hackers are double agents who exploit software for malice and then give their information to the companies for good

- exploratory testers are white-hat hackers that exploit software for any reason and give that information for companies. From compromise to more dimensions like “hard to use”, “doesn’t work as intended”, “annoys users”. 

- because exploratory testers are more generalized version of hackers, exploratory testing will go away at the same time as software hackers go away. You get AI that write programs, but will need exploratory testers to figure out if programs that are produced are what you want. 

Huib Schoots ([@huibschoots](https://twitter.com/huibschoots))

I don’t think software development will dramatically change the next 5 years. Many companies are doing agile already and more will start doing agile. I do hope they will get better at it, since I see a lot of “fake” agile around. More and more companies will start implementing CI and CD. Also more companies will try DEVOPS. This has an impact on testing. This means that more stuff will be automated, also many checks will get automated. Scaling agile will be important, since many companies have implemented agile development and will now try to scale it to the whole company.

Impact on testing will be minimal. I think some companies will try to get rid of testers, but they will find out that they need testing skills in their teams. I have seen this in the Netherlands with several companies. They misunderstood testing and thought that developers could do the testing easy (BTW this was also motivated by the many “old school” functional testers they employed). As started when agile came around, the need for testers to do excellent testing will continue to grow. Testers do not need to be programmers to be able to automate checks, but they will need to become more technical. By that I mean that they have to get involved in unit testing, TDD, etc. They also need to make sure the right checks get automated. So deep understanding of automated frameworks, understanding of software development, insight in how coding works, being able to perform code reviews, etc.

Sami Söderblom ([@pr0mille](https://twitter.com/pr0mille))

Q: What will software development look like in 1, 3 or 5 years?

A: I think the hype around DevOps will grow and reach even the most resistant places within 1 to 3 years. I’m not that keen on hyping about the culture and activities around it, because that was something we were doing when I started in IT over 12 years ago. A lot of it falls under common sense (whatever that may be :). The technology is however something worth hyping about. Tools in modern build and deployment pipelines, containers, higher abstraction levels in coding, etc. are closing the gap between programmers and people who focus on other things. They lower the bar of us mortals joining development work and being useful at it. And I’m not talking about just testers, but everyone who has a stake at delivering company’s products and services. Technology that enables this kind of teamwork is worth hyping about.

But 5 years ahead… I have no idea. Whatever quess I’d make would probably be wrong, so let’s move on. :)

Q: How will that impact testing approaches?

A: From what I’ve seen, testing will be and actually already is done faster and more tools will be used to aid it. What happens in development already and especially what will happen within the next years will put us testers into a test. We will be stripped down from our comfortable life of fiddling around with nonsense, and driven into a chaos. But this is not a bad thing. I welcome the idea in a sense that, because everything moves so quickly and with less and less resources, we cannot afford to build and maintain massive test case libraries, uphold cumbersome ceremonies, and ultimately focus on something other than testing itself. Skill becomes imperative. Skill to master testing itself, and the tools used in it. Chaos is cakewalk to the skillful. :)

Of course there is always the discussion whether we need testers or not. I think that the more we have to explain our relevance, the more we become irrelevant. We can point out our relevance through our actions. By joining development teams and through hands-on work we can matter. Even though our co-workers may know a lot of things from the realm of software testing, they choose to focus on other things. They live by other incentives. Digging for information (which is the ultimate goal of testing) often requires going “against the grain”, asking the hard questions and looking into directions no one else is looking. We are the masters of that, because we choose to focus on that. Skillful in testing. And where as our primary focus is on risks, we can also find information about new technologies and possibilities altogether. Just yesterday I kept a workshop to a group of 20 developers, who were doing marvellous things, but who - due to the reason they focused on something else - had very little knowledge about certain crucial quality dimensions and techniques that could help them to pinpoint certain risks with minimal effort. I could also share my experiences on deploying tools and approaches that are widely used in DevOps environments, because I’ve had the luxury of working with some of the best in that field. At that point I wasn’t just a tester. I was a part of the development team. A relevant part. What I’ve done in testing and as a tester for these 12+ years helped me in that.

James Thomas ([@qahiccupps](https://twitter.com/qahiccupps))

[1] What will software development look like in 1, 3 or 5 years?

If I was answering this kind of question for a panel discussion, these are some of the kinds of things I’d be thinking…

“Software development” is a big topic. It’s has been in the past, and is now, an extremely diverse discipline and I would be surprised if that changes in the short or medium term. Its diversity comes in many flavours. For example; the applications being developed, the technologies used to develop them, the methodologies used to service that development, the people selecting and working in those methodologies, the companies in which those people work, the needs and desires of the customers of those companies, …

So, a glib answer: I doubt the big picture will change very much.

If we want to look for smaller pictures, though, I might ask what factors of software development we could consider. Some I’ve mentioned above; others might be: practices, roles, gender differences, thought leaders, the positions put forward by thought leaders, perception of success vs failure for development projects, public perception of success vs failure, skill levels of software engineers, roles and role titles and role separation, attitudes towards certfication, …

If we want to predict change (including no change), I might ask what kinds of metrics (formal or informal) I would consider worthwhile comparing between now and then. These would naturally differ with each factor and for each I would wonder what data I could find to analyse that would give me some basis on which to establish a view of “now” and form a view on how it would be reasonable to project forward to a “then”.

I would take a view on whether to actively look for the predictions of others. I believe there’s a tendency of predictions to converge towards a norm if the predictors see each other’s predictions because people tend not to want to be outliers or to suggest radical change is likely. (Similarly with experimental results in science.) 

I would want to be clear about the extent to which I think that any predictions I made were general and how much they are biased towards my own relatively narrow view of the industry (I have been at the company that I founded for 15 years and had no commercial software development experience prior to that).

Having said all that - and given that I’m not on the panel, am only serving a provoker of ideas for you, and so feel that I can be less rigorous in my research and more off-the-cuff in my predictions - here’s a few things from what I’ve been reading and hearing and from thought experiments:

  • continued trend towards what we might call “disintegration” 

  • … more and more dependencies will be distributed and outside the control of the creator of any piece of software. 

  • … my application will rely on third-party libraries, on third-party APIs, on third-party hardware, … 

  • … none of which is directly under my control and much of which is pulled in at /runtime/

  • continued trend towards faster and faster roll-out of product in smaller and smaller increments

  • continued trend towards a profusion of devices (including Internet of Things) and platforms in which users expect products to run 

  • … and ways in which they expect their data to “just” be available on and synchronised across them

  • security - at small and large scale - will continue to be a major headache. 

  • … new devices and software will be vulnerable at the consumer level; new services and software will be vulnerable at the corporate level.

  • software will continue to become “self-learning” and consequently less-understood by anyone. 

    **

[2] How will that impact testing approaches?

So, in the same vein, here’s some possible outcomes:

  • the disintegration of “the product”, platform profusion and faster roll-out 

  • … is likely to lead to increased pressure to automate combinations of configurations
  • … but in turn might call for smart ways to decide what to test and how to test it and when and where
  • … and prompt questions about what the boundaries of any product are: where does “our stuff” stop and “their stuff” end?

  • … and for whose benefit will we be trying to draw that distinction: ourselves, our company, our customers?

  • … and will the decision be the same in each case, context, time? (Answer: no)
  • security problems will put pressure on testing teams to provide “certainty” and reassure stakeholders that “this will never happen again”
  • … when big, public projects fail, this pressure will be even greater

  • … perhaps there will even be public debate about what testing means and should be

  • … and if there is, perhaps also public debate about what software development is and looks like, so that the public might understand it better

  • … as most people probably have some idea what factory work looks like these days, but probably little about what a software house does.

  • “self-learning” systems will provide hard questions about what “expected behaviour” is

  • … and testers (and the people they serve) will need to decide the extent to which they care to /understand/ vs /observe/

Jerry Weinberg ([@jerryweinberg](https://twitter.com/jerryweinberg))

Jerry’s response requires a bit explanation because I went through this subject in a lunch conversation with him. I regret of not having for example voice recorder or making notes during or after the discussion. But the key point of the discussion, and what Jerry thought about the future of testing, is summarized in following quotation from one of his books (Change Planned and Unplanned):

For an engineering discipline, though, forty years is not a very long time to mature. One difference, then, is that we’re a very young engineering discipline, and maturity tends to mean reliability. In steam boiler engineering, for instance, on a capacity basis, the safety of boilers increased by a factor of 1013 in the 110 years between 1850 and 1960. Put another way, if the safety of boilers had not increased by a factor of 1013,  we would all be dead, many times over, killed in steam boiler accidents.

Of course, we would not actually all be dead, for without great increases in reliability, the boiler industry would never have grown to the enormous size it enjoys today. Even so, in 1955-63, 29 explosions occurred, with 32 deaths and 31 injuries. Like software engineers, mechanical engineers don’t think that’s good enough, and many of them study successes and failures to improve their record. 

I’ll leave it for you to figure out how this metaphor is related to future of software development and furthermore, future of testing.