Has the number of bugs in Windows updates increased in the past couple of years? If so, what is the reason for the increase in bugs? That's the question that former Microsoft Senior SDET Jerry Berg, better known as Barnacules, answered in a recent video.
Berg worked for 15 years at Microsoft and one of his roles was to design and develop tools and processes to automate testing for the Microsoft Windows operating system. He left the company after Windows 8.1 shipped to the public.
Microsoft changed testing processes significantly in the past couple of years. Berg describes how testing was done in the late 2014 early 2015 period and how Microsoft's testing processes changed since then.
Back in 2014/2015, Microsoft employed an entire team that was dedicated to testing the operating system, builds, updates, drivers, and other code. The team consisted of multiple groups that would run tests and discuss bugs and issues in daily meetings. Tests were conducted manually by the team and through automated testing, and if tests were passed, would give the okay to integrate the code into Windows.
The teams ran the tests on "real" hardware in a lab through automated testing. The machines had different hardware components, e.g. processors, hard drives, video and sound cards, and other components to cover a wide range of system configurations, and this meant that bugs that affected only certain hardware components or configurations were detected in the process.
Microsoft laid off almost the entire Windows Test team as it moved the focus from three different systems—Windows, Windows Mobile and Xbox—to a single system. The company moved most of the testing to virtual machines and this meant according to Berg that tests were no longer conducted on real and diverse hardware configurations for the most part.
Microsoft employees could self-host Windows which would mean that their machines would also be used for testing purposes. The main idea behind that was to get feedback from Microsoft employees when they encountered issues that they encountered during work days. Berg notes that self-hosting is not as widely used anymore as it was before.
The main sources of testing data, apart from the automated test systems that are in place, comes from Telemetry and Windows Insiders. Windows Insider builds are installed on millions of devices and Microsoft collects Telemetry from all of these devices.
No Comments
