Sinterklaas and the quality consultant

In the Netherlands we have a traditional Dutch children’s celebration called Sinterklaas. It has many similarities to the Anglo-American Santa Claus. Sinterklaas is a saint who has his birthday on December 5^th, and he celebrates this by bringing the children presents. The presents are distributed by his helpers, who are not elves, but are called Pieten. A children’s television bulletin reports on a daily basis on the progress of Sinterklaas and his Pieten in collecting and distributing the presents.

This year, according to the children’s bulletin, the distribution of the presents was going to be fully automated. A Piet can now scan a big bag of presents with a scanner, to see if all presents are there. The bulletin showed that lots of times the scanner gave a red signal. Apparently it meant that something was wrong with the bag. There was no clear error message, Piet only knew that the present of some child was not there in the bag. At other times the scanner gave a green signal, yet it was said that the child was no longer living at the address listed in the computer. Chaos everywhere, because it seemed that data was wrong, presents were missing from bags and there was no indication whether they needed to be packed or were already packed, but in a wrong bag. The Pieten could only scan a child’s name and then have to scan each and every bag to see if it was the one for that child.

For me all kinds of red flags appeared!… Sinterklaas used to have a big book with the data of all the children in the Netherlands in it. Did the data migration from the book into the system go wrong? Why didn’t the children pass their new addresses when they moved houses in the last year? Or even worse: did someone hack the system and change the addresses? I thought about GDPR; would Sinterklaas get in trouble with the law over bad security and privacy?

If Piet scans a bag, the only thing he can see is if a present for one certain child is in the bag. It would be more logical to scan and see whose presents are in the bag and whose presents are missing. Did something go wrong in the requirements analysis phase?

The children’s bulletin tried to help Sinterklaas by asking the children to check their addresses on the tv channel’s website. The same night, the website went down due to the enormous flow of visitors. Apparently no one thought of a load test on the website and take precautions. Activation of an extra server when the amount of visitors reached the limit would have prevented the website from going down.

What Sinterklaas apparently lacks is a good quality consultant. I would love to get in touch with him. The main requirement of the customer, Sinterklaas, was to have a fast, easy and error-free handling of the presents. What he received, however, is a system that is difficult to use, does not meet its purpose, and has unclear error messages. Apparently Sinterklaas and his Pieten did not specify the functional requirements in sufficient detail. A quality consultant could have reviewed the specs and challenged them if needed. And yes, in this case that was needed.

But it’s not only the functional requirements that are lacking. As we saw, also the non-functionals needed more attention. A quality consultant would have organised a thorough product risk analysis with the most important stakeholders. The risks that would have been identified before the system was built, have no turned into issues. The usability is below expectations, the data integrity is a real problem, privacy is apparently not taken into account, performance is lacking, and those are only the obvious problems.

A quality consultant might have posed questions about the approach of the development process. Apparently the users of the system, Sinterklaas and his Pieten, were unpleasantly surprise when they started using the scanners. Haven’t they been involved during the development of the system? An agile, cyclic approach of the system development, with regular input and feedback from users, would have prevented much of the issues that now only come to light after the go-live.

Furthermore, a quality consultant would have conducted a thorough test phase. From what it looks like, Sinterklaas and his Pieten did not perform a user acceptance test before the go-live. And see what happens: the scanners produce false positives and false negatives. During the product risk analysis one of the risks that should have popped up is the availability of the system. Which requires a solid load and stress test. Instead, now Sinterklaas ended up with a server that went down, and no back up and recovery possibilities. And when the scanners do work, the data is inconclusive, and sometimes plain incorrect. C’mon guys, a data migration test should have been a standard approach in these situations!

And last, but certainly not least, a quality consultant could have helped with the go live plan. At what moment in time should the scanner system go live? Just before the biggest event of the year? A go-live earlier in the year, and a try-out with a small group of children, might bring to light the same issues, but with a much smaller impact. And what go-live tests should have been done, to have enough certainty that the system is working as expected? Go-live is more than just turning the switch…

In the Netherlands Sinterklaas is an ancient children’s tradition. Traditions are passed on from generation to generation, and for Sinterklaas and his Pieten, that’s a good thing. Santa Claus is an American tradition, but Santa won’t be on solid ground very quickly in the Netherlands. I hope that Sinterklaas will learn from the fawlty introduction of the scanner system, and that the Pieten will have the distribution of the presents in order befor December 5^th. On the other side of the ocean Santa Claus seems to be working very efficient, and if traditions do change, Sinterklaas might be out of competition…