Category Archives: Programming

Problems of Software Testing

It always annoys me when I find an error in either an operating system or a user program. The more stupid a bug is, the more it seems incomprehensible how such bugs can remain in a product with today’s modern program development tools.

Perhaps the theory and practice of programming is not as advanced as we would like to believe? Perhaps haste and the rapid hunt for money explain the presence of primitive errors? Or is it something else entirely?

I started to think about this and imagine a testing system that could be used to filter out all programming and system errors. And I realized that such a system cannot be created, at least as far as one type of tests is concerned, the “black-box” (black-box, or external) testing.

This is a testing technique where the tester does not have the source program, only the final product. This is what you need to determine if it is flawless or not. Let’s say that you have the specification in your hands, you check the expectations contained in it one by one, if you have taken them all one by one and found no errors during the run, then the tested product can be considered flawless. Not at all. The specification may not include special cases. The disk runs out of space, the memory runs out, the network connection is interrupted, the user can give the program data that he could not give according to the specification, but the user is just like this, intentionally or accidentally, gives data that the specification makers didn’t think so.

But even if we have a complete specification that includes behavior descriptions for all edge cases and special situations, we cannot perform our testing task perfectly.

To understand this, let’s look at a very simple case, we get a max() function from somewhere (say, in a DLL), which expects two integers and returns the larger of the two. Can we black-box test this function in such a way that we can declare that the function is 100% error-free and will return the larger of the two received integer values under all circumstances?

Let’s write a test program, include the calls max(1, 2) and max(2 ,1), in both cases we should get 2 as a result. To be on the safe side, we call the function in the max(2, 2) way, so we should still get 2. Are we ready with the test? Is the code 100% error free?

In principle, yes. In practice, however, we have to consider the case of the malicious coder, this is a programmer who deliberately hides an error in the code, and can do this in very sophisticated ways. In principle, our 100% error-free code can actually contain a lot of errors.

Because what if there is a detail in the code that works depending on the received arguments, and if one of the arguments is, say, 1,000,000, it returns not the larger argument, but the smaller one. We can only test this and detect the error by calling the max() function in all kinds of combinations up to 1,000,000. And since we can’t know where the error is hidden, we have to try all combinations of calls not only up to 1,000,000, but up to the maximum integer that can be represented on the machine.

This test will take longer than the very first one, we will run it, and we will not find any errors. Can we now declare the function 100% flawless?

No not yet. Our programmer can be even more malicious than that and hide an error based on date and time. To find this, we need to set our machine’s clock to all possible dates and times and repeat the test for all possible integer combinations, since the programmer could have combined the date, time and argument tests in such a way that the function could conceivably will only give an incorrect answer to a single combination in one specific second of the next fifty years, otherwise it will be flawless. But a single error is enough to make the code not completely error-free.

I think we already see the hopeless future. In addition to the arguments and the system time, our malicious coder can take the size of the program, the size of the total and free memory, whether there is an even or odd number in a certain register, it can store how many times it has been called, and it works well or poorly depending on it. The possibilities are practically endless, and because of this it is completely impossible to finish in human time.

Of course, we can say that these examples assume rather absurd malice on the part of our coder, but we didn’t examine how viable this idea was, but we wanted to find out whether perfect black-box testing is possible in principle.

Based on what I have said so far, I see it as proven that external means cannot be used to determine whether a program or code is flawless or not. We can’t even give a percentage estimate, since we don’t know how many undetected errors there are for each detected error.

I don’t want to give the impression that all those who issue a wrong code deserve to be exempted. There’s no question about that. It is simply a fact that black-box testing cannot give us complete security.

So where do we look for complete security? First of all, we can trust our own code created for ourselves, since we know that it was not written by a malicious coder and not with the intention of causing harm. On the other hand, we can increasingly trust those open source codes, which in principle can be checked by many people, so it can be found out if there is possibly malicious code in it. Although, just because something is open source doesn’t mean it’s verified. In principle, every open source code should be accompanied by a checklist of who analyzed the code, when, what parts, and to what depth. Needless to say, such a thing does not exist nowadays, and it is hard to imagine how many man-years it would take to create a checklist for an open source program. And this list would only be reliable until the first modification, after which it would have to be revised after each modification.

Now that we’ve seen how reliable black-box testing is, let’s see if we can improve its reliability. For this, we have to open the “black-box” to some extent, i.e. we have to look into the code as well. For this, we can use a tracker, a reverse translator, or an automatic analysis program. Thus, we have slightly more chances to find a strange constant or conditional jump instruction, which may indicate malicious code. For example, this may work for a max() function, but we can write a max() function ourselves instead of spending a lot of time analyzing it. However, the real codes are usually much, much more complicated than the max() function in our example.

Finding hidden code in a larger program by tracing or disassembling it without understanding the operation of the entire program is a very big task.

So it looks like we won’t be able to reach 100% certainty with the “open black-box” technique, or even with the analysis of the open source code.

To examine if we can go further with other tools, it is best to take a closer look at antivirus programs to see what level they have reached in the automatic detection of malicious codes. Unfortunately, we can say that there is a lot of uncertainty. When the heuristic analysis of an otherwise popular, widely used, purchased search engine has to be turned off because it deletes programs that we cannot work without (not one we wrote ourselves), then it seems that the analysis finds code that is not a virus, or it is too lenient or meek, thus making even truly viral code look harmless.

Perhaps expecting 100% performance from automatic code analysis is as impossible an expectation as trusting writing a program to tell any other program whether it will enter an infinite loop or stop at some point. We already know that this is impossible. Perhaps the 100% detection of malicious code and program errors by automatic analysis is an equally impossible task. Of course, I can’t prove this.

Now is the time for someone to make a list of open, yet to be solved computing problems, similar to David Hilbert’s 23-point list (which listed problems to be solved in mathematics), this list could include the question: “Is it possible to write a program that proves the correctness of a program, if not, why not?”. Perhaps we already see the answer to the question, probably the intractability of the stopping problem is the key here as well.

I know that if such a program were to be completed, it would first have to be verified that it is correct, and perhaps this is the reason why this program will never be completed either…

Nyíregyháza, September 27, 2018 – April 7, 2019

English translation: August 29, 2023

Delphi Is 25 Years Old

I was with him from the beginning. I have been working with Borland Pascal since it was released after 7.0 as Delphi 1.0. Our highlights: Delphi 1.0, Delphi 5.0, 6.0, 7.0, Delphi 2007, Delphi XE4, XE7, Delphi 10.1 Tokyo, 10.2 Rio. The numbering scheme has changed several times, as has the owner, first Borland Delphi, then Codegear Delphi, and now Embarcadero. But the most important features have remained: the fastest translator, an easy-to-learn language that is constantly evolving, adding new language elements, and a great, intelligent, easy-to-use IDE. An excellent code editor with a screen designer. The designer and VCL were a very important step forward after Borland’s Pascal Object Windows Library.

Pascal was the first serious programming language I learned, and it remains my favorite programming language to this day. Native string type, which makes text handling easy, overflow checks protect against the security risks that so many C-C++ programs still struggle with today, lots of error possibilities, which cause a lot of headaches for users. Delphi’s rich set of types and components makes it suitable for all kinds of development. The “class” type facilitates object-oriented programming, which is much more convenient and faster than with the old “object” type. VCL components are easy to use, their logical field and method sets, field and method names all help to make programming as simple as possible. An active user community contributes to the expansion of the component set, often with free and open source solutions.

For me, Delphi 2007 was the best release, I love it and I still develop with it to this day. Delphi 6 was also a very successful version, I worked with it at my office for almost a decade. However, I think there are a lot of organizations, including the one where I worked, who got stuck in Delphi 6 development, did not move on to more modern versions in time, and now it is impossible for them to switch. It’s kind of like COBOL, there’s still a huge amount of COBOL code in the world, simply because they can’t rewrite the codebase.

Codegear Delphi 2007

From the XE series, I liked the XE7 the most. It was a very good step to launch the Starter Edition and then make it free, which gave students, those familiar with programming, and small businesses an excellent development tool. I have a 10.1 Starter Edition, it has the nice feature that it remains free and does not expire like the Community Edition that replaces Starter. This edition is also a good tool for the little ones, but the introduction of the time limit is a step back from the Starter Edition.

And now about the negatives. Each version brought new features, but also bugs, the only version I was completely satisfied with was Codegear Delphi 2007, and so far I haven’t encountered any bugs in this version. Currently, for example, direct variable declaration in function blocks generates an error signal on the IDE interface, but at the same time the code can be compiled flawlessly. And this error now occurs in two versions in a row! Has no one tested this feature at all and the bug has survived two releases? In previous versions, it also happened that the IDE translator and the compiler did not work in sync and did not show the same errors, which is of course quite annoying. And what I really miss is the stack log display and the quick help of Borland times (now even the Delphi help is full of C++ stuff, which nobody cares about, since I work in Delphi!), the help is unbelievably slow for the previous chm version, in comparison, I just pressed F1, and lo and behold, the help text appeared immediately, along with lot of examples! And the biggest problem is the lack of efficient automatic memory management, currently the developer is fighting a ceaseless and hopeless battle with memory leaks and rolling up security bugs. To this day, this is Delphi’s weakest point. Delphi also lacks a source documentation tool similar to JavaDoc, which could be used to generate html and chm help from comments in the source code, in such a way that the IDE can also use this help. That is, we can get quick information about our own codes while writing programs in the same way as about the contents of Delphi units.

But anyway, for me it’s the best programming language and development tool I know, and of course I’m really looking forward to the next release, 10.4, which will probably bring new things in the field of memory management. And, of course, I wish Delphi a similarly successful next 25 years! Together, on!

Nyíregyháza, February 15, 2020 – March 22, 2020

English translation: August 30, 2023

Safe Programming Language

My current post on LinkedIn:

Safe language features:

– range check
– arithmetic overflow check
– stack overflow check
– strict typing
– if it also uses dynamic typed variables then a safe runtime type check mechanism
– if a variable goes out of scope, its resources must be freed
– if an object has a live reference it should not be freed
– You should be never allowed to free a resource twice or more
– nil pointer check, or completely avoid pointers (Java!)
– no nil value allowed to assign to any variable (it’s insane that in Java a String could be nil, it should be empty string instead of nil)
– no object should have a nil value, they should be empty object instead.
– no access across freed or otherwise invalid pointer
– code analyzer to detect (if possible) infinite loops
– exception handling, do not allow empty catch blocks or any swallowing of an exception
– detailed stack trace and memory allocation map

…and that’s it for now…