First, since not all the relevant code is here, we have to make some assumptions. First, length = strlen(key). Second, this is in main and not in a function that's called by main. Third, the code uses strcpy to copy argv[1] and not `key=argv[1]'.
Given these assumptions, and hoping that there's nothing else undisclosed that could impact this, the code above should be working correctly. However, it could be simplified.
Sometimes it's better to look for the negative instead of the positive. In this case, the code above is doing both. So, in situations, ask yourself this. Do you take action if it's all alpha or if something is not alpha? In this case, it's the latter. If all the key chars are alpha, the code merely continues from there.
With that in mind, it can be simplified by applying the NOT operator.
if (!isalpha(key[i]))
{
printf("All characters must be alphabetic.\n");
return 0;
}
Anytime you're in a situation where it feels like you have to have an empty code block after an if statement, or just don't know what to put in it, but what you really want to put the real code in an else clause, that's a really good indicator to flip the test to the opposite, like I did here. BTW, you could have just had a curly brace set and it would work, but that's the mark of a noob, so don't do it. You could do it as a test to see what happens, or play with the NOT operator, ! . I always recommend writing short test progs to understand how things work. Nothing like seeing what your own experiments do!
About return values:
It looks like you've conflated three different return value practices. They have little to nothing to do with each other, except that a value is returned.
The first is the program return value. The accepted best practice is that when a program ends normally, it returns 0 to indicate it completed successfully, as expected. When a program terminates on it's own but because of an unexpected or error condition, it should return a non-zero value. The actual return value is up to the programmer. It's usually to indicate what the problem is, or which part of the code generated the error.
The second is bool values and the interpretion of numbers as bools. Simply put, a 0 is always interpreted as false. Period. End of story. A non-zero, whether positive or negative, is always interpreted as true. On the flip side, if a bool has to be cast as an int, false is always cast as 0 and true as 1.
Finally, a return from a function.
When a function is supposed to return a bool and a numeric value is given, it is evaluated as true or false and then that is returned.
Frequently, a function returns an int. For example, the family of issomething functions will all return a 0 when false. When true though, instead of returning 1, these will return a number that's a power of 2, depending on which function it is. (This appears to be a throwback to assembler, where registers are used and certain operations would set certain bit flags in a register.) So, these functions combine the first two concepts. They always return an int, but the value returned can be interpreted as a bool.
Hopefully, this sorts out your confusion.