In testing your code, it appears to ask for the string correctly. However, if there is a non-alpha in the string, it will go into an infinite loop in the while loop because the only time you increment i is inside one of the if statements. In fact, this leads to another problem because you can potentially increment i twice inside of a single pass through the loop.
There are a couple of different ways to improve on this. One way would be to increment i once at the end of the loop instead of all over the place inside if statements. Another alternative would be to simply use a FOR loop instead of a while loop, since you already know how many characters to process (strlen() ).
Next, your formulas for encoding are flawed. Let's look at one:
cipher[i] = ((int)plaintext[i] + key) % 26 + 65;
Now, let's use a key of 0 and a letter of
cipher[i] = ( (int)'a' + 0 ) % 26 + 65
= ( 65 + 0 ) % 26 + 65
= (65 % 26) + 65 (parentheses indicate operation priority)
= ( 13 ) + 65
As you can see, this doesn't work. It should have resulted in 'a'. The secret to this formula is that % only works on alphas represented by numbers from 0 to 25 inclusive. You tried to apply it to the ASCII value of the letter.
Next, why do you have
else return 1? Anything other than a lower case alpha would immediately trigger this return and exit the program!
Finally, you are not handling non-alphas (or at least not correctly). I think you thought about it, but you made an error. I'll leave this one for you to discover. ;-)
Fix these and it should get you close to success. There may be other issues, but this will get you going.
If this answers your question, please click on the check to accept. Let's keep up on forum housekeeping. ;-)