Your code has several issues. First, the way that you set the for loop up (incrementing both I and j on every pass) means that no matter what, the index for the key, j, is going to increment for every character in the plaintext string, whether it is a letter or a non-alpha. This is bad. It means that the key will get out of synch as soon as a non-alpha is processed. The key index should ONLY be incremented when a letter is actually encoded, and reset when it gets too big (you should be able to figure out what too big implies.) Because of this, it should NOT be part of a loop construct. A simple var can handle the job nicely.
Next, your encoding has multiple errors. You are correctly subtracting 'a' or 'A' from the letter to be encoded. Think about the reason for doing this and then ask why you aren't also doing it for the key letter? Next, what is the purpose of the %26 and why do you apply that to the key only? This is not as simple as you think. You should really go back and review the videos that cover this.
Have you thought about the case (upper/lower case) of the key letter?
Finally, because of an issue with the pset instructions, I'll give you this one. The prompt for input,
printf("input:"); will cause a check50 error. The instructions say to prompt for input, but then give an example of how the program should behave, which doesn't actually prompt for input, it just waits for input. The latter is what you should be doing. (This is being looked at for correction.)
This should get you going. IF this answers your question, please click on the check mark to accept. Let's keep up on forum maintenance. ;-)