Here's how it typically works (at least in the U.S.). First, you should warn him that you don't feel you can write a strong enough recommendation for him to get this job, and that you're convinced he'd be better off with another recommender. You should try to explain why, so he can learn why you think this position is not a good fit for him.
If he insists that he wants to apply and has no other options for a recommendation, then you should try to write a supportive but honest letter, typically by taking your previous letter and strategically modifying it to fit this particular job. By "supportive but honest", I mean making the best case you reasonably can under the circumstances, but being honest about any weaknesses and not endorsing him in any way you'd regret. Often, instead of saying "I don't recommend hiring him" you can instead say something with enough qualifications that anyone can read between the lines.
For example, suppose an excellent teacher is applying for a job he simply doesn't have the research accomplishments for. You can send a letter that comments in
detail about his teaching and concludes by saying "Although Bob does not yet have any peer-reviewed publications, I'm confident that his work in progress will lead to a publishable paper. When combined with his excellent and enthusiastic teaching, this makes him a good candidate for a faculty position that prioritizes teaching over research." If you send this to a faculty search at a research university, they will read it as an automatic rejection, without your having to say so explicitly. (Note that if Bob actually applies to an appropriate teaching position, then you should not send this text, but rather a more enthusiastic version with fewer qualifications.)
Of course you may not want to set up an automatic rejection, but rather just to make sure the hiring committee is aware of your concerns. Another possibility is to conclude by saying "Bob is in many ways a strong candidate for this position. My one reservation is. " If the hiring committee agrees with you that this is a concern, then they will reject him, but at least you gave him a chance to find out whether this issue worries the committee.