They both have very unique uses and advantages. The knife is much better for close-range stealth kills, whereas the harpoon isnt exactly subtle but is much more geared towards quick, medium-range kills (not to mention it can travel through walls)
Edit: also while I think it would be cool if knife insta-killed, the two hit kill encourages backstabbing