Jade Tigress, there's nothing wrong with not having an instant affection for or interest in infants, but your reaction, which you say borders on violence at times, seems a bit excessive.
Your info says you are only twenty, and you mentioned being the youngest in your family, and having little close-up experience with babies until currently.
May I suggest that your severe reactions may partly be influenced by feelings of stress and anxiety. Looking after a helpless baby, and with little prior experience, is a big responsibility, and I know you understand that. When you can't even feel any connection with the infant, that makes it even more difficult to give the child the care it deserves and needs. I know intellectually, you are aware of the importance of not making mistakes or neglecting or even leaving the baby alone, but that knowledge likely just increases your feelings of anxiety and irritability. It's possible as well that you are over-responding to the signals that infants send, their scent, the cries and other sounds they make, which normally excite a protective response from adults, but which you may be over-stimulated by, thus making you uncomfortable.
It might help if you can use whatever means you normally do to calm yourself (meditate, sing, dance, do yoga, whatever) before your babysitting duties kick in.
I've never had an affinity for or attraction to infants myself. I worked for over twenty-five years with children over the age of two. I found that as soon as children begin learning language, they become, to me, delightful and interesting, and I become fiercely protective of them and wish to do all I can to keep them healthy and happy and learning.
And I have looked after infants, many times, and though I don't find myself getting maudlin over them, it doesn't bother me at all, partly because I know what to do and how to care for them properly (something else where your lack of experience may be causing you stress), and partly because I pay close attention to any infant I'm caring for in order to see those earliest attempts at communicating, which are amazing and important and exciting to recognize.
However, I never experienced the desire for children that other women do, and consequently never allowed myself to become pregnant. Not every woman wants or needs to be a mother. Nevertheless, it is (IMO) important to understand and be able to communicate with and care about children, and be at ease around them. I am a terrific auntie, according to my siblings kids, and I dearly love them, and that is a very satisfying non-mother position fro me.
Good luck, JT, and...er...chill out, try to relax with the kid a bit! The good thing is, the baby's not yours, you get to give it back! ;-)