The words of God came to Israel through the prophet Moses. At that time, YHWH spoke His words (Torah), but later He used many prophets to speak His words.
Jeremiah 29:19 Because they have not hearkened to my words, saith YHWH, which I sent unto them by my servants the prophets, rising up early and sending them; but ye would not hear, saith YHWH.
Hebrews 1:1 God, who at sundry times and in divers manners spake in time past unto the fathers by the prophets,
The words YHWH spoke through the prophets are also the "Words of God". By using "Torah" in Matthew 4:4, you exclude all of YHWH's words through His prophets. Should we not live by the words of the prophet Micah?
Micah 6:8 He hath showed thee, O man, what is good; and what doth YHWH require of thee, but to do justly, and to love mercy, and to walk humbly with thy God?
So, when YHWH instructs us not to add to or take away from the Torah (Deuteronomy 4:2), what does that mean?
In addition, Psalm 19:7 says that the Torah is perfect. The Hebrew word that translates to "perfect" is "tamiym". Tamiym also means "complete, whole, sound, entire, without blemish". In other words, the Torah is complete - it is not to be altered.
There is also something that needs to be cleared up: Yes, YHWH communicated through multiple prophets and not just Moses, who He used to communicate the Torah. But, not all of YHWH's words carry the same weight. Here are three examples:
First, only breaking the Torah is sinful (1 John 3:4). There is no mention of disobeying YHWH's words through other prophets as being sinful - although it would be very wise to listen to them.
Second, YHWH gives us a way to test for false prophets in Deuteronomy 13:1-5. Even if the prophet comes performing signs and miracles, if they tell you to serve other gods (meaning to obey gods who have different torah), then that prophet is false. Therefore, for prophets after Moses to be true, they couldn't contradict the Torah - which includes Deuteronomy 4:2 in that they couldn't alter the Torah.
Third, YHWH says in Jeremiah 6:19 "Hear, you earth: I am bringing disaster on this people, the fruit of their schemes, because they have 1) not listened to My words and 2) have rejected my Torah." I added those numbers because I want you to see that in YHWH's eyes the Torah is a special set of His words. If the Torah was the same as the rest of His words, He would have just said "they did not listen to My words". But, instead you see Him making a very clear distinction between the Torah and the rest of His words.
The prophets (not to belittle them) merely told Israel and Judah to repent, turn back to following Torah, and prophesied future events. In order to be authentic prophets per Deuteronomy 13:1-5, they couldn't have altered Torah. In short, the Torah is different from the rest of YHWH's words in that it's what all prophecy must agree with/being founded on and only by breaking Torah do we sin.