Hitler didn't really go after Africa, his interest in the area was minimal at best.
The German forces under Rommel were only sent to the area because the Italians were about to lose their entire African empire to the British, if they Itatlians had been able to defend Libya and take Egypt on their own I doubt any German forces would have been sent to Africa.