For detailed exegesis, a good commentary is the best resource. There are decent one-volume commentaries, but the commentaries on individual books have enough space to cite the various scholar's opinions. That's helpful, since you may not just want to take the author's opinion. The best commentaries use the original language as the base, though they almost always give an English translation, are can be used by someone who only knows English. Most university libraries will have commentary sections. Buying a full set of commentaries is fairly expensive, though if your interests focus on specific books you could get just a few. If you prefer online reading, the major commentaries are available from Logos and other online programs (though they won't sell individual books for some of the series).
However to do exegesis you need more than the details of the specific passage. You need an overview of what the Bible as a whole, and particularly what that book and author are doing. At a minimum I'd read an introduction to the OT or NT. There are a number of good introductions. Which one to choose depends upon whether you accept or reject current scholarship, and your specific interests. Since inerrancy tends to be an American speciality, I'm going to guess you'd want to use critical scholarship.
The only current NT introduction I know is Ehrman (because I had a student who was interested specifically in the kinds of questions Ehrman deals with well; but it also seems to be the most commonly used in university NT courses). It is available in the Internet Archives, somewhat to my surprise. The problem with it is that it's strong on the historical background of the NT, but not so good on giving a synthesis of work on the historical Jesus, and other things that you really want to do exegesis. Of the books that show up in Amazon, the one by Paul Achtemeier, Joel Green, and Marianne Meye Thompson looks safe. I'd make a different recommendation for someone with conservative views, although the reviews for that book in Amazon point you to the alternatives. It looks like it would be reasonable for someone with no academic background in theology, though it's still a substantial book.
I'd hesitate to recommend any OT intro. I know less about current OT scholarship.
You can also find interesting things online. A source I find particularly useful in the NT Wright page,
www.ntwrightpage.com. earlychristianwritings.com has online references for most of the books, but (1) half their links are broken; and (2) the quality is variable. The highest quality things they refer to are by Wallace, but he may be more conservative than you want, though he's a good scholar. bible.org, which houses most of Wallace's online stuff, has a large library of Biblical work, but be aware that they are a moderately conservative source.
If you like classical sources, Calvin's Institutes of the Christian Religion was intended as an introduction to theology, to give people the background to do exegesis. It is online, as are his commentaries. See ccel.org. (Luther, unfortunately, wasn't as good an exegete, though he was a more creative theologian.) Scholars still refer to Calvin's commentaries, although not for technical issues.
). I'm not sure if I believe in what you call biblical unerrancy - there's things I don't take as literally as others (I have some doubts about dating things in history based on the Bible for example) - but I believe that the events described there in essence are literally true and I try not to pick and choose what I want to believe from the Bible. Considering all that I'd appreciate a rather conservative source over a recent one. Well things don't necessarily need to be old, but I'd appreciate them to stick to the classical approach to christianity.
