Bought a house last year. The previous owners had all new windows installed two years before we bought it (we lived next door and saw the guys doing the job).
Had an energy audit done on the house, only to find out the guys who installed the windows cut corners and didn't put inuslation around the windows before installing them, hence we get a lot of drafts.
The energy audtior said the ultimate solution is to take out the windows, insulate them, and reinstall them. However, he said there's the risk of damaging the windows doing this procedure... not too mention the time and expense.

So, I'm wondering, will caulking the windows and frames on the inside and outside be a reasonable alternative, or should I just bite the bullet and go for the energy auditor's solution?

You may want to try removing the inside trim on just one side of one window and take a look-see. If there's any gap at all between the window frame and the studding, you've got a few choices... caulk the gap (if it's narrow and not too deep), lightly chink it with pieces of scrap fiberglass batt (if it's alittle wider and deeper), or my favorite, low-expansion spray foam (if you can get the little tube into the crack). Be careful with any of these techniques, so you don't cause the frame to push in, binding on the sashes when you raise/lower them.

If you can, a good solution would be to remove the window trim on the inside of the house, and inject sealing foam (available in aerosol cans) into every opening around each window frame. Let the foam harden for a few hours, trim with a sharp knife, and reinstall the trim work. This will do an amazing job for you.

What kind of windows did they install...whole new frames or "pocket replacement" windows?
If you don't have new sills and frames...they are pocket replacement windows and all that was taken off to remove them was the strip of trim right next to the new window sashs on the inside. If that is the case, simply remove that strip of trim, and get some non-expanding foam insulation and fill that gap behind there. Use some masking tape to keep foam from sticking to the finish surfaces.
You can stuff in fibreglass, but it doesn't seal as well as the foam. Be prepared to use up the whole can (s) of foam...there is probably some gap or crack somewhere in the house that can use the excess. Or find those guys that installed 'em without insulation and...

Funny I found nothing in this question pertaining to "trim"!!! It seems there are a lot of people that are under the assumption that everybody has the same windows all over the country. There`s no "cookie cutter " answer. Caulking is a good viable option. It`ll stop your drafts, but it won`t do a thing for your R value. With out knowing the aspects of the installation, it`s pretty difficult to give a viable solution. Don`t discount the energy auditor. I`m quite sure he did`nt just fall off the turnip wagon!!! If you doubt his word, ask for credentials. Some times the most difficult sounding solutions are the best and easiest.

first of all, the house never had insulation there in the first place. in the second place, the homeowners probably didn't want to repaint if the trim was removed to put the insulation in correctly. if you don't want to either, then just caulk the trim to the siding and the trim to the windows. this will create a dead air space, which is the best insulator. most good replacement windows come with foam wrap to insulate the inside of the window frame anyways. personally i think your energy auditor is just trying to blow smoke up yours, so you'd think that he was worth it. if you really want to do it right; cut the paint/caulk around the trim on the inside and then pull the trim off. then, remove the sash weights, fill the void with fiberglass cut to size, then re-install the trim. set the nails, putty the holes, caulk the trim, and then repaint it. that's why the previous owner didn't go the extra mile, especially if they were thinking of moving. good luck.

weather stripping could be an option

Check with the company that installed them and ask if the windows are covered by a warranty. If so, ask if the warranties are transferable to the new owners.
When you call, dont tell them the problem. Talk to them as if your interested in new windows and need to know abut the warranties offered with the purchase. AFTER you get the information, call them back and explain your issue to them. It would help if two people made the calls, the first to gather info and the other to explain the issues.

I agree with william v. However, if you do have trim on the inside then carefully remove it and see if there is gap between window and rough frame. Use fiberglass insulation pieces to fill the gap, do not pack it in tightly as this reduces the r value. If the gap is very small use a good caulk to fill the void. I would avoid using the foam stuff, as it can expand dramatically and cause window frame to warp. Put the trim back on and your draft should be taken care of. When the weather gets nice, carefully caulk the outside of window frames. Realize the above assumes the drafts are coming from around the window frame. If the windows are not made right, the draft could be coming from other areas of the window.

Well caulking would be a good alternative for now. At least this will help you to understand how much of a draft is getting in, and also give you a temporary fix in the middle of winter. If this does work, and it works good, I would just leave well enough alone. Just remember to recaulk every couple of years, or at least inspect the caulking every other fall.