What is my liability if I accidentally shot a neighbor’s dog while hunting on my own property?

Laws on this kind of thing will differ from state to state; this is a Texas answer. In Texas, you've got livestock and non-livestock laws. Obviously for some parts of the country, the protection of livestock is pretty important because it's part of the economy. And then you might have other laws protecting animals in general, as long as they're not the kind you would hunt. In Texas, you can't just shoot a livestock animal, or a non-livestock animal (like a dog), unless there are circumstances that give you a legal right. The fact that it's on your property doesn't create a right to shoot it.

Now, of course, that's not what you did--the animal wasn't any threat, and you shot it by accident. In Texas, the question would be whether there was some kind of negligence on your part. (And animals aren't like people, so depending on where you live in Texas there could be some rules on whether it's necessary for your fence to meet certain standards or else you just have to put up with animals doing what animals will do--roam free.)

Even if you should have been more careful, it's quite possible that your liability is limited to the value of the animal. For civil purposes, pets are considered property--and the pet owner can't get more than the value of the property. 

By analogy, think about your car. If your car is worth $2,000 and there's an accident, you can't get $5,000 to repair your car--you can get the value of your car. Traditionally, that's how animals have been seen too. The mostly-dead-dog's  owner can't get $1,700 for vet bills for an animal that could be "replaced" for much less (I mean, it's 11 for gods sake...not that I don't love my own dogs and hope that they healthily outlive their breed expectations). The law on this will obviously change to treat pets more like what they actually are (not people, but also not "fungible" like livestock), but this is where the law usually is right now.

Pets aren't actually like cars, but that's usually how the law works, and in this case it might be good for you (a Georgia attorney will know better about the laws in Georgia).

In some states, like Texas, a pet owner could recover money for his own mental suffering because of the loss of a pet. (Imagine that Fifi is killed horrifically in full view of her 6-year-old owner, who requires some psychiatric visits after the event...or something like that.) Of course, your neighbor isn't asking for those kinds of damages, so that's probably not a concern here.