I just had a tank taken out of the ground last year (by a company owned by a huge Rutgers fan), and it had a small leak. They pretty much all have a leak, so if you don't have insurance for the tank and remediation you're looking at serious cash out-of-pocket, even if the leak hasn't come within (IIRC) 2 feet of the water table. If it has, forget having a good life from here forward because the state will own your ass for a long time. PA is more accommodating than is NJ, but it depends on where you live and how close your neighbors are to your property. IIRC, PA is a risk-assessment state, meaning that if you have a tank that leaks they will assess the risk to your neighbors and other factors (e.g., well water), and might not make you do a huge cleanup. NJ, quite naturally, isn't as user friendly--if there's a leak, regardless of whether it poses a potential risk to anything (we're almost all on city water so there may not be a risk to anyone), they still make you get all of the contaminated soil out of the ground.
All things being equal, and if you have an adequate (1/2 inch?) gas line already coming into the house, conversion should run around $3-4k, depending on unit and the company.