If you pay off the loan early, you can deduct the remaining points in the year you paid off the loan. If you deducted $500 in points over five years then refinanced again, you could deduct the remaining $2500 in points that year.
However, this does not hold true if you refinance the loan with the same lender. In that case, you just keep taking $100/year in deductions for the old loan for as long as you hold the new loan.
So the weird question: what happens if a bank got swallowed up by another bank? If I'm holding a WaMu loan and refinance it with Chase this year, do I get to accelerate the points from the old loan or do I have to keep amortizing them? For tax purposes, is the swallowing bank considered to be the same entity as the bank that it ate?
I know, I should call the IRS and ask this one, but I'm curious to know if anyone has the answer.