Well for some reason Telepacific decided that they needed to come out to a place where I consult and swap out their T1 router and replace it with another brand. Something about since the hardware is running fine it's at its "end of life" and has to be replaced. Whatever.

 

Apparently they got into it and checked over some of the configs to copy to the new box. But not all of the configs. They managed to set the LAN side of their router back to the private IP it was using, 10.6.1.1. But they didn't put back in the static route that routes the subnet over to my router, 10.6.1.2.

 

So I get on the phone with them and explain to them that they need to turn off all the NAT and delete all the addresses from my subnet off their router and put back the static route. They put back in the simple static route but yet it still wouldn't work. I find out that the address I'm seeing from outside as the last hop is NOT their premise router, but one of the higher up core routers that the premise router talks to. So they decide it's an issue internal and they need to get the next up guy to do some tweaking and let me know they'll call me back.

 

An hour later the tech calls me back, and he's managed to get the routing going all the way to the premise router, but still won't get over to my router. This time he's got a tech from the company that makes the premise router on the phone to help him out. I explain again to everyone how they just need to put a static route in the router and everything will be ok. The vendor rep argues with me that it won't work and that the LAN interface has to have a public IP too in addition to the WAN interface and that I can't have 2 routers talk to each other using a /30 private IP network. He keeps trying to assign one of my subnet IPs to the router so that he can turn off NAT. I tell him that everything is fine, it's ok to put the static route in, my router will be using one of the public addresses and I set it up so whenever it sends traffic out to the internet it'll be using a public address as the source address. Only my router and the premise router need to be concerned about the private /30 and everything will be ok.

 

In the middle of this I decide to do yet another traceroute and finally I see the premise router in the hops, and the route is bouncing back and forth between the premise router and the core router. Yay! We have SOME progress here, the route is hitting the premise router! So I let them know they are doing good and they just need to stick in the static route and we'll be all set. The ISP tech pops it in, I run my traceroute, looks good, I traceroute to another IP from the subnet, looks good. I go ahead and try an ssh to make sure I'm actually talking to one of my computers, and FINALLY, I'm in!

 

I let the ISP tech know everything is good and to lock down the config and I'm about to hang up when the vendor tech comes back on. Apparently he had put us on hold to go get HIS supervisor. The ISP tech tells the vendor guy that everything is fine and it's working now. So the vendor guy is like "What?!? It can't work! Let me see what you did!" He gets in there and starts looking at it and you hear him and his supervisor arguing and saying that it can't be like this and it can't be like that and all of a sudden IT STOPS WORKING!

 

The ISP tech says he didn't change anything and the vendor guys are both playing dumb insisting they didn't do anything. The ISP guy gets back in and says "Well they deleted the IP off the LAN interface!" GAH! Stupid vendor! Leave it alone if it works! Just cuz you don't understand doesn't mean it won't work and you should putz with it! The ISP tech puts everything back and now FINALLY, after being down since last night at 17:56 connectivity has been restored.

 

Maybe I should be doing routing consulting. Anyone wanna hire a guy that knows how to do complex routing?