Author Topic: Windows Server 2003 and Dual NICs  (Read 6546 times)

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Windows Server 2003 and Dual NICs
« on: January 25, 2010, 09:40:39 AM »
Okay, I'm not normally the one asking for tech advice around here, but I get stumped sometimes too, so I hope someone has some ideas, because I'm completely out.

Situation
I have a brand new server, a Dell T100, running a fresh and fully patched install of Windows Server 2003 R2 Enterprise.

Its role is to be a dedicated Websense Express server for a client.

It has two NICs, one that serves web traffic, the other that (once it's working) is supposed to serve up the "this page is blocked content" message if triggered.

However, Websense isn't the issue yet; I don't even have it configured yet, as I'm dealing with what appears to be a Windows issue of some sort.

The Top NIC has an IP address of 10.5.47.205 and a gateway of 10.5.47.133.

The Bottom NIC has an IP address of 10.5.47.206 and the same gateway, 10.5.47.133.

No DHCP, I have the addresses manually assigned.  The Bottom NIC is plugged into a port on a switch that is set up for "port mirroring", nothing special there.

And the bottom NIC will NOT receive traffic.  If I check the "status" of that network connection, it shows a handful of "sent" packets, and 0 "received".

The other NIC appears to be working fine.

Now, ordinarily, I'd be thinking this is a hardware issue, or a problem with the switch or the port or even the network cable.

But it's not.  Here's why.

This is the exact same problem I was having with the old server.

Right now I'm using a different physical server, different NICs, different network cables, and two different ports on a totally different switch, and a fresh Windows installation.

And I'm having the exact same problem that led me to end up buying a new server for this purpose.   :roll:

Granted, the old server was way underpowered for this role, so as an overall it still isn't a waste of hardware, but I'm flummoxed here.

Anyone have any ideas?  Am I missing something obvious here?
« Last Edit: January 25, 2010, 05:07:22 PM by Demosthenes »


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline Socrates

  • Wannabe Professional Blogger
  • **
  • Posts: 567
  • Coolio Points: +123/-2
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #1 on: January 25, 2010, 11:17:07 AM »
Is the bottom NIC enabled in Windows? or are the settings correct there?

That is about all I can contribute, and you probably already checked that but you didn't say so...
--this space intentionally left blank--

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #2 on: January 25, 2010, 11:35:18 AM »
Is the bottom NIC enabled in Windows? or are the settings correct there?

That is about all I can contribute, and you probably already checked that but you didn't say so...

Yep, both NICs are enabled.  Everything should be working just fine except it's not.  No idea why.  But I'm clearly missing something important here, so don't worry about suggesting something obvious.  

I should mention too, from anywhere on the LAN, I can't even ping that bottom NIC.  Attempting to ping 10.5.47.206 just results in FAIL.
« Last Edit: January 25, 2010, 05:07:45 PM by Demosthenes »


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline Socrates

  • Wannabe Professional Blogger
  • **
  • Posts: 567
  • Coolio Points: +123/-2
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #3 on: January 25, 2010, 12:53:34 PM »
Huh, maybe try flipping the NIC's in the box.  If the bottom one still doesn't work in the top slot than it's hardware, if not some sort of setting.  Or try other slots entirely.

The other thing that comes to mind is to switch the inputs into each NIC (the patch cables in the back) and see if you get the same results.
--this space intentionally left blank--

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #4 on: January 25, 2010, 01:02:30 PM »
Tried that on the old server.  It isn't hardware.  I've 100% eliminated hardware as the issue by deploying an entirely new server, with different NICs, different network cables, different switch, everything.

Switching inputs doesn't make any difference either.


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline Socrates

  • Wannabe Professional Blogger
  • **
  • Posts: 567
  • Coolio Points: +123/-2
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #5 on: January 25, 2010, 01:58:03 PM »
Tried that on the old server.  It isn't hardware.  I've 100% eliminated hardware as the issue by deploying an entirely new server, with different NICs, different network cables, different switch, everything.

Switching inputs doesn't make any difference either.

You're right that is a puzzle, a really annoying one I imagine.

Have you eliminated everything upstream from the server?  Wait you said it was a different switch....  how about a different IP address?
--this space intentionally left blank--

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #6 on: January 25, 2010, 02:01:34 PM »
I suppose I could try giving the bottom NIC a different IP.  It couldn't hurt, right?

The most effed up part of this is that the old server was working just fine configured exactly like this.

Then, one day it refused to authenticate to the domain.  I uninstalled and reinstalled the "Client for Microsoft Networks", since that can sometimes become corrupt, and that corrected the domain authentication issue, but the bottom NIC wouldn't receive any packets after that. 

I spent about 3 days troubleshooting it, the switch, the NIC, everything and finally gave up and chalked it up to something wrong with Windows.

Since the old server was barely adequate for this role, I told the client "since I need to reinstall Windows and start over with this box, now would be a good time to just build it on new hardware."

And I'm on the new hardware, but still in the same situation.  And feeling rather stupid about the whole deal.   :x



EDIT: BTW, thanks for the suggestions.  I'm at the point of "flailing around for answers" now, since I've eliminated anything rational at this point.
« Last Edit: January 25, 2010, 02:06:02 PM by Demosthenes »


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline xolik

  • King of the Geekery
  • Hacker
  • ****
  • Posts: 5159
  • Coolio Points: +540/-25
  • Gender: Male
  • HAY GUYS
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #7 on: January 25, 2010, 03:45:33 PM »
I didn't see this, but I may have missed it so forgive me if I did, but have you checked the Windows Firewall settings? I know it seems simple and obvious, but sometimes we overlook those things.
Can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie?

=-=-=-=-=-=-=-=-=-=-=
[The Fade^C Compound]
-=-=-=-=-=-=-=-=-=-=-

12AX7

  • Guest
Re: Windows Server 2003 and Dual NICs
« Reply #8 on: January 25, 2010, 04:26:01 PM »
Would you have to have Internet connection sharing off? On?

http://support.microsoft.com/kb/897616

 I have no idea; as Ive never used Windows Server software.

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #9 on: January 25, 2010, 04:56:21 PM »
I didn't see this, but I may have missed it so forgive me if I did, but have you checked the Windows Firewall settings? I know it seems simple and obvious, but sometimes we overlook those things.

Like I said to So-Crates above, I'm not offended by any pointing-out-of-the-obvious here.  I've been wracking my brains out for days trying to figure this one out, and if it's something obvious, I don't care... I just want it to work.

We have the Windows firewall turned off by Group Policy, but yes, I did double check just to be safe and it's off.

  • I also did try giving the bottom NIC a different IP
  • I tried putting the bottom NIC in a different subnet
  • I verified that internet connection sharing is turned off
  • I tried pointing the bottom NIC at a different gateway
  • I verified that both NICs are using internal DNS, not something external/public

While that bottom NIC is enabled, I can't even get the stupid server to join the domain.  It just hangs for like 10 minutes and then claims no such domain exists.  If I disable that NIC, it joins the domain no problem using my domain admin credentials.

When I re-enable it, it goes back to HURRRRRRRR OMG I CANT TALK TO ANYTHING LOL BBQ.

Xolly, you're the closest to what I do, Windows Server wise.  Am I completely fuxx0r3d here, or what?

Also, it doesn't matter which NIC is enabled, or which one is plugged into which port on the switch.  When both are enabled, one of them is always totally unable to receive packets (which I can see by Properties=>Status on the NIC.  It'll show sent packets just fine, but "received" stays at 0 and I can't ping the IP of that NIC, even though I can ping the other one).

 :?
« Last Edit: January 25, 2010, 04:58:27 PM by Demosthenes »


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #10 on: January 25, 2010, 05:30:37 PM »
Something I found in Websense's knowledgebase, buried deep under mountains of stupid.

Quote
All Websense components are installed on the same server, which has two NICs installed:
  • NIC 1 is assigned the IP address associated with the server (the Policy Server and Filtering Service IP address) and is configured to send block page information.
  • NIC 2 has no IP address (TCP/IP is unbound from the card), is configured to Monitor the traffic, and is plugged into the switch's span port. (It listens to the traffic going out the port connected to the gateway device's internal interface.)
  • Both NICs are plugged into the same core switch, which also connects to the gateway device.

Motherfuckers. 

You know, I thought the support dolts I've dealt with there were fucked when they told me I had this set up right.  It didn't make sense to me to have the 2nd NIC on one server set up with an IP address on the same subnet.

But no, they said "it shouldn't matter, everything should work just fine".

What gets me, is this.

HOW THE HELL WAS THIS EVER WORKING ON THE OLD SERVER!??

I inherited this situation; I know for a fact this was working fine on the old server, until one day when it mysteriously stopped.  When I troubleshot this on the old server, I thought it peculiar that the 2nd NIC had an IP address in the same subnet, but I figured it was some hokey Websense thing.

Now I'm really wondering how the hell this was working at all before.  Ever.


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline ivan

  • Forum Moderator
  • Hacker
  • *
  • Posts: 4925
  • Coolio Points: +497/-50
  • Not a Mod, nor a Rocker. A Mocker.
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #11 on: January 25, 2010, 05:42:59 PM »

Now I'm really wondering how the hell this was working at all before.  Ever.

This happens to me a lot. I always have to resist the urge to spend hours figuring out how something that had to be fixed managed to work before it was fixed. And the paranoia... Has it occurred to you that someone snuck into the server room and started randomly setting IP addys just to fuck with you?

I wouldn't put it past them.



"I TYPE 120 WORDS PER MINUTE, BUT IT'S IN MY OWN LANGUAGE!"  -Detta

xolik: WHERE IS OBAMA'S GIFT CERTIFICATE?
Demosthenes: Is that from the gifters movement?


Detta: Crappy old shorts and a tank top.  This is how I dress for work. Because my job is to get puked on.
Demosthenes: So is mine.  I work in IT.


bananaskittles: The world is 4chan and God is a troll.

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #12 on: January 25, 2010, 05:47:55 PM »
I honestly don't know.  With both NICs on the same server assigned IP addresses in the same subnet, you can't even get the fucking server on the DOMAIN, much less have it operate as a functional web filter.

It's what a Nuclear Power School instructor I used to have used to refer to as a Left-Handed Football Bat.



Also: There's nobody that could have done that at the client site.  The room's locked all the time, plus, nobody but me has the credentials to get onto the server in question.  They take security very seriously.

And so do I.  Which is why I reset all the admin-level passwords on their domain shortly after I took over for my predecessor.

And it's not like that bottom NIC could just spontaneously acquire a static IP address of 10.5.47.206.  That has to be manually assigned to it.

In short, it's impossible that this ever worked configured in that way.  And yet it clearly did.


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline xolik

  • King of the Geekery
  • Hacker
  • ****
  • Posts: 5159
  • Coolio Points: +540/-25
  • Gender: Male
  • HAY GUYS
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #13 on: January 25, 2010, 08:57:01 PM »
Oh jeeze, I've never done this kind of thing before. Mostly we have dual NICs on our servers then just team the mofos to get them working better. Haven't had an occasion to split them up for separate tasks like that before.

I'm gonna ponder this for a bit and see if I can come up with anything helpful.

And turning off firewall via GPO is a GREAT idea. So many issues are caused because we make idiots admins on production servers that get the bright idea the turn on the windows firewall then wonder why nothing works afterwards. We've got a Datacenter firewall in place. You don't need it on your damn servers.
Can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie?

=-=-=-=-=-=-=-=-=-=-=
[The Fade^C Compound]
-=-=-=-=-=-=-=-=-=-=-

Offline Chris

  • Administrator
  • Hacker
  • *
  • Posts: 3780
  • Coolio Points: +277/-8
  • Gender: Male
  • IT'S A TARP
    • View Profile
    • The Geekery
Re: Windows Server 2003 and Dual NICs
« Reply #14 on: January 26, 2010, 06:37:52 AM »
From what I gather, you're having trouble getting the server to respond to two different IP addresses...

Quote
No no no no, you don't.

I was working on the same thing just recently, and figured out how to do it in windows server 2003. And it's working just fine.
Make sure you either have or can install Routing & Remote Access (should be in Administrative Tools)
If you need to install it, just do so with basic NAT/Firewall checked (one of the options required to install RRAS - Routing and Remote Access Service), but we're not going to use NAT.

Remove your gateways from your adapters (by going to your NIC->TCP/IP Properties->Advanced->Gateways->Remove (hopefully you can get my drift))

In the RRAS snap-in, right click on Static Routes and add a new route.
(I'm going to assume that your LAN connections are named "Network Card 1" and "Network Card 2" respectively for ease of typing)
Set this up:
Interface: Network Card 1
Destination: 0.0.0.0
Network mask: 255.255.255.255
Gateway: 192.168.0.1
Metric: 1

Click ok, and right-click "Static routes" and add another new route:
Set this up:
Interface: Network Card 2
Destination: 0.0.0.0
Network mask: 255.255.255.255
Gateway: 192.168.0.2
Metric: 1

Click "OK" and then right-click on "static routes" and click "show ip table"
make a note/screenshot of it as it is now

Go back to your network adapter properties, go to the "Network Card 1" properties
Go to TCP/IP Properties->Advanced->Gateways->"Add"
Gateway: 192.168.0.1
Automatic Metric: UNCHECKED
Metric: 20
Close out of that

go to the "Network Card 2" properties
Go to TCP/IP Properties->Advanced->Gateways->"Add"
Gateway: 192.168.0.2
Automatic Metric: UNCHECKED
Metric: 20

Now get a new list of the routes in your IP routing table by going into the RRAS snap-in, right click on "static routes" and click "show ip routing table"
You should have at the top, something like this (the first 4 lines are the critical ones, if these aren't right, re-try the order in which you setup the static routes & add the gateways)

Destination Network Mask Gateway Interface Metric Protocol
0.0.0.0 255.255.255.255 192.168.0.1 Network Card 1 1 Static (non...)
0.0.0.0 255.255.255.255 192.168.0.2 Network Card 2 1 Static (non...)
0.0.0.0 0.0.0.0 192.168.0.1 Network Card 1 20 Network Mgmt
0.0.0.0 0.0.0.0 192.168.0.2 Network Card 2 20 Network Mgmt

Be sure to make your destination NAT on your routers MATCH from router to IP resepectively, or the whole thing won't work, ie:
router: 192.168.0.1 needs to forward traffic to 192.168.0.5
router: 192.168.0.2 needs to forward traffic to 192.168.0.4

Essentially what you're doing is bonding a gateway to a NIC. This may work for you, it has definately worked for me when I needed to setup a web server that would respond on 2 different public IP's.

-Some

Offline ivan

  • Forum Moderator
  • Hacker
  • *
  • Posts: 4925
  • Coolio Points: +497/-50
  • Not a Mod, nor a Rocker. A Mocker.
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #15 on: January 26, 2010, 04:30:20 PM »
Guys, Demosthenes found the problem. The second NIC needs a zero IP addy for Websense to use it as a sniffer.
"I TYPE 120 WORDS PER MINUTE, BUT IT'S IN MY OWN LANGUAGE!"  -Detta

xolik: WHERE IS OBAMA'S GIFT CERTIFICATE?
Demosthenes: Is that from the gifters movement?


Detta: Crappy old shorts and a tank top.  This is how I dress for work. Because my job is to get puked on.
Demosthenes: So is mine.  I work in IT.


bananaskittles: The world is 4chan and God is a troll.

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #16 on: February 09, 2010, 09:58:59 AM »
Actually the problem was mostly because I had a fundamental lack of knowledge on how Websense is supposed to work, networking-wise, and I made the mistake of listening to the Websense "experts".

They had me set this up with NIC #1 plugged into one port on a switch and NIC #2 plugged into a port that was a mirror of the first one.

Which effectively makes a loop.  Needless to say, this causes havoc in Windows' TCP/IP stack.

Five different Websense techs looked at how this was set up and said "yep, this should work just fine, I don't understand why we're not seeing any traffic when other machines surf the web on this network."

 :roll:

The solution?

Simple, actually.  The FIREWALL needed to be plugged into one port, and NIC #1 needed to be plugged into a port on the same switch that mirrored the port into which the firewall is plugged into --- duh.  You know, so that the Websense server can actually monitor web traffic passing through the firewall?

NIC #2 can be plugged into literally any ordinary port on any switch on the network.  All NIC #2 is supposed to do is serve up the "hay stupid, this site is blocked" page when they go to something picked up by the "monitoring" NIC.

I wasted literally weeks trying to make this work via their goddamned instructions.  This would have been relatively easy to accomplish if one of them -- even one of them -- had bothered to explain to me how this was supposed to work to begin with.

Fuckers.



Anyway, thanks for the suggestions, you guys.  They didn't help, but it certainly didn't hurt.  They were better than anything the idiots at Websense were suggesting, anyway.


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"

Offline xolik

  • King of the Geekery
  • Hacker
  • ****
  • Posts: 5159
  • Coolio Points: +540/-25
  • Gender: Male
  • HAY GUYS
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #17 on: February 09, 2010, 01:43:24 PM »
Don't you love it when your company pays a fuckton of money to consultants that don't even know their own goddamn product?





Citrix, I'm looking in your direction....
Can you imagine an imaginary menagerie manager imagining managing an imaginary menagerie?

=-=-=-=-=-=-=-=-=-=-=
[The Fade^C Compound]
-=-=-=-=-=-=-=-=-=-=-

Offline Demosthenes

  • Evil Ex-Hackernetwork Moderator
  • Administrator
  • Hacker
  • *
  • Posts: 9775
  • Coolio Points: +557/-72
  • Gender: Male
    • View Profile
Re: Windows Server 2003 and Dual NICs
« Reply #18 on: February 09, 2010, 01:54:02 PM »
Worse still, every time I said something along the lines of "Are you SURE this is supposed to be set up this way? I don't understand how this could possibly work..." they always assured me that yes, indeed, it is supposed to be set up that way.


Coolio Points: 89,000,998,776,554,211,222
Detta Puzzle Points: 45

"Why is the cork on the fork?"