WLAN Site Survey interference (check your noise floor)

I recently was tasked with doing a WLAN site survey in a healthcare facility.  There were two WLAN Engineers, me being one of them.  As we set up our gear down in the basement where we were staging all of our equipment, we noticed some rather odd results.

We started out with a Cisco 3502 with an autonomous image, both radios set to power level 11 as you might expect.  The 2.4 GHz radio set to channel one.

We loaded up our maps in the project and immediately noticed something strange:

We saw that we were sitting right next to our survey access point and seeing negative 59 or worse.  Much worse, actually.  We saw -76 as a signal strength, so we thought that there might be something wrong with the AP.  We pulled another AP out of the survey kit, also on channel one, and it produced the same result.  It had the same power and channel settings as the first one.

We started comparing laptops and coming up with other theories.  The only thing that was consistent was the signal level and both of us using the Proxim 8494 USB stick.

We decided to change the channel of the access point, and when we did, everything came back to “normal”.  If you happen to notice the dates are not in sync, that is because I had to go back and recreate some screenshots after we figured out what the problem was.

Since we were getting behind schedule, we decided to run with our survey on channel eleven and figure out the problem with the APs being on channel one later.

After we finished surveying,  I was able to break away and recreate the problem in our staging area predictably.  I also noticed that if I relocated to another wing of the facility, the problem did not exist.

After consulting a few friends (a shout out to JH & KP) I decided that I wasn’t as knowledgeable about as I thought I was in regards to how AirMagnet software worked.

I used my PCMCIA SpecAn card and set it to look at 2.4 GHz.  I was shocked when I saw the results when in our staging area:

Channel one was at nearly 100% duty cycle all the time.  Knowing this isn’t how the spectrum normally looks, I decided to walk the basement from end to end.  The diameter of the interference was about 300 feet, and only disappeared when I passed the mechanical room. 

I hopped on the elevator and went up each floor, walking the hallways with laptop in hand.

When I got to the fourth floor, this is what that signal looked like:

I came to the conclusion the interference was strongest in the immediate vicinity of our staging area.  After walking with my laptop hugging the walls, I was able to determine the source was in the clean laundry storage.  After walking through several times, I finally saw a small camera mounted above the door I entered through.  After consulting the proper employees, it was determined that it had been installed 6+ years ago by someone who no longer worked there.

This facility has been complaining about dropped VoWiFi calls for quite a long time.  Since the WLAN profile was set to 802.11a only, but a bug in the Cisco controller software allowed it to broadcast both frequencies, that explains the dropped calls.  The phones were set to either frequency, thinking that they would only be on 5GHz since the profile was configure that way.

There are several lessons learned from this task.

1.       Don’t assume your WLAN isn’t working due to a bad design.  Invest in the right tools and training so you can detect these anomalies and fix them.  This particular problem had gone on for years!

2.       If you are stumped, don’t be afraid to reach out to others in the industry.  The two people I consulted with are “theoretically” competitors, but they helped me out with their thoughts.

3.       If you don’t have any type of frequency coordination in your enterprise, it might not hurt to have a policy that gives some direction.

4.       Baselines.  What did it look like when it did work?  A routine WLAN validation with AirMagnet Survey Pro, along with some protocol and spectrum captures sure doesn’t hurt!

I still don’t understand why that the camera with 100% duty cycle affected my AirMagnet survey and analyzer results.  After cycling through our survey access points and finally wrapping my head around it, I noticed that our signal strength was -69 and the noise was -67.  Yes… the noise floor was higher than signal strength of the survey access point.

My access point was right next to the survey laptop.  The signal strength of the AP was really in the negative thirties, but it shows up as almost negative seventy.  I don’t understand why I don’t see -35 for Signal right next to -67 for noise.

Honestly, I should have paid attention to the noise floor as soon as I notice a result that I did not expect.  I also did not check to see what channels the existing access points were one.  Had I looked, I would have noticed that none of the APs were on channel one.  Hindsight is 20/20 I suppose. 

When we powered up the next survey AP and saw it was -75, I still didn’t notice the -70 noise floor.

Thinking about this experience, I do wonder how this will affect my WLAN Validations in the future.  What will my validations look like when I have a continuous wave transmitter hogging a channel?  Will it show a low coverage area?

Have you ever been down this road before and seen what I have seen?  What was the result?

Does anyone have any suggestions as to why I was seeing a -75 dBm right next to a survey access point?