Carlos Vega and John Richo, Culver City staff, returned Chair Burtners call August 25, 2004. They have installed or will be installing a total of four Firetide radios in their Town Center area around Main and Culver. A DSL line is being used to connect the network to the Internet. They used redevelopment funds (approximately $18,000) for the project and are making it free to the public. The intention is to increase traffic in the redevelopment area. Some businesses, especially one small hotel, are planning to take the signal indoors for their customers. Culver City does not have plans at this time to extend the network. They want to wait to see whether WiMax will be a better technology to provide service in other areas of Culver City.
Carlos and John stated that Firetide is being very aggressive in promoting their radios and have been willing to do what it takes to satisfy them. Firetide was willing to set up the network to demonstrate its performance before receiving a commitment from the city. Similarly, Joseph Hsieh, Wireless Hotspot, told me this morning that Firetide would be willing to set up a demo in downtown Fullerton.
The Culver City network is live and operating, although it has yet to be announced to the public. One needs simply to register online to begin to make use of it. Carlos and John offered to show us the control system at city hall that they are using to monitor the system and control available bandwidth. City staff will maintain and operate the system. They have been very pleased with the responsiveness of Wireless Hotspot to their needs and desires. They changed the design of the system several times and Vernier incorporated several suggestions in the control software that they made.
Norm Thorn briefly visited Culver City to experience how the network was functioning. When he arrived he discovered several hotspots in the area, but had trouble determining which was the city sponsored network because it was not yet clearly identified as such. Once he found the site he went through the log-in process and noted several Java script errors. The log-in requires that the user provide a name, zip code, and e-mail address and to set up a user ID and password. As soon as the process is complete, one can log-in with the user ID and password alone.
Norm found the network to be slow, about equivalent to a dialup connection in accessing web pages. He found only two wireless access points with the networks SSID, although he drove in four directions until he lost the signal. Consequently, the Culver City network is not large enough to be used as a good test of a wireless mesh network.
Chair Burtner has received a response from Firetide to the critique of their technology made by Jack Unger. A copy of the critique and response is attached.
Discussion Regarding Wireless Hotspot/Firetide Proposal
Members of the Technology Working Group spent considerable time discussing Norms experience with the Culver City network, the nature and degree of Wireless/Hotspot/ Firetides commitment to the Fullerton installation, and Firetides response to Jack Ungers critique. This discussion was recorded and has been transcribed. Helen Hall pointed out that no vendor can answer every question, that the committee should determine which vendor has been more responsive, met all of the criteria in the Statement of Work, and adequately answered questions. She noted that the City has contractual expectations on performance and the vendor will have to live up to those expectations. The committee needs to decide which vendor it believes fits the Citys needs and move on. Helen noted that the next step after recommending a vendor would be to draft a contract.
Responsibility of Technology Working Group During Network Installation
Discussion then centered on how deeply involved the Technology Working Group should be involved in the implementation of the network. Helen stated that she would probably be responsible for the actual deployment of the mesh network, but suggested that the committee might be responsible for additional tasks such as marketing and working with the downtown businesses. Chair Burtner questioned how much the committee should be involved with these aspects of the project once the advisory function of the committee has been met by selecting a vendor. City staff, especially redevelopment staff such as Kay Miller, are the ones who are most capable of working with the downtown businesses to market the network and insure its successful usage by the public. Committee members are willing to assist with the drafting of the contract, the design of the portal, and assist with other phases of the implementation, but not take a leadership role. The committee is due to sunset at the end of December 2004.
Selection of Vendor
Chair Burtner requested that the discussion return to consideration of the selection of a vendor, either CDCE or Wireless/Hotspot, with which to draft a contract. It had been agreed at the previous meeting of the committee that a decision would be made at this meeting. Helen Hall pointed out that if the committee requested Wireless Hotspot to perform a demo or test, the committee would be required to invite CDCE to do the same. Johnson Lew expressed concern regarding the unsatisfactory nature of Norm Thorns experience with the Culver City network and suggested that the committee needed to get some additional answers. Helen Hall pointed out that the committee can recommend a vendor and, before signing the contract, seek additional information as has been requested for the Culver City installation. If the committee is not satisfied, the City does not have to sign the contract. Paul Stover moved that the committee designate Wireless Hotspot as the final vendor selection to deploy the wireless mesh network in downtown Fullerton. Nilo Niccolai seconded the motion. The motion passed 4:2.
Paul Stover pointed out that the committee would need to select a portal name and that it is relatively hard to find names that are not already in use. He provided two possibilities: wirelessfullerton.com or myfullerton.com. If the committee wanted something catchier, members should give it some thought and provide suggestions to Chair Burtner as soon as possible.
Helen Hall will begin working on the contract with Wireless Hotspot.
|To: Technology Working Group |
|From: Joseph Hsieh|
|Date: August 26, 2004|
|Subject: Response to Firetide Critique and Proposal Addendums|
This memo and attached document responds to Roger Burtner, Chair of Technology Working Group, request for clarification based on Mr. Jack Ungers critique of our initial proposal.
Wireless Hotspot, Inc along with its affiliated vendors and sub-contractors are willing to commit as part of our proposal additional options including but not limited to deploy Firetides upcoming tri-mode (80211.a/b/g), two-frequency mesh routers.
We propose the following options to be taken in consideration to determine the winning bidder for the Downtown Fullerton Wireless Project:
- Firetide and Wireless Hotspot, Inc will furnish additional specification of the upcoming tri-mode, dual frequency Hotpoint 1500 (tentative name) mesh routers to address some deployment concerns
- Wireless Hotspot, Inc will do whatever possible within its reasonable resources to ensure successful deployment to fulfill the requirements of the Statement of Works that will be mutually defined at the time of contract
- Firetide has invited any members of the Technology Working Group to participate in a demonstration of the upcoming tri-mode, dual frequency Hotpoint 1500 as they become available in Los Gatos, California
- Firetide will provide the necessary means to ensure a successful deployment, including but not limited to dispatching Firetide technical representatives during deployment timeline.
- Firetide will adopt the Fullerton project as a showcase example by putting marketing resources and attention behind the project.
- Wireless Hotspot, Inc and Firetide has agreed to create a cost-effective migration strategy should the Hotpoint 1000R be deployed prior to the Hotpoint 1500 release that would allow the City of Fullerton to upgrade the 1000R to the trimode, dual frequency system without reinvesting in the infrastructure.
On behalf the Wireless Hotspot, Inc., Firetide, and Vernier Networks we would like to extend our gratitude to the careful attention of the member of the Technology Working Group. We await your decision with optimism!
Wireless Hotspot, Inc
The following reiterates the critique provided by Mr. Burtner. Some comments are inline while others are responded in subsequent paragraphs.
1. According to all the information on Firetide, their routers appear to contain one single radio that works on 2.4GHz. This configuration has two serious throughput-reducing consequences:
a) Every wireless hop through the backbone mesh will result in halving of the throughput because two wireless timeslots are required to forward each packet onward. Every wasted (or double use of a) timeslot reduces the throughput by one half.
b) Unless the Firetide units are completely obstructed by obstacles from the Firetide units on adjacent streets (which significantly degrades the efficiency of the mesh) then transmissions from all the Firetide routers will, in effect, form one large wireless collision domain. Under moderate to heavy use, the multiple collisions will significantly reduce the backbone throughtput. As if these two throughput-reducing effects weren't bad enough, there is an additional throughput-reducing consequence caused by the failure of this Firetide/YDI equipment combination to use different frequency bands, and thus separate the wireless backbone packets from the wireless access point packets.
The Firetide radios are using one of the three (Channel 1, 6, and 11) non-overlapping 802.11b frequencies. This leaves only two 802.11b frequencies for use by the YDI access points to connect to end users. The YDI access points use directional antennas which is beneficial in focusing the transmitter and receiver coverage down the streets where the end-users will be located, however, even given these focused wireless beams, two 802.11b frequencies are not enough to prevent the YDI access point beams from "crossing" each other. At several street intersections, high-power same-frequency beams from YDI units will mutually interfere with each other. This interference will significantly degrade available end-user throughput.
The combination of the Firetide self-interference and throughput reduction on 1 channel and the YDI self-interference on the remaining two 802.11b channels will result in a network that will become dreadfully slow even under relatively moderate traffic loads. Finally, the throughput testing proposed by Wireless Hotspots sound woefully inadequate to verify (or fail to verify) the slow, interference-laden network performance that this network will provide. Wireless Hotspots proposes to test the network by: i."Run applications that will determine the latency of the overall network". This testing will likely not traffic-load the network enough to reveal the slow throughput under real-world usage traffic loading. ii. "Have a city official walk through the wireless network to verify that connectivity is good". This test will result in verification that connectivity is good but will miss the fact that throughput under real-world loading will be poor.
In conclusion, while the rest of Wireless Hotspot's proposal looks good, at the wireless, ISO Layer 1 physical layer alone, the network will never be able to deliver satisfactory throughput when used by more than a handful of users. In short, the network will not only NOT scale, it will not even deliver satisfactory performance for an initial quantity of Internet-access users.
The throughput problems could be significantly mitigated if Firetide could supply a 5 GHz wireless mesh router utilizing two 5 GHz radios operating on different 5 GHz frequencies. The backbone wireless packets would then be isolated from the end-user wireless packets and the overall network would experience a very substantial increase in the overall end-user throughput that it could provide. Also, the experience of losing half of the backbone throughtput at each backbone-node hop would be eliminated.
Firetide Response To Critique
The two main critiques are those presented on page 1;
- Bandwidth reduction of 1/2 per hop.
- Self-Interference of Mesh.
Starting with (b) -- Ethernet at its origins was also a single unified collision domain. Each and every user of Ethernet was connected to the same medium, usually a coaxial cable. Protocols were developed to allow for all of those users to coexist and use the same medium happily without undue degradation of performance. The air-space or RF frequencies shared by 802.11b products are separated into a few specific channels and on each of those channels the wireless network devices must employ protocols designed to minimize and degradation of performance from its use. Given the fact that we all know that Ethernet works, and that Ethernet was actually developed in May 1973 at Xerox PARC designed by Robert Metcalfe who had expanded the ALOHA packet radio concepts and applied them to cable. We know that the shared collision domain is a non-issue when properly addressed by the network devices.
The challenge is how each individual device is design to access those channels because there is no coordination among elements. No matter what solution presented by any vendor, there is no way to guarantee that all of the devices will "play well together". There are always, with any solution, risks of interference by devices that have nothing to do with the proposed network.
So the best practice in such situations is to worry about oneself first and foremost and follow two very important rules. Be disciplined and be a good neighbor. Firetide follows both of those principals closely.
Our method for channel access is based on statistical properties designed to optimize our usage and minimize the chance that anyone else can interfere with the transmission. Based on what/when we hear others using the channel we make our decision to access the channel, and when we do send, we attempt to be on and off the channel as fast as possible in order to minimize the time we expose ourselves to interference.
Once again, other more complicated methods exist, but only work if absolutely everyone in the area implement those methods, something that cannot be guaranteed when using unlicensed radios like Wi-Fi so Firetide decided not to try to build-in a complicated scenario to absolutely guarantee absence of self-interference because there are other radios around that are not following that same discipline and behaving the way we would want.
The critique mentions that it would be a good idea to segment specific links based on channel usage. However, in a situation where a problem arises, it would take several tries to get agreement among peers which channel to move to. Net effect, you take more time sorting out a better channel, and moving there and still have no guarantee that the new channel is clear from interference.
The critique mentioned that the use of two radios for backhaul might allow for some segmentation of portions of the mesh so some links could use the first radio, and other links the second radio. In this scenario the time it takes to move the message from one radio to the other takes so much time and effort that you would probably have been better off attempting to resend it on a busy channel one or two more times that switching to a different radio, that again may have its own interference issues.
Firetide is committed to continually improving on its product. One such improvement (timeframe not yet determined) will be the introduction of the notion of a Passive acknowledgement. That is, by virtue of hearing another mesh device forward a packet that we have sent, we know, implicitly that it was receive OK, there is no need for the device to send back an ACK to say that it received it ok. This will mean that less traffic will go between two nodes, in turn, meaning there are less synchronization problems of following each transmission by the return of a tiny ACK. Additionally, the time spent to wait for and process the passive ACK leads to a random delay in transmission that will also help avoid collisions on the next packet forwarded.
This is but one example of how Firetide can improve through software along, and help further mitigate any risk of the already low chance of self-interference.
Part (a), bandwidth.
There is much discussion and much disagreement.
Suffice it to say that Leonard (Len) Klienrock published a paper around 15-20 years ago that proves that a multi-hop radio network's bandwidth asymptotes to 1/3 of the total available bandwidth. Not 1/3 each hop, not 1/2, but one-third of the total overall. Kleinrock is arguably the world's leading authority and researcher in the field of computer network modeling, analysis and design and a father of the Internet.
New Information Regarding the Firetide/YDI Equipment Combination In my earlier review of this equipment combination, I advised that there would be significant throughput degradation due to Firetide self-interference as well as additional degradation due to self-interference between the YDI access points backhauling traffic to and from the Firetide mesh nodes. Upon further analysis of this proposed configuration, I have uncovered three more significant concerns:
The YDI AP-ANT LR is an amplified long-range unit, designed to go many miles. The use of this unit for links of only one or two blocks represents massive overkill. The high power of the YDI AP-ANT LR will virtually guarantee substantial self-interference with a consequent throughput reduction.
Wireless Hotspot Inc: YDI is end-user-to-mesh-node. In other words, YDI talks to users, Firetide talk to Firetide only. The YDI-ANT can be carefully deployed to avoid interference. Downward tilts at for each YDI antennae will reduce and mitigate interference. Testing and optimization is part of the deployment plan.
Access points normally do not communicate with other access points; access points typically communicate with end-user client radios. The only exception is when access points implement wireless distribution service (WDS) feature. The YDI spec sheets for the product do not mention WDS as a feature leading me to question whether the YDI access points will be able to successfully function to provide backhaul capabilities between access points.
Wireless Hotspot Inc: This point should be disregarded as the explained in the additional explanations below #2 Firetide nodes are NOT end-user-to-mesh-node connectivity
Most serious of all, this equipment combination specifies a quantity of from 23 to 30 YDI AP-ANT LR access points. The proposal also specifies a total of from 17 to 23 Firetide mesh routers. The ratio of YDI access points to Firetide mesh routers is equal to or greater than 1:1. This indicates a severe mesh network design flaw. Having as many backhaul YDI access points as there are Firetide mesh nodes defeats the purpose of deploying a mesh network in the first place.
Wireless Hotspot Inc: This has been addressed by our new refined proposal (with less access points and nodes. The first one was very exhaustive, the new one is more limited. We estimate about 10-12 nodes for the first phase)
These points deserve some additional explanation:
Regarding Point #1 This point still applies whether the YDI access points are used to provide end-user-to-mesh-node throughput or to provide mesh-node-to-mesh-node throughput. Either way, the high power of these units will result in self-interference with reduced throughput.
Regarding Point #2 This point only applies if the Firetide radios are used to provide end-user-to-mesh-node connectivity and the YDI radios are used to provide (or to attempt to provide) mesh-node-to-mesh-node connectivity. Restating it is not completely clear from the vendors proposal, which radio (Firetide or YDI) is going to be used to perform which function - end-user-to-mesh-node throughput or mesh-node-to-mesh-node throughput. If the vendor is proposing to use the Firetide radios for mesh-node-to-mesh-node connectivity, then Point #2 should be disregarded.
Regarding Point #3 This point states that the ratio of access points to mesh routers is too high, indicating a faulty design. This point is only valid if the vendor intends to use the Firetide radios for end-user-to-mesh-node throughput and the YDI radios for mesh-node-to-mesh-node throughput. If the vendor is instead proposing to use the Firetide radios for mesh-node-to-mesh-node throughput, and if the vendor proposes an equal number of Firetide and YDI radios (one Firetide and one YDI radio) at each node, then this 1:1 ratio does not, in itself, indicate a severe design flaw. The consequences, however, of deploying the Firetide radios for mesh-node-to-mesh-node throughput will still be a throughput-limited network, as I originally reported on June 3, 2004.
Throughput Exposures and Recommendations
Scalability is not clearly defined.
The Fullerton Wireless Network SOW specifies a network designed with the capability of being able to support many additional potential users and applicationsand a network that must be expandable and scaleable. The actual number of desired additional users is not specified. The actual amount of desired network throughput is not specified. Use of the term scaleable without addressing throughput and number of simultaneous users can lead to significant vendor misunderstanding. A vendor could define scaleable to simply mean adding users or expanding the coverage area while being oblivious to (or accepting) the consequent throughput reduction. Unless the City addresses network throughput issues clearly and completely in the SOW, the pilot network may fail to meet the throughput expectations of the City, City agencies, City businesses, other stakeholders, and City visitors.
Redefine scalability to include throughput specifications. Make clear that as additional nodes are added and that coverage is extended to new areas that Citys specified throughput requirements must continue to be met. Additionally, include quality-of-service (QoS) parameters that include the number of simultaneous end-users and the minimum amount of throughput available at different hop-distances from each mesh-fabric-to-Fullerton LAN connection point. Without measurable throughput requirements for both lightly-loaded and heavily-loaded network traffic conditions, it will not be possible to determine if the installed wireless network meets the requirements of the Fullerton SOW
Firetides Hotpoint 1000R can guarantee a true T1 experience within 2-3 hops. Currently, the mesh is designed to reach backhaul within 2-3 hops.
A two-frequency-band wireless architecture is not specified.
Specify a two-frequency-band wireless architecture that uses one 2.4 GHz radio for end-user-to-mesh-node communications and one (or more) 5 GHz radios for mesh-node-to-mesh-node communication. A two-frequency-band architecture eliminates all collisions between end-user and mesh-fabric packets and allows the network to deliver substantially more throughput than a single-band wireless network can deliver.
Firetide is planning to unveil a two-frequency wireless architecture. Depending on the timeframe we have planned in a migration upgrade if single-frequency implementation is deployed, or if time permits we can deploy two-frequency wireless mesh routers at the time of deployment.
Dynamic 2.4 GHz-band frequency agility is not specified. Any mesh architecture that fails to use all three of the non-overlapping 2.4 GHz channels will experience severely limited throughput because all of the wireless mesh nodes will be in the same wireless collision domain.
Specify a mesh network architecture that automatically segments itself into several collision sub-domains - each sub-domain using a different 2.4 GHz channel. This maximizes the chance of the network maintaining specified end-user-to-mesh-node throughput levels as interference levels and traffic loads change over time.
This will be done and part of the deployment plan. The technical explanation of the deployment has been minimized in the RFP to make it more clear and understandable.
|Interconnection points between the mesh fabric and the Fullerton wired network are not specified, nor is the available Fullerton network throughput specified. |
Specify the points where the mesh fabric will interconnect to the Fullerton wired network. Specify the throughput level that Fullertons wired network will make available to the wireless network at each interconnection point.
This was demonstrated from the deployment map.
|Acceptance testingstandards are not specified. |
Specify acceptance-testing standards that include throughput under both unloaded conditions (few user/low-bandwidth applications) and loaded conditions (many users/high bandwidth applications). See the QoS recommendations under Scalability, above.
Wireless Hotspot is retaining a third party, Frank Keeney, to address this issue. We plan to fix or facilitate any weaknesses in the deployment upon Mr. Keeneys recommendations.
Support Exposures and Recommendations
| Non-proprietary hardware is specified but only proprietary hardware is currently available. At this point in time, all mesh network vendors use proprietary hardware for mesh-node-to-mesh-node fabric connectivity. |
Budget for and require the selected vendor to provide near-real-time support to the Fullerton system administrator. Because mesh network standards do not yet exist, network support from other (outside) vendors will, in all likelihood, NOT be available. In the unlikely case that outside support is offered, it may be offered by either inexperienced or overpriced vendors.
This is part of our maintenance and support fee as outline in the proposal. We will have a reasonable turn-around time per incident that cannot be resolved remotely.
|End-user support needs are not addressed. |
Visitors and transient users may require little support however business users will likely require significant levels of support. Prioritize and address this ongoing support need as soon as possible.
This will be accomplished in two fashion
* Online and self-help support for business users built into the portal
* Technical support forum, incident and case tracking, and knowledge base can be considered to facilitate troubleshooting and end-user support.
Interference from non-network sources is not addressed, yet such interference is the norm. Interference symptoms include the inability of end-users to connect, slow network throughput, and frequent end-user disconnections.
Require vendors to be responsible for:
- Conducting an interference-reduction education program in the community.
- Locating sources of on-network and off-network interference.
- Reconfiguring the wireless network to minimize the throughput-robbing effects of interference.
There will be workshops, education opportunity, and hands-on demonstrations to inform community about frequency issues.
We will periodically check for interference and rogue access point conflicts.
We will also reconsider reconfiguring the network as necessary as part of the support and maintenance contract.
Network management and monitoring capabilities are not specified.
Require the selected vendor to design, install, and support a network management and monitoring system. Require the network management and monitoring system to collect network performance statistics so the performance of the pilot network can be analyzed, documented, and reported. Require the vendor to train Fullerton system administration personnel in the operation and use of the network management and monitoring system.
This is all possible and intrinsic to our management server provided by Vernier Networks.