Tag Archives: ftth

Louisville’s Google project failed, but it was experimental success

by Steve Blum • , , ,

Microtrench

“Have a healthy disregard for the impossible", is a quote attributed to Google co-founder Larry Page. It’s a philosophy that took Google from two Stanford grads in a garage to being, on some days, the biggest company on the planet. It’s an acknowledgement that people aren’t always – or even usually – correct when they say you can’t do something. And it’s acceptance that sometimes the experts will be right.

(N.B. “Always listen to experts. They’ll tell you what can’t be done and why. Then do it!”, with thanks to Robert Heinlein).

It’s conventional wisdom in the tech world, where failure is treated as an apprenticeship. But it’s 180-degrees from the practice of politics, where adversaries are quick to thrust spears of blame into the tiniest chink in a project plan. That’s not the case, for the most part, in Louisville, Kentucky, where Google tried to build a fiber network with an extremely shallow microtrenching technique that didn’t work. The attitude in Louisville seems to be more Silicon Valley than House of Cards.

According to a story in Gizmodo (h/t to Fred Pilot at Eldo Telecom Blog for the pointer) Google tried for months to fix things, then decided to abandon the project because the technique simply didn’t work…

Google Fiber got something out of its time here. It learned that nanotrenching—the cost-saving process of burying fiber optic cables just two inches underground—was a bust. “We currently do not have plans that call for 2 inch trenches, our primary specifications are focused on going deeper,” a Google Fiber spokesperson said in an email.

“It is such a shame to think that we wouldn’t be having any of this conversation if they would have dug their little holes two inches deeper,” [Councilman Brandon] Coan said.

Gizmodo got the headline on its story wrong, though. It wasn’t Google’s experiment in Louisville that “failed”. The company tested a hypothesis and proved it false. That’s a successful experiment. What failed was a venture where both Louisville and Google invested their reputations.

Google is none the worse for it: there’s no shortage of cities still eager to give it a go if Google ever restarts fiber construction in a big way. To its credit, Louisville’s political leadership remains upbeat about the experience, judging from the Gizmodo story. Political types will dwell on the failure and ignore the success. But whiz kids in search of a garage, and the tech investors who back them, will remember Louisville’s success.

Pai talks up rural 5G, but puts his money on 4G subsidies

by Steve Blum • , , , ,

Salinas windmill cell site

5G technology has a role to fill in rural broadband service, but it won’t be the kind of 5G that mobile carriers are hyping. That’s according to Federal Communications Commission chair (and Charlton Heston Courage Under Fire Award winner) Ajit Pai. He was speaking at rural broadband trade show in New Orleans last week.

There’s no makable business case on the horizon for densified 5G mobile networks in rural communities. AT&T dismisses rural 5G as an “infill” technology, and it and other carriers are not leaning on rural cities and counties for pole access, as they are in richer and more populated parts of California. Pai acknowledges that, but points to fixed 5G service to homes and businesses as a substitute for fiber to the premise systems…

“Contrary to what some people have suggested, I actually think 5G has a very promising future in rural America and part of the reason is, in terms of the possibilities of fixed wireless, given the fiber penetration that some of your members have,” he said. “I think the ability of rural telecom carriers to think broadly about the future of these networks and how to extend this great fiber penetration you’ve got, there’s a huge amount of promise there.”

Pai’s FCC has a mixed record on 5G fixed wireless. On the one hand, the FCC is working on opening up tremendous swaths of spectrum – in the 3.5 GHz, 4 GHz and 6 GHz bands, particularly – to support broadband service. On the other hand, the FCC and other federal agencies are spending billions of dollars to lock rural communities into fixed 4G service for generations to come.

The FCC’s Connect America Fund program is paying for AT&T’s program to replace rural copper networks with limited capacity 4G service, and supporting similar efforts by Frontier Communications. AT&T also won the contest for a national public safety network – FirstNet – that will likewise be 4G based. Pai is not putting his money where his mouth is: the 4G-based systems that the Trump administration is subsidising do not have the potential capacity of the copper networks they’re replacing, let alone substitute for fiber.

Microtrenching fail drives Google Fiber out of Louisville

by Steve Blum • , , ,

Jack rabbit 625

Google Fiber is bailing of Louisville, Kentucky because it screwed up its fiber build there. In an attempt to move quickly and save money, Google forgot the iron law of engineering:

Good, fast, cheap. Pick any two.

Google went with fast and cheap, and it turned out not so good. The problem was microtrenching, and its little brother, nanotrenching. Which particular techniques were the problem isn’t clear, but the result is. According to Google’s blog post yesterday…

We’re not living up to the high standards we set for ourselves, or the standards we’ve demonstrated in other Fiber cities. We would need to essentially rebuild our entire network in Louisville to provide the great service that Google Fiber is known for, and that’s just not the right business decision for us.

The lessons we’ve learned in Louisville have already made us better in our other Google Fiber cities. We’ve refined our micro trenching methods and are seeing good outcomes elsewhere.

But it’ll cost too much to rebuild its plant in Louisville, so it’s adios. According to a story on WDRB.com (via a link on Google’s blog post), the epoxy compound that Google was using to fill up the shallow slits it dug in streets for its fiber – that’s how microtrenching is done – failed. The fix they planned to use was to go back, scrape the epoxy out of the slits and refill them with asphalt. Reading between the lines of Google’s blog post, that technique didn’t work any better – the implication is that Google would have to rip everything out and start over again if it wanted to keep doing business in Louisville.

Google says it going to move ahead with fiber projects in other markets. The list includes Kansas City, Austin, Provo, Charlotte, Atlanta, Orange County, Salt Lake City, Raleigh-Durham, Nashville, Huntsville and San Antonio.

The list doesn’t include San Francisco, or any of the other Bay Area cities where it’s relying on Webpass, an Internet service provider – primarily wireless – it acquired in 2017.

CPUC approves ownership transfer, re-start of Nevada County FTTH project

by Steve Blum • , , ,

Bucket on pole

Updated at 10:41 with statement from Race.

The Bright Fiber FTTH project in Nevada County was resurrected this morning by the California Public Utilities Commission. In a unanimous vote (on the consent calendar, if you follow such things), the CPUC approved transferring control of Bright Fiber Network, along with a $16 million grant, to Race Telecommunications. Several people spoke for and against the project – wireless Internet service providers were against it, the Nevada County board of supervisors and the Gold Country Broadband Consortium were in favor.

After the vote, Race issued a statement that says, in part…

“Race is humbled and grateful that the Commission has moved quickly to approve this transfer of control of Bright Fiber and review the needed CASF project changes so that Race can construct this important project for the community,” said Raul Alcaraz, President of Race.  “Our team at Race has been blessed to have a successful track record of CASF projects, and we feel confident we can deliver a reliable, fast and affordable broadband system.”

Race and Bright Fiber will hold a Town Hall meeting for the community in the evening on Wednesday, January 30th to describe the amended project to interested community members and to answer any questions.  The location and time of the Town Hall will be announced shortly.

Nevada County FTTH project gets new lease on life

by Steve Blum • , , ,

Spiral event 30oct2014

Update: the CPUC unanimously approved the transfer of Bright Fiber Networks, and the $16 million CASF subsidy, to Race Telecommunications this morning.

The California Public Utilities Commission is scheduled to vote today on whether or not Race Telecommunications should be allowed to take over ownership of Bright Fiber Network, which received a $16 million subsidy from the California Advanced Services Fund (CASF) in 2015 to build an FTTH network to serve 1,900 homes near Nevada City in Nevada County.

The project was developed by Spiral Internet, a long established independent Internet service provider in California’s Gold Country. Spiral has been working to raise the necessary matching funds – $10.7 million – for the past four years, but has not been able to attract investors. If approved by the CPUC, ownership of Bright Fiber Network and rights to the CASF grant would be transferred to Race. Spiral would continue to operate its primarily DSL based ISP business.

Race brings two critical elements to the table: it has access to capital and experience building residential fiber networks in rural California. It’s received several CASF grants over the past ten years, and has what appears to be an excellent track record with the CPUC.

The project would be redesigned. Instead of laying fiber underground, as Spiral originally planned, Race would install cables on existing utility pole routes. The cost of the project to taxpayers would be $70,000 less.

Two wireless Internet service providers in the area – Smarter Broadband and ColfaxNet – filed bitter objections to the transfer of control, as they have done in the past. The draft resolution in front of the commission would reject their arguments once again.

As of last night, the resolution was on the commission’s consent agenda. Unless a commissioner asks that it be discussed and voted on separately, it’ll be approved along with several other items that are considered non-controversial in a single motion.

Tellus Venture Associates assisted Bright Fiber with preparation of its CASF grant application. I’m not a disinterested commentator. Take it for what it’s worth.

San Francisco muni FTTP project hits the rocks

by Steve Blum • , , , ,

San Francisco’s $1.9 billion plan to build a citywide fiber to the premise system is dead. At least for now. According to a story by Joshua Sabatini in the San Francisco Examiner, temporary mayor Mark Ferrell didn’t intend to file the paperwork needed to put a tax measure on the November ballot by yesterday’s deadline (h/t to everyone who sent me the link – much appreciated). There’s no indication he changed his mind and, according to the Examiner, would-be private sector partners were told to stand down…

The Office of Contract Administration sent a June 13 letter to the three bid teams informing them of the delay. “The City and County of San Francisco has decided to further consider factors essential to the success of the project prior to issuing a Request for Proposals (RFP),” the letter said. “Given the groundbreaking nature, complexity, and cost of this project, it is important that we reduce uncertainties to the extent possible prior to issuing an RFP.”

The letter continued, “In the coming months, the City intends to research a number of factors, including how market conditions and the construction environment would affect the project.”

Ferrell iced the project because a poll showed that voter approval of a tax increase “was just short of the two-thirds needed to pass”, according to the Examiner. Perhaps. It’s also relevant that Ferrell will soon hand over the mayor’s job to London Breed, who won the job in a special election earlier this month. She hasn’t said yet what she plans to do and all Ferrell can say is that he’s leaving behind a “briefing binder”. Translation: they’re not besties.

Three teams were in the running to manage, operate and, perhaps, partly fund the project. At least two were led by local Internet service providers, Monkey Brains (with the assistance of Black and Veatch, Nokia and Zayo) and Sonic.net. The third contender is list only as “FiberGateway”. There’s no obvious broadband company that goes by that, but for what it’s worth, Altice, a mid sized cable operator with a relative handful of systems in California, uses it as a product name.

California rural electric co-op gets $1.8 million to extend FTTH service

by Steve Blum • , , , ,

Another 413 homes in small, desert communities in Riverside County are getting high speed, fiber to the home service, via the Anza Electric Cooperative and a grant from the California Advanced Services Fund (CASF). The California Public Utilities Communities approved a $1.8 million subsidy – $4,300 per home, amounting to 70% of the total cost – extending an earlier CASF-funded FTTH project that reached 3,750 customers in the co-op’s core service area in the Anza Valley.

The new build covers the Pinyon community and the Santa Rosa Reservation, but it skips over Mountain Center and Garner Valley, because Frontier Communications upgraded its service in the area. It’s receiving federal money to deliver broadband service at 10 Mbps download and 1 Mbps upload speeds, and was able to demonstrate that its VDSL upgrade was delivering around 20 Mbps down/2 Mbps up, at least to some homes. That’s on the low end of what VDSL technology is capable of delivering, but it’s more than enough to make Mountain Center and Garner Valley ineligible for California subsidies.

In the communities that are eligible, Frontier relies on 1990s style DSL technology which generally runs at about 2 Mbps down and less than 1 Mbps up, where it’s available at all. It told the CPUC it wouldn’t make any more service without CASF subsidies of its own. Which it won’t get because Anza Electric Co-op moved faster.

People in communities to the east and west, though, are – or soon will be – getting access to symmetrical speeds of up to 1 Gbps. The co-op’s flagship consumer package is symmetrical 50 Mbps speeds for $49 per month, with a $25 per month, 10 Mbps down and up offer to low income households. There’s no commitment or bundling required for either package (although an unlimited domestic phone line can be had for another $20 per month).

Santa Cruz gets more fiber, more gigabit service

by Steve Blum • , , , ,

AT&T’s recent fiber to the home (FTTH) upgrades in Santa Cruz mean that Cruzio isn’t the only Internet service provider bringing gigabit class infrastructure into town (unless you have a sneaking suspicion that it’s a competitive response – in that case you can thank Cruzio for it too). U.C. Santa Cruz’s Jim Warner tracked it down…

AT&T has been working on an FTTH deployment in parts of west Santa Cruz. The work has progressed to the point where some addresses are showing availability of gigabit service in AT&T’s on-line service availability tool. When you enter a “good” address – one where gigabit service is already available – you see, among other things, a web offer for “fiber” service at 100 Mbps and 1,000 Mbps download and upload speeds (subject to the usual disclaimer: “actual customer speeds may vary and are not guaranteed”).

The 100 Mbps packages is capped at 1 terabyte a month; the gigabit package offers “unlimited data”.

An example of what FTTH looks like “on the pole” is in the picture above. The thin curved lines that appear to loop back into the new tap are not fibers. They are simply plastic retainers to keep the protective caps from falling to the ground. To be ready to serve any address, one of the taps needs to be placed on almost every pole.

It is harder to see what’s going on underground. We need to wait for details about the project to know if areas where utilities are underground (rather than on poles) will be included.

The quality of AT&T’s craftsmanship is highly variable and not all of it looks as clean as in the picture presented. So far, I’ve seen FTTH work in the area bounded by Walnut Ave., California St., Almar Ave. and King St. This is a poor way to gauge the scope of their project, though. I visited the AT&T retail store but discovered staff get no special advance information about what the company is working on.

Wireline carriers, such as AT&T and Comcast, each get one foot of vertical space on each pole for their service. AT&T has attached their new fiber network to their legacy copper network to avoid needing to completely rearrange the pole or pay for another foot of pole space.

CPUC considers FTTH upgrade subsidy for Marin County town

by Steve Blum • , , , ,

Bolinas, a coastal community of about 700 homes in Marin County, is up for a $1.9 million broadband infrastructure subsidy from the California Public Utilities Commission next week. It’s the first grant proposal submitted to, and considered by, the CPUC since assembly bill 1665 was signed into law last year by governor Jerry Brown.

AB 1665 imposed severe restrictions on how money from the California Advanced Services Fund (CASF) can be spent. It lowered California’s minimum broadband standard to 6 Mbps download/1 Mbps upload speeds – if service is available at that level, then the legislature reckons no upgrade is needed. It also fenced off areas where incumbents, mainly AT&T and Frontier Communications, are getting federal subsidies, and allowed them to block grants in other parts of their service territories at will, unless they’re getting the money themselves.

The Bolinas Gigabit Network cleared all those hurdles, at least according to the draft resolution posted by the CPUC: there’s no broadband service available at 6 Mbps down/1 Mbps up, AT&T isn’t getting federal subsidies to serve the 600 homes (out of 700) that will benefit, nor did it exercise its jus primae noctis right of first refusal and preempt the project.

It’s also an exceptional project. According to the Bolinas Community Public Utility District, it’s “one of the largest…and densest communities which remains under- and un-served for broadband in rural California”. Its population is relatively well off – $74,000 annual median household income, versus $64,000 statewide – and exurban.

The character of the community led the company behind the project – Inyo Networks – to project that 80% of homes will subscribe to fiber to the home service, costing $90 per month for a symmetrical gigabit (a household with a child in a low income school lunch program can get 25 Mbps up/down for $30 per month). Whether that take rate is plausible is a fair question, but it does underline the rare, if not unique, circumstances of the project.

(The post AB 1665 CASF rules are still being written. It’s possible that higher income areas won’t be eligible or will have a lower priority in the future. But for now, a community’s wealth, or lack thereof, is not an established criterion).

In rural areas of California, the federally subsidised carveouts that Frontier and AT&T received are scattered about in checkerboard fashion. Finding a CASF-eligible area with a population that’s sufficiently dense and affluent to support an independent upgrade, and a local company with the resources and track record to implement it is a needle-in-a-haystack job.

Needle or not, Bolinas deserves due consideration, under current standards and rules. Delivering equal service to the haystack, despite AB 1665, is the CPUC’s true challenge.

San Francisco muni FTTP short list is down to three choices

by Steve Blum • , , , ,

The City and County of San Francisco is still tight-lipped regarding details of its $2 billion fiber-to-the-premise project, but its latest cryptic update indicates that the scheduled one-on-one interviews with potential bidders are complete and the first cut was made.

Thanks to a tip from a kind reader, I checked the City’s purchasing website and found this notice, dated yesterday, 19 April 2018…

Notice of Pre-Qualified Bidders for Citywide Fiber to the Premises Network, Lit Fiber and Wi-Fi Services RFQ

The City has completed its evaluation of Citywide Fiber to the Premises Network, Lit Fiber and Wi-Fi Services RFQ. Respondent Teams that are selected and placed on the pre-qualified bidders list are not guaranteed a contract. The following Respondent Teams have been selected:

Bay City Broadband Partners
FiberGateway
Sonic Plenary SF Fiber

Protests of the Pre-Qualified Bidders for this RFQ must be received…no later than 12:00 P.M. (PST) on April 26, 2018.

Before the interviews, the City acknowledged that four groups were in the hunt. This latest list is missing Golden Gate Broadband Partners, of which no public information or even basis for speculation exists. Same story with FiberGateway, unless you count the fact that a cable operator, Altice, uses it as a brand name.

Bay City Broadband Partners is claimed by a local wireless Internet service provider, Monkey Brains, and is said to include Nokia, Zayo and Black and Veatch. As far as I know, Sonic.net hasn’t said anything publicly but it isn’t a stretch to suspect they’re leading Sonic Plenary SF Fiber.

The three remaining contenders are still a long way from getting any kind of a contract. Later this year, the City will issue a formal request for proposals, and only the three blessed “respondent teams” will be eligible to bid. Although the City indicated it would be contributing money to the project, it hasn’t said how much.