Microtrenching fail drives Google Fiber out of Louisville

by Steve Blum • , , ,

Jack rabbit 625

Google Fiber is bailing of Louisville, Kentucky because it screwed up its fiber build there. In an attempt to move quickly and save money, Google forgot the iron law of engineering:

Good, fast, cheap. Pick any two.

Google went with fast and cheap, and it turned out not so good. The problem was microtrenching, and its little brother, nanotrenching. Which particular techniques were the problem isn’t clear, but the result is. According to Google’s blog post yesterday…

We’re not living up to the high standards we set for ourselves, or the standards we’ve demonstrated in other Fiber cities. We would need to essentially rebuild our entire network in Louisville to provide the great service that Google Fiber is known for, and that’s just not the right business decision for us.

The lessons we’ve learned in Louisville have already made us better in our other Google Fiber cities. We’ve refined our micro trenching methods and are seeing good outcomes elsewhere.

But it’ll cost too much to rebuild its plant in Louisville, so it’s adios. According to a story on WDRB.com (via a link on Google’s blog post), the epoxy compound that Google was using to fill up the shallow slits it dug in streets for its fiber – that’s how microtrenching is done – failed. The fix they planned to use was to go back, scrape the epoxy out of the slits and refill them with asphalt. Reading between the lines of Google’s blog post, that technique didn’t work any better – the implication is that Google would have to rip everything out and start over again if it wanted to keep doing business in Louisville.

Google says it going to move ahead with fiber projects in other markets. The list includes Kansas City, Austin, Provo, Charlotte, Atlanta, Orange County, Salt Lake City, Raleigh-Durham, Nashville, Huntsville and San Antonio.

The list doesn’t include San Francisco, or any of the other Bay Area cities where it’s relying on Webpass, an Internet service provider – primarily wireless – it acquired in 2017.

PG&E plans faster, wider power cuts during high fire threats in 2019

by Steve Blum • , , , ,

Pacific Gas and Electric will cut off electricity more automatically, more thoroughly and over a wider area when “extreme fire risk conditions” are present. That’s one of the wildfire risk mitigation measures it promises to implement this year.

Along with five other privately owned Californian electric utilities, PG&E submitted its wildfire prevention plan to the California Public Utilities Commission yesterday. It says it will inspect more lines, cut down more trees and harden more equipment in the coming months and years, as well as aggressively turning off power when the threat of wildfires is high. The proactive power cuts will be greatly expanded, to include…

  • 25,200 miles of low voltage distribution lines, up from 7,100 miles.
  • 5,500 miles of transmission lines, up from 370 miles. Instead of limiting it to lines carrying 70 kilovolts or less, lines of up to 500 kilovolts will be cut off if necessary.
  • Potentially 5.4 million customer premises, up from 570,000 customers.
  • Areas that face an “elevated” fire threat, in addition to those that face an “extreme” one.

PG&E also says it will streamline “decision criteria to reduce the level of judgment in the criteria to the extent feasible”. In other words, reduce the opportunity for managers to dither over whether or not to cut power.

One result is predictable and entirely acceptable: more PG&E customers will complain because their power is off. That happened last year, when PG&E proactively cut power in some northern California communities in October. It’s not a huge leap of logic to suppose that the backlash made managers more reluctant to turn off the juice in November. High winds and dry conditions were present once again, and led to the Camp Fire in Butte County, which killed 86 people and destroyed the town of Paradise.

A PG&E transmission line is suspected of sparking that fire. Under the new plan, it could have been turned off – it was in a high risk area, conditions were extreme, and it was 110 kilovolts (within the new limit but over the old one) – and probably would have been if the decision had been based on automatic criteria rather than a subjective judgement call.

The plan will be reviewed by the CPUC and by the federal judge that’s supervising PG&E criminal probation. Judge William Alsup has been sharply critical of PG&E and suggested it should do many of the things proposed in the plan, although not all his suggestions were included in it.

The wildfire prevention plan notwithstanding, yesterday was not a good day for PG&E. A natural gas line exploded in San Francisco and set several buildings on fire. There were no reports of injuries. It was apparently caused when a fiber optic construction crew hit a gas line. Whenever underground construction work is done, the contractor is supposed to notify PG&E and other utilities, which are then responsible for coming out and marking where their lines are. That’s a job that PG&E is accused of shirking in the past by the CPUC. Responsibility for yesterday’s blast is yet to be determined.

Wildfire mitigation plans
Bear Valley Electric Service
Liberty Utilities
Pacific Gas and Electric
Pacificorp
San Diego Gas and Electric
Southern California Edison

T-Mobile tries to make California merger case with soft engineering and hard hype

by Steve Blum • , , , ,

Ebbc mobile broadband availability 2012

T-Mobile and Sprint claim that if they are allowed to merge, then California will see “enormous public-interest benefits”. That’s what the companies told the California Public Utilities Commission in testimony submitted as part of the regulatory review of their proposed deal. That claim is founded in large part on T-Mobile’s description of a glorious 5G future that includes download speeds of up to half a gigabit and coverage that reaches deep into the most rural areas of California.

The catch is that this wonderfulness is “projected” and not promised. Even if the infrastructure is built, T-Mobile’s president, Michael Sievert, is careful to add a footnote in small print that reminds us…

Average data rate is not equivalent to the actual user experience. The user experience will be affected by a number of variable factors, including received signal strength, location of the mobile device and base station, and whether the device is in motion, among others.

His testimony is supplemented with an impressive collection of county by county maps offered by his chief technology officer, Neville Ray, that show how much better 5G service will be if T-Mobile can scoop up Sprint. It appears that Ray is assuming that Sprint won’t grow much – his projections for Sprint’s 5G coverage area look a lot like its current 4G and below footprint.

The real problem is that this sort of modelling produces coverage predictions that far outstrip actual results. An example is shown above. It’s two mobile broadband coverage maps for the east San Francisco Bay Area posted by the CPUC in 2012. The one on the left is an aggregate of the availability reports generated by the four major mobile carriers using predictive modelling, the one on the right is based on actual mobile download tests conducted by the CPUC, and then run through the same process. The carriers claimed nearly everything was green, the color of good and great. The CPUC’s measurements showed that service in most of the region is brown to yellow, the color of, well, you get the picture.

Oral testimony in the case begins today in a CPUC courtroom in San Francisco. We can only hope that it won’t be as larded with marketing hype as T-Mobile’s and Sprint’s written statements.

Collected documents from the CPUC’s review of the proposed merger of Sprint and T-Mobile.

T-Mobile-Sprint merger gets a hard look in California this week

by Steve Blum • , , , ,

California’s review of the proposed merger of T-Mobile and Sprint goes into high gear on Wednesday. The California Public Utilities Commission will hold a hearing to allow lawyers for the two companies and the organisations that oppose the deal to cross examine experts, and others, who submitted written testimony about it. Three days have been blocked out, although it might not go that long.

The best supported and most coherent opposition to the merger comes from the CPUC’s in-house watchdog unit, the public advocates office (formerly known as the office of ratepayer advocates). It submitted lengthy arguments and a tall stack of data that shows why tipping the mobile broadband market into oligopoly status is a very bad idea

The Commission should deny the proposed transaction because of the irreparable damage to competition in the wireless market and the low-income customer markets as well as the absence of specific, measurable, and verifiable benefits to the merger. The loss of a competitive player in these markets would create significant risk of parallel conduct and higher pricing for consumers. This pricing risk is demonstrated by the two largest players in the wireless market today, AT&T and Verizon, which generally offer higher-priced plans than T-Mobile and Sprint. New T-Mobile would rival or exceed these companies in market share, creating a strong incentive for oligopolist behavior. New T-Mobile would also comprise nearly 60 percent of the wireless prepaid market that predominantly serves low-income customers, pacing excessive market power under the control of a single company and creating a virtual monopoly over these services. Because [T-Mobile and Sprint] are not under [traditional wireline telephone rate] regulation, protections cannot be implemented that are adequately enforceable and verifiable to address these risks.

Just so. The CPUC has little authority over mobile carriers, and none where mainstream consumer pricing is concerned. It can impose conditions on any approval of the merger – the public advocates office suggests some as a fall back position – but those would provide only temporary and limited protection to Californian consumers.

The best way – the only way in this case – to fix a problem is to not cause it in the first place.

Other groups with a stake in the outcome have also jumped in. The California Emerging Technology Fund (CETF) and the Greenlining Institute expressed concern about the proposed merger’s impact on low income and minority communities, but didn’t particularly object to it so long as money and other benefits were required as mitigation. It’s worth noting that CETF’s primary source of funding is money extracted from companies that have had mergers or similar consolidations under review at the CPUC, and Greenlining relies on so called intervenor compensation from the CPUC, and companies appearing before it, as a reward for raising such issues.

The Communications Workers of America also has a dog in the fight. At this point, the union wants the merger blocked, claiming it would “would result in the loss of 3,432 jobs in California”, but it also left the door open to a settlement. In the past, CWA has opposed telecoms mergers, but then flipped and supported them once its needs were met.

T-Mobile and Sprint naturally objected to all these statements, filing detailed rebuttals, and scheduling time for cross examination at this week’s hearing. I intend to write a future post about those rebuttals, but if you want to read them yourself, you can find it all here:

Collected documents from the CPUC’s review of the proposed merger of Sprint and T-Mobile.

Federal appellate judges skeptical of FCC’s net neutrality reasoning

by Steve Blum • , , ,

Network neutrality advocates faced off against the Federal Communications Commission and its telecoms industry partners in a federal appeals court in Washington, D.C. on Friday. For more than four hours, a panel of three federal judges grilled both sides as they considered whether the FCC acted “arbitrarily and capriciously” when it rolled back net neutrality rules in 2017.

The central question is whether broadband service is a simple telecommunications service – like phone service – or a value-added information service. FCC lawyer Tom Johnson insisted that the traffic management and routing technology that enables communication via the Internet – the domain name system and caching – is fundamentally different from the traffic management and routing technology that makes it possible to dial a phone number.

Judge Patricia Millett didn’t seem to buying that argument…

Millett: They both have the capability, the capability of acquiring information, receiving information, through a telephone as much as…

Johnson: Under the broader theory that these are broad statutory provisions in which we are able to make different classification decisions based on the fact that we’re talking about different services here.

Millett: That’s what I’m trying to figure out, are you talking about different capabilities between phones and if someone uses Facetime?

Johnson: Yes, your honor, we just don’t think that phone service in offering a pure transmission pathway for ordinary voice communication offers the same dynamic experience that broadband does in accessing the Internet, but in addition it also offers…

Millett: I hear you saying those words, I’m really trying to make sure I’m understanding what the difference is, because people use telephones to acquire all kinds of information…If I want to get information from my pharmacy – I’d like to have something refilled – I can call over the phone and push a bunch of buttons and eventually I’ll have a prescription refilled. I can also go on the website and type in and tell the doctor’s office I’d like a prescription refilled. But it seems to be the exact same functionality. One is voice and one is typing, but that can’t be difference.

Johnson never directly answered Millett’s question. His defence was that there was sufficient ambiguity in federal law to create an opening for the FCC to exercise its discretion as the federal government’s expert agency in these matters. Typically, federal courts defer to that – they don’t consider whether an expert agency made the best decision, but rather whether the decision reached was one of many possible and reasonable interpretations of the facts and the law.

The plain fact is that broadband is a telecommunications service. People use it to transmit and receive information between two point without changes. The give and take at Friday’s hearing indicates that the three judges understand that point well.

Listen to the hearing here.

CPUC judge wants complex pole attachment issues to be even more complicated

by Steve Blum • , , ,

Messy fiber attachments

Utility pole associations, which manage joint use of poles by electric utilities – privately and municipally owned – and telecoms companies of all sorts, should be regulated by the California Public Utilities Commission, according to a ruling by an administrative law judge (ALJ). The ruling focuses on a narrow dispute between two companies: big picture, it’s little more than advice to CPUC commissioners. But it’s bad advice.

The ruling concerns a dispute between Pacific Gas and Electric, which owns poles throughout northern California, and Crown Castle, which is an independent, competitive telecoms company that owns and leases fiber routes, and builds and operates cell sites. PG&E offered to lease space to Crown Castle on its poles, as it’s required to do by the CPUC. Crown Castle said it would prefer to buy outright a foot’s worth of space on PG&E’s poles, so it can get about its business of building fiber networks without worrying about keeping a landlord happy. AT&T, and other incumbent telephone companies, buy space on PG&E’s poles, but they buy the whole communications zone, which can be several feet of pole space. They are then responsible for leasing out attachment space by the foot to independent companies like Crown Castle, also as required by the CPUC.

Crown Castle didn’t want to pay for and be responsible for the entire communications zone on PG&E poles, and if they did, it’s not entirely clear that they would have the same obligation as AT&T and other incumbents to lease space to other telecoms companies. So Crown Castle and PG&E ended up in a stalemate. To resolve it, Crown Castle asked the CPUC to step in and arbitrate the dispute. The ruling issued yesterday by ALJ Patricia Miles says that PG&E is meeting its obligations by offering to lease attachment space on poles and Crown Castle has no particular right to buy it by the foot.

But the ruling goes one step further and says that procedures and contracts developed by joint pole associations – the Northern California Joint Pole Association (NCJPA) and the Southern California Joint Pole Committee – need to be approved in advance by the CPUC…

There can be no doubt that disputes such as the present one will arise again. For this reason, if NCJPA is going to continue to facilitate sale and purchase transactions pertaining to public utility poles among its member entities, the Commission should require NCJPA to submit (before implementation) for Commission review and approval…its agreements, forms, procedures and handbooks which concern the transfer, sale, lease, assignment, mortgage, or encumbrance of public utility poles. Such transactions, which are being handled by NCJPA on behalf of its members, are clearly within the Commission’s jurisdiction.

The ALJ’s report amounts to a suggestion for commissioners, who would have issue a formal decision requiring joint pole associations to get advance approval for pole attachment deal terms. If commissioners go down that road, it will be a lengthy and contentious process. Not all utilities that own and/or attach to poles are under CPUC jurisdiction. An association of municipal electric utilities objected to a draft version of the ALJ’s ruling, pointing out that the CPUC has no authority over them and shouldn’t try to tell them how to manage their own poles.

In the meantime, the ALJ’s ruling will be widely read and could influence similar one-on-one disputes between independent telecoms companies and electric utilities.

Yesterday’s ruling affirmed PG&E’s current practice, and in that regard offered a bit of clarity to the already horribly complex problem of how utility pole routes are shared and managed in California. But by threatening to extend CPUC oversight to joint pole associations, the ruling adds an unneeded and unwelcome layer of uncertainty.

The CPUC has ongoing proceedings that involve a number of issues related to pole access and, because of PG&E’s bankruptcy filing and past criminal conviction, federal judges will have a lot to say about it. The California legislature and civil courts are also involved because of the dozens of deaths and billions of dollars worth of damaged caused by fires started by electric lines. Now is not the time to drag even more players into this mess.

More documents involving the PG&E – Crown Castle dispute are here.

Federal judge slams PG&E and CPUC for deadly wildfires

by Steve Blum • , , , ,

A federal judge lambasted Pacific Gas and Electric’s and the California Public Utilities Commission’s wildfire prevention efforts, and the California supreme court allowed a key wildfire cost sharing decision by the CPUC to stand yesterday. That follows PG&E’s bankruptcy filing on Tuesday.

Judge William Alsup is PG&E’s probation officer. The corporation was convicted of criminal misconduct following a deadly natural gas line explosion in San Bruno in 2010, and it is accountable to Alsup for how well it’s complying with the penalties handed down, which include good behavior requirements. Alsup thinks PG&E is a danger to the public, and he doesn’t have a high opinion of the CPUC’s efforts to rein it in. According to a story in the San Jose Mercury News by Matthias Gafni and John Woolfolk, representatives from both PG&E and the CPUC tried to convince Alsup that his proposal to require PG&E to inspect more than 100,000 miles of electric lines before this summer’s fire season begins is a bad idea. He wasn’t buying any of it…

“Does a judge turn a blind eye and let PG&E continue what you’re doing, let you keep killing people?” U.S. District Judge William Alsup said inside the San Francisco courtroom. “Can’t we have electricity that is delivered safely in this state?”…

The judge also questioned the California Public Utilities Commission, the state agency charged with regulating PG&E and other investor-owned utilities.

“How did it happen so many fires occurred under your regulations?” Alsup asked a representative of the state regulator. “It sounds harsh, but that’s what the people of California deserve to know, how did that happen?”

After three intense hours, Alsup told the parties he would rule later, but the state of California did not have time to waste with another fire season approaching.

Alsup hinted he might require PG&E to use the same, aggressive power cutting tactics that San Diego Gas and Electric uses when wildfire danger is high. SDG&E began proactively de-energising lines after wildfires in 2007 that it and Cox Communications were responsible for starting.

Those fires were costly to SDG&E, and yesterday the California supreme court refused to hear its appeal of a CPUC decision to not allow it to pass on some of its liability costs – $379 million – to customers.

California’s utility costs, regulation in play as PG&E files for bankruptcy

by Steve Blum • , , , ,

Pacific Gas and Electric filed for bankruptcy protection yesterday, beginning a process that could lead to significant changes in how electricity and natural gas service is delivered in northern California, and how much it costs. It also has the potential for changing the cost sharing calculations that determine how much telecoms companies pay to share poles and conduit with PG&E.

Assuming the federal court allows the bankruptcy to go forward – not a safe assumption according to analysts quoted by Barrons – private contracts and some regulatory directives by the California Public Utilities Commission could be, um, reimagined. PG&E hinted at what might change in its press release announcing the filing

As part of the filings, PG&E also filed various motions with the Court in support of its reorganization, including requesting authorization to continue paying employee wages and providing healthcare and other benefits. In the filings, PG&E also asked for authority to continue existing customer programs, including low income support, energy efficiency and other programs supporting customer adoption of clean energy. PG&E expects the Court to act on these requests in the coming days.

Translation: if you work for us, don’t make assumptions about your paycheck or benefits package for the time being, and if you’re relying on rents extracted by the CPUC or California legislature, make contingency plans.

It’s not time to push the panic button – an experienced bankruptcy judge won’t start slashing and burning – but it isn’t the time to rely on old certainties either. A new U.S. marshal just rode into town, and hasn’t decided whether the local sheriff is the solution or the problem.

Monday, in an emergency meeting held amidst a crowd of raucous protestors, the CPUC gave PG&E permission to borrow more money, which it will have to do to pay for operations during the bankruptcy proceedings – so called debtor in possession financing.

The CPUC also filed a brief with the federal judge overseeing PG&E compliance with criminal sanctions resulting from the deadly San Bruno natural gas explosion in 2010. According to Politico, the CPUC objected to a hugely expensive electric line inspection throughout PG&E’s territory proposed by judge William Alsup, arguing “the proposal interferes with their oversight and would endanger public safety”. It’s arguable whether Alsup’s idea would help or hinder public safety, but there’s no question that it shoves the CPUC aside. Which might be why he’s proposing it: CPUC oversight did not prevent the San Bruno explosion or the Camp Fire or any of the other fires, deadly or otherwise, that PG&E is implicated in.

Charter’s vague compliance claims should be publicly verified by CPUC

by Steve Blum • , , ,

Charter Communications claims it’s providing near-gigabit level broadband service in virtually all of its Californian territory. Well, some of its Californian territory: in a filing with the California Public Utilities Commission, in opposition to a formal vetting of its claims that it is complying with service upgrade conditions imposed by the CPUC when it received approval to buy Time Warner cable systems, Charter says “it is already making service available at 940 Mbps to over 99% of the relevant households passed as of the end of year 2018”.

The filing doesn’t define “relevant” although it’s easy to assume it means all Californian households that had access to broadband service at the time of the merger. Maybe that’s because that’s what it means. Or maybe because Charter is hoping that commissioners are sloppy readers and won’t notice the weasel word.

It is also to be hoped that commissioners will take notice of another filing by Charter earlier this month, in which it promised to finish analog to digital upgrades in several California communities later this year. That’s a direct admission that it hasn’t met another CPUC imposed condition that required those upgrades to be completed by last November. There’s been no public announcement, by Charter or the CPUC, that an extension was requested or granted, although I suppose it’s possible some kind of understanding was reached behind closed doors. Or maybe Charter is hoping that commissioners won’t notice the disconnect.

I know from personal experience that Charter’s first impulse when asked to document compliance is to withhold as much information as possible, offering only as much as you might otherwise glean from their advertisements and other public statements. The threat or reality of CPUC action is an effective way of holding them accountable for a promise, and for holding their attention while they make good on it.

The CPUC should not take self-interested and unverified statements written by Charter’s lawyers at face value. Nor should it allow Charter to hide everything behind a blanket claim of confidentiality. Now is a good time to take a hard, quantitative and verified look at how – whether – Charter has met all of the statewide conditions the CPUC imposed on it in 2016. The matter should be reopened, investigated and, absent a compelling reason to suppress specific information, the results and underlying data should be made public.

Quickly.

I assisted the City of Gonzales with its successful effort at the CPUC to force Charter to upgrade. I am not a disinterested commentator. Take it for what it’s worth.

FCC colluded with mobile carriers to “game” judicial procedure, congressmen charge

by Steve Blum • , , , ,

Charlottesville streetlights

People at the Federal Communications Commission might have leaned on AT&T, Verizon, Sprint and the Puerto Rico Telephone Company to go shopping for sympathetic judges who would be more likely to bless its preemption of local ownership of streetlight poles and similar municipal assets. A letter sent by a pair of democratic congressmen – Frank Pallone (D – New Jersey) and Mike Doyle (D – Pennsylvania) – directs republican FCC chairman Ajit Pai to enlighten them on why the four wireless companies filed largely identical and completely ludicrous appeals of its September order

It has come to our attention that certain individuals at the FCC may have urged companies to challenge the Order the Commission adopted in order to game the judicial lottery procedure and intimated the agency would look unfavorably towards entities that were not helpful. If true, it would be inappropriate for the FCC to leverage its power as a regulator to influence regulated companies to further its agenda in seeking a more friendly court. To date, four FCC licensees have petitioned the federal judiciary for review of the Order in separate filings and in separate circuits.

Whether or not the four mobile carriers colluded with friends at the FCC to find a friendly court, in the end it didn’t matter. The City of San Jose completed a legal Hail Mary pass and got the all the cases sent to the ninth circuit federal appeals court in San Francisco, which tends to take a more skeptical view of FCC authority.

The mobile companies, which claim the FCC committed a grievous error by not giving them the pink slip along with the keys to the city, are still in a position to disrupt the appeals launched and joined by dozens of cities, counties and associations from every corner of the U.S. To head that off, local governments are asking the ninth circuit to hold a “case management conference”…

Because this case involves appeals by representatives of industry (which argue that the agency did not go far enough in adopting remedies sought by industry) and representatives of local governments and organizations (which argue the FCC’s actions significantly exceeded the agency’s authority), there is also likely to be a more complex pattern of briefs filed in support of, and in opposition to the FCC’s Order than is reflected in a typical agency appeal.

Right now, nine cases – three filed by mobile carriers and six by local governments – are in the San Francisco court, with four more – one by AT&T and three by local governments – apparently inbound from the federal appeals court in Washington, D.C.