Tag Archives: psps

Telephone and cable companies stonewalled California emergency officials during massive power outages

by Steve Blum • , , , ,

Cell site outages 28oct2019

Mobile carriers generally cooperated with California emergency officials during the week long siege of public safety power shutoffs in October, while cable and telephone companies hid behind confidentiality claims. Paul Troxel, who heads the 911 program at the California office of emergency services, testified at a California Public Utilities Commission hearing on Wednesday and told commissioners that neither the state’s emergency operations center or local officials knew where access to 911 service and disaster information, such as evacuation orders, were unavailable…

Outage data was not reported by all providers. Some providers were very responsive and provided outage data as requested by Cal OES, while others were slow to respond due to confidentiality concerns related to outage data. Frankly, information from the wireline and VoIP providers was not provided until the end of the event. Because of the lack of complete reporting, Cal OES had to work with the Federal Communications Commission to activate the disaster information reporting system.

The FCC’s data isn’t intended to support real time operations. It’s typically 12 to 24 hours old and only aggregated data is provided – at the county level for mobile carriers and the state level for telephone and cable companies. But it did provide a useful check on the accuracy of the data provided voluntarily by telecoms companies through their industry association, which is their standard method.

It didn’t work so well. According to Troxel, at one point the industry’s organisation, the California Utilities Emergency Association (CUEA), reported that 57,000 wireline customers were out of service, while the FCC’s disaster information reporting system said the figure was 224,000. CUEA’s mobile outage reports weren’t much better – in one county the FCC said 133 cell sites were down while CUEA claimed only nine were. The problems with the industry association’s data were severe enough that OES requested reports directly from telecoms companies, with varying degrees of success.

During the hearing, Verizon, T-Mobile and, perhaps, AT&T executives promised to provide timely, detailed outage information in future emergencies, and make it public. A senior AT&T operations executive said the company would do so, after a company lawyer deflected the question by extolling the wonderfulness of the current system – that’s an arm wrestling match that needs to be resolved. The other representatives, from Frontier Communications, Comcast, Charter Communications, Cox and Sprint, wouldn’t make any promises at all.

Meaningful answers and we’ll get back to you, as CPUC drills down on phone, broadband outages in emergencies

by Steve Blum • , , , ,

Cpuc phc telecoms outages 20nov2019

Telecoms company representatives – telco, cable and mobile – were grilled for three hours yesterday by CPUC commissioners about their ability to maintain communications capabilities during power outages and other emergencies. And their willingness to provide actionable, real time network status information to officials and the public.

The central issue is whether the California Public Utilities Commission should establish regulations for things like backup power, network resiliency and outage reporting, for voice, text and, perhaps, broadband service. Commissioner Cliff Rechtschaffen cut to heart of it, asking the eight representatives “would you support this as a regulatory requirement?”.

Three of the mobile companies – AT&T, Sprint and T-Mobile – were represented by senior operational and engineering executives. Although they didn’t express any great enthusiasm for new regulations, they engaged with questions posed by commissioners and generally gave knowledgable answers about their networks, back up capacity and emergency management procedures.

Verizon sent a lobbyist. He reiterated an early statement by Verizon that it would be happy to provide lots more information about future outages in something like real time, and make it public because our network is so damn good.

AT&T’s and T-Mobile’s execs signed onto that pledge. Mobile networks were a particular focus – 80% of 911 calls are made using mobile phones, according to a CPUC staffer. The objective is to identify and publicise communications gaps, where people can’t call 911 or access evacuation maps on the web. Sprint’s rep was more reticent, but it might not matter if T-Mobile is successful in acquiring the company.

It wasn’t clear whether AT&T’s wireline network was included in the promise. At one point, an AT&T lawyer jumped up and seemed to say no. Instead, he lauded AT&T platoons of lobbyists and public relations people, and their “longstanding” efforts to keep state and local officials informed.

Right.

Frontier Communications also sent a corporate lobbyist to the hearing. Not much came of it. She didn’t promise to share detailed or real time outage information, let alone make it public. She did say that only 85% of Frontier’s customers are served by central offices that have back up generators that can keep facilities powered for at least 72 hours. “Facilities further out” in “remote areas” rely on shorter-lived batteries and portable generators.

Translation: the urban systems we got from Verizon are okay, rural communities, not so much.

The three cable companies – Charter Communications, Comcast and Cox – sent regional managers, who typically have a lot of operational responsibility at the local level, but take their marching orders on corporate policy from headquarters. That seemed to be the case yesterday. All three were cordial and, within their field of expertise, knowledgeable enough, but not forthcoming when pressed for information sharing commitments. I’ll get back to you was a frequently heard response. Back up power on cable networks didn’t seem to be as robust as telcos. Comcast’s rep said that all their network devices in the field have back up power, but only 4 to 24 hours worth.

Telecoms companies to explain broadband, phone failures during California power cuts

by Steve Blum • , , , ,

Green acres utility pole

With another Pacific Gas and Electric company power shut off looming later today, the California Public Utilities Commission is calling in telecommunications companies and demanding that they be prepared to explain their “responsiveness during the latest wildfires and public safety power shut offs to keep telecommunications services on”.

A hearing is scheduled for this morning in San Francisco, with “top officials” from California’s major telecoms companies directed to “publicly appear and publicly address their response during the latest wildfire events [and] public safety power shutoffs”. The list includes California’s two big telephone companies – AT&T and Frontier Communications – as well as Comcast, Charter Communications and Cox on the cable side, and all four mobile carriers – AT&T, Sprint, T-Mobile and Verizon. Electric utility executives are also invited, albeit a bit more politely.

During the last wave of power cuts, intended to prevent wildfires, hundreds of thousands of Californians lost telephone and broadband service. Data collected by the Federal Communications Commission indicated that 455,000 telco and cable subscribers in 32 California counties lost connectivity, and 3.3% of cell sites were down. There might have been more – the FCC didn’t say how many telecoms companies were reporting outages – reporting was voluntary – or whether the total included all customers who couldn’t connect because they lacked back up power in their homes and businesses.

Who shows up will be as significant as what they say. Expect AT&T and Frontier to send people – they’re still regulated to an extent by the CPUC – but whether their representatives are actual decision makers or lobbyists with inflated titles remains to be seen.

Cable companies tend to resist any encroachment on their turf by the CPUC, so their response, if any, will be interesting to watch. Normally, mobile carriers don’t have much to say to state officials, but since AT&T has to be there anyway, and Sprint and T-Mobile are trying to make nice with the CPUC so they can gain approval for their merger, we might get some meaningful information from them.

Large scale telco, cable and mobile service outages follow California power cuts

by Steve Blum • , , , ,

Cell site outages 28oct2019

Hundreds of thousands of Californians lost their wireline broadband and phone service over the past week, as the state’s major electric utilities cut off power to millions of people in an attempt to prevent wildfires from breaking out. Mobile broadband and telephone subscribers were equally hard hit, with one county – Marin – losing more than half of its cell sites at one point.

The Federal Communications Commission has been tracking wireline and mobile service outages since last Friday, when the power cuts were hitting hard in Pacific Gas and Electric’s northern California territory, and public safety power shutoffs were beginning to bite in the southern California service areas of San Diego Gas and Electric and Southern California Edison. I’ve compiled all of their reports through yesterday into a single document, which you can download here.

From a telecoms point of view, the outages were at their peak on the FCC’s Sunday morning (0830 California time, 28 October 2019) report. At that time 455,000 telco and cable subscribers in 32 counties were without their landline connections and 3.3% of the total number of cell sites were down.

Some counties were hit much harder than others. Marin County lost 57% of its cell sites, while there were no reports of cell site outages in Santa Barbara County. Calaveras, Humboldt, Lake, Napa, Santa Cruz and Sonoma counties lost between 19% and 39% of cell sites.

It’s not clear what the wireline outage figure represents. Participation in the FCC’s disaster reporting system is voluntary. The list of willing companies hasn’t been made public and there’s no way of knowing if all of the telephone and cable companies in those counties are cooperating. The reports from the ones that are cooperating are based on “communications infrastructure status and situational awareness information” and “network outage data”. Which might not include all, or maybe even most, of the households and businesses which are offline because their equipment – cable and DSL modems, for example – don’t have backup power. The network might be fully functional, but if customer premise equipment is down, then service is too.

So that 455,000 customer wireline outage figure might be low.