Tag Archives: voip

FCC report on T-Mobile nationwide outage is a case study in network complexity and best practices (or lack thereof)

by Steve Blum • , , , ,

Tmobile billboard 2 las vegas 6jan2020

The installation, and incomplete configuration, of a new router and a fiber link failure, both in the southeast U.S., combined with software and hardware bugs to take down T-Mobile’s national phone network in June, according to a report published in October by the Federal Communications Commission. The cascade of problems that began with a fiber route going down led to a “registration storm” in Atlanta as “mobile devices repeatedly attempted and failed to register” on the network, first using 4G, 3G and 2G mobile systems, and finally trying to complete calls via WiFi connections.

The storm spread “out of the Atlanta market and across the country”, disrupting phone service within T-Mobile’s network, and traffic between T-Mobile and other carriers. Millions of calls went nowhere…

Based on confidential call success and 3G and 2G call failure data shared by T-Mobile, together with data on 911 calls and calls originating outside of T-Mobile’s network, [FCC staff] estimates that at least 41% of all calls that attempted to use T-Mobile’s network during the outage did not complete successfully. This estimate does not include any possible call failures arising from T-Mobile subscribers’ VoLTE or Voice over Wi-Fi call attempts, which could not be determined. However, [staff] expects that if this number could be determined, it would result in [staff’s] estimate being much larger.

The impact on 911 calls wasn’t as severe as on general voice traffic, because emergency calls bypass the registration process. The system is designed to let people call 911 whether they have an active account or not. But the impact was still significant. T-Mobile said that 24,000 calls to 911 centers did not go through, and the FCC’s report said that people could not get the help they needed…

Based on the record, the June 15 outage on T-Mobile’s networks prevented some consumers from summoning the help that they needed during emergencies. Not only were some consumers unable to reach PSAPs by dialing 911, but they also were unable to reach roadside-service providers, medical professionals, and family…Fortunately, the Bureau did not receive any comments suggesting that individuals experienced physical harm as a direct result of this outage.

Usefully, the FCC’s report is not an indictment. It is a very readable case study with lessons learned that apply to all mobile carriers and fiber transport providers, and recommendations for preventing a reoccurrence. And a promise of corrective action ahead.

Phone service is phone service and emergency obligations apply regardless of technology, CPUC decides

by Steve Blum • , , , ,

Telephone companies have to follow disaster readiness and response rules laid down by the California Public Utilities Commission, regardless of the technology they use. That’s the CPUC’s opinion anyway. In a sharply written unanimous decision published yesterday, commissioners rejected challenges to telephone (but not broadband) emergency response obligations that they imposed on incumbent telcos, cable companies, mobile carriers and VoIP providers alike last year.

The regulatory logic that underpin those obligations also formed the basis for the CPUC’s initial response to the covid–19 emergency and the disaster resiliency standards for communications services that it recently adopted. The same cast of characters are fighting those edicts using similar arguments, so yesterday’s decision is both a good indication of how the commission will respond and how it will defend itself when the fight moves to federal courts, as it surely must.

AT&T, Charter Communications, Comcast, Frontier Communications and their lobbying front organisations claimed, among other things, that the CPUC’s disaster relief requirements were preempted by federal law because when phone service is delivered via 21st century voice over Internet protocol (VoIP) technology instead of 19th century copper wires and exchanges it magically transmogrifies from a telecommunications service to an information service.

Not true, the commission said. First of all, a federal court has already determined that telephone service is defined by the service provided and not by the technology used…

As the Court’s analysis demonstrates, the phrase “to facilitate communication by telephone” encompasses services beyond traditional landline service if the service facilitates “two-way communication by speaking as well as by listening,” regardless of the “[t]he exact form or shape of the transmitter and the receiver or the medium over which the communication can be effected.” Wireless service and VoIP service both facilitate two-way communication by speaking as well as by listening.

Second, while generally upholding the Federal Communications Commission’s repeal of network neutrality rules, a federal appeals court in the District of Columbia said last year that there’s no blanket preemption of state regulation of information services…

The [D.C. appeals court]…presents a more reasoned analysis, which preserves state authority over consumer protection matters that the FCC has either no authority to preempt or where no actual conflict exists. [It] supports the Commission’s consumer protection efforts in the Decision. Therefore [the telco and cable company] preemption argument fails.

Similarly, the CPUC rejected arguments made by AT&T and the mobile industry’s lobbying mouthpiece that the FCC reigns supreme over any wireless service. The decision said emergency response requirements have nothing to do with market entry or the price of service, which the CPUC cannot regulate per federal law, but are instead “‘other terms and conditions’ of wireless service”, which the same law firmly places under state jurisdiction.

AT&T guilty of obfuscation, delay, deception, inaccuracy, evasion, omission and contradiction regarding 911 service, CPUC says

by Steve Blum • , , , ,

Bluto pencils

AT&T has to pay a $3.75 million fine because of its “pattern of obfuscation, delay, and deception” in dealing with the California Public Utilities Commission, and the “inaccuracy, evasion, omission, and contradiction” in its description of its 911 service. The core issue was whether AT&T is required to file particular paperwork regarding next generation 911 services. The answer from the CPUC is an emphatic yes. AT&T’s refusal to do so and the manner in which it refused earned it the multimillion dollar fine.

The CPUC’s unanimous vote upholds a ruling earlier this year by an administrative law judge. AT&T appealed to the commission, claiming it had done no wrong because it merely slipped through legal loopholes created by differences in technology.

It’s a claim AT&T continues to make, most recently when it objected to new CPUC disaster readiness rules. That argument was debunked by the commission’s decision, which reiterated that 911 service is 911 service, regardless of how it’s provided or what network segment of the 911 system is being provided…

The Commission seeks to protect Californians who need safe energy delivery and reliable communications through the natural and man-made disasters to which California is increasingly prone. The Commission’s need for “accurate information from the utility in order to, among other things, ensure that it is providing just, reasonable and safe service” is acute, given the inherent information asymmetry between regulator and regulated entity. AT&T has not provided accurate information pertaining to the issues before us…

Emergency service tariff violations are not garden-variety regulatory misfeasance. The transport of emergency communications is a life and death matter. The difference between transport that guarantees 98%, 99.9% or 99.999% availability for a given trunk line can well mean the difference of an ambulance or fire truck that arrives on time and one that does not.

If anything, the decision said, the $3.75 million fine “may even be too modest” because of AT&T’s “financial resources” and the severity of its violation of “the public trust attendant on the utility services it provides”.

Commissioners made one significant change to the penalty. Originally, AT&T would have been given 30 days to file the necessary paperwork, and if it didn’t, the fine would have doubled to $7.5 million. Instead, AT&T will be fined $15,000 for every day it’s late.

Power out? No 911? California bill allows cable, telcos to say stick it

by Steve Blum • , , , ,

Woolsey fire victim

Companies that provide voice over Internet Protocol (VoIP) and fixed wireless Internet service (WISPs) won’t, for the most part, have to keep their networks running during disasters, under a bill that was just amended in the California assembly. As now written, senate bill 431 generally confirms resiliency requirements – e.g. 72 hours of backup power and maintain access to “basic internet browsing for emergency notices” in high fire threat areas – imposed on mobile carriers by the California Public Utilities Commission this week, but draws the line there.

That’s a big win for Comcast, Charter Communications, Cox Communications and other cable companies that offer telephone service via VoIP technology. All they would have to do is mail a warning label to customers once year and tell them to stick it on their phone. The warning would advise customers that losing power means losing 911 service, although it might not come across that way since weasel words like “may be impacted” are allowed.

It’s a double win for AT&T and Frontier Communications. They get the same VoIP break as cable companies, plus they’ll be able to offer service via fixed wireless facilities that have no backup power or disaster preparedness requirements at all, instead of upgrading or maintaining wireline networks, which must be kept running during emergencies and electric outages.

An earlier draft of SB 431 would have extended back up power and other network resiliency requirements to pretty much any Internet service provider in California, but that text disappeared. Presumably, that’s the work of the platoons of lobbyists that AT&T, Comcast, Charter and the rest deploy in Sacramento to stuff millions of dollars into lawmakers pockets. It’s no coincidence that the bill is scheduled to be heard in the assembly communications and conveyances committee next week, which is stuffed with lawmakers who have particularly benefited from big telecom’s generosity and have graciously repaid those favors, at least until netizens take notice.

AT&T blasts loopholes as it tries to escape $3.75 million fine in California

by Steve Blum • , , , ,

As expected, AT&T appealed a 3.75 million fine levied by a California Public Utilities Commission administrative law judge for “wilful disregard” of its public safety obligations. The penalty followed months of wrangling with CPUC staff over what kind of information AT&T is required to provide about services, such as 911 emergency calls, that ride on voice over Internet protocol technology (VoIP).

AT&T’s appeal dives headfirst into the minutia of how 911 service is provided now, and how it will be provided once it’s completely switched over from legacy plain old telephone service (POTS) to modern digital technology. It also twists and turns through the legal technicalities of when and how it’s supposed to keep the CPUC informed, and whether breaking a particular rule is one time thing or a continuing violation. Maybe that tactic will work. If, say, AT&T can convince commissioners, or maybe a California court down the road, that there’s a loophole that allows them to refuse to give the CPUC information about rates and terms for a particular service, then that might enough to get them off the hook.

But that won’t answer the fundamental policy question of whether the CPUC can and should regulate modern telecommunications platforms that provide similar – if not the exact same – service as old school POTS. In its appeal, AT&T did not repeat its previous Alice in Wonderland argument that telephone calls made with one kind of digital technology are a telecommunications service, while identical calls made with another kind are not. It did, however, cite a California law that expired at the end of last year that generally blocked the CPUC from regulating VoIP or other Internet protocol enabled services. As the CPUC defines its role in regulating VoIP and other de facto digital telecommunications services, AT&T and other monopoly model telco and cable incumbents will be back in Sacramento, trying to resurrect that ban.

CPUC whacks AT&T with $3.75 million fine for “wilful disregard” of public safety obligations

by Steve Blum • , , , ,

AT&T was ordered to pay a $3.75 million fine by the California Public Utilities commission for blowing off demands for information about its 911 service in 2019. Administrative law judge Karl Bemesderfer issued a “presiding officer’s decision” in a disciplinary proceeding launched last year after AT&T refused to file reports detailing its rates and terms for “next generation” 911 services that ride on Internet protocol technology, rather than old style plain old telephone service.

Besides being a sizeable slap to AT&T, the decision is a reminder that defiance of CPUC directives can be expensive. That’s something T-Mobile and Sprint might take notice of: if wrangling over informational filings is worth a fine of a few million dollars, how much does it cost to baldly merge two giant companies without permission?

The decision blasted “AT&T’s wilful disregard for the State of California’s obligation to ensure the public’s safety through oversight of the 911 system”…

We conclude that by their deliberate repeated refusals to respond appropriately to the letters from [CPUC communications division director Cynthia] Walker, their knowing misrepresentations regarding their handling of 911 traffic, and their deliberate ignoring of [a commission decision and general order], and applicable law, Respondents have engaged in conduct that merits a fine…

We conclude that Respondents’ conduct is not so egregious as to merit a maximum fine nor so excusable as to merit a minimum fine. For their repeated refusal to respond to the letters from Director Walker we find that a fine of $10,000 per day or $2.5 million is appropriate; for their misrepresentations regarding the handling of 911 traffic and their deliberate disregard of [a commission decision], we find that a fine of $5,000 a day or $1.25 million is appropriate, for a total fine of $3.75 million.

If AT&T doesn’t immediately file the necessary information, the fine will double to $7.5 million.

The decision doesn’t try to carve out new regulatory territory for the CPUC. Although the service in question is delivered via voice over Internet protocol (VoIP) technology, which the CPUC was generally barred from regulating until this year, there was an exception for 911 service.

AT&T, or anyone else with a particular interest, have until the beginning of May to file an appeal, and CPUC commissioners can request a review. Assuming AT&T appeals, as it certainly will, the fine will be put on hold until the process plays out.

Accidentally honest AT&T tells CPUC to grab the horse by the tail and face reality

by Steve Blum • , , , ,

Chp horses capitol 3feb2016

Unintentionally, an AT&T witness injected an insight of startling clarity into the debate over whether or not broadband is a common carrier service. It happened during a hearing to determine if the company should be held in contempt of California Public Utilities Commission orders. The witness was discussing the difference between legacy digital methods for transmitting telephone calls and contemporary Internet protocol technology.

He said…

It’s like the difference between a horse and buggy, and an automobile.

Just so.

That difference is one that the California legislature reckoned to be irrelevant more than 100 years ago, when it rewrote the regulations for common carrier passenger road transportation.

Today, if you want to offer point to point road transportation service in California, you need to be certified as a “passenger stage corporation” by the CPUC. Passenger stage now includes passenger buses and passenger vans, but stage is a term that goes back to Gold Rush days and beyond, when horsepower meant horses. Originally, it referred to segments of a journey. Over time, it became the name of the horse drawn vehicle being used – stagecoach turned into stage.

Transportation was undergoing the same, radical technological shift in the early decades of the twentieth century that telecommunications is today. What didn’t change was the even older concept of a common carrier service, one that was available to all at published prices and level terms.

The California legislature decided in 1917 that it’s the service that’s important, and not the underlying technology. It passed a law defining a transportation company as any person or corporation that owned or operated “any automobile, jitney bus, auto truck, stage or auto stage used in the transportation of persons or property as a common carrier for compensation over any public highway in this state between fixed termini or over a regular route”, unless it was completely within the borders of a city. The “railroad commission of the State of California” – later to become the CPUC – was “vested with power and authority to regulate every transportation company in this state”.

As we say these days, the law was technology neutral. The details of the statute have changed over the years, but the fundamental principles and terminology haven’t. Regardless of what powers the vehicle, companies carrying people and cargo for hire are providing a legally identical, common carrier service.

Horse and buggy or automobile. TDM or IP. Candlestick phone or VoIP modem. It doesn’t matter.

It’s the same service.

Internet magic means phone calls aren’t phone calls, AT&T tells CPUC

by Steve Blum • , , , ,

Alice tall 625

We’re all mad here.

On Thursday in San Francisco, AT&T defended itself against charges that it’s in contempt of California Public Utilities Commission orders and that it broke CPUC rules and state law. AT&T is admitting that California law no longer bars the CPUC from regulating Internet protocol enabled service such as voice over Internet protocol (VoIP), but doesn’t appear to be giving up the fight. Instead, it’s falling back to a second line of defence that was thoughtfully provided by the Federal Communications Commission.

The dispute centers on next generation 911 service, but it’s also the first test of the CPUC’s ability to regulate services that ride on Internet technology since the expiration of a state law that previously blocked such regulation. I sat in on the AT&T contempt hearing for a few minutes – would’ve spent more time, but that wasn’t the way my day went. It was just a brief taste, but the flavor was consistent with AT&T’s written response. Which was mostly dry arguments about who provides each piece of the increasingly complex communications path between the public and 911 answering centers, and how that maps to the equally complex web of California’s regulatory obligations and AT&T’s deliberately byzantine corporate structure. Links to AT&T’s filing and the hundreds of pages of exhibits are below.

Previously, AT&T’s defence rested, in large part, on the California legislature’s 2012 decision to bar the CPUC from regulating VoIP and similar, Internet-delivered services. No longer. Its latest response mentions that now-expired law only in passing, and in the past tense.

But AT&T prepared a fallback position. In an attachment, AT&T tries to define next generation 911 service as an “information service”, as opposed to 911 service based on legacy technology , which it admits is a “telecommunications service”. This nonsense is the result of the Federal Communications Commission’s 2017 decision to repeal network neutrality rules and declare, in Alice in Wonderland fashion, that transporting data from point A to point B via the Internet isn’t telecommunications.

AT&T jumped down that same rabbit hole by claiming, in effect, that phone calls that ride on that one, particular kind of digital transportation aren’t phone calls. Unfortunately, AT&T isn’t trying to make its case to the Queen of Hearts.

Why, sometimes I’ve believed as many as six impossible things before breakfast.

AT&T’s Response to Administrative Law Judge’s Ruling Regarding Order to Show Cause, 6 January 2020
Exhibit 1
Exhibits 2 through 8
Exhibits 9 through 15
Exhibit 16

AT&T faces contempt hearing as CPUC defines VoIP regulatory role

by Steve Blum • , , , ,

Bluto pencils

The first shot in what could be the defining regulatory battle over broadband in California was fired in the closing days of December by the California Public Utilities Commission. An administrative law judge (ALJ) ordered AT&T

To show cause, if any, why [AT&T] should not be:

  1. Found in contempt of [a 2019 CPUC decision regarding disaster preparedness].
  2. Found in violation of the Public Utilities Code and [a CPUC rule requiring telcos to file price/service terms (aka tariffs)].
  3. Fined, penalized, or have other sanctions imposed for failing to comply with a Commission decision, [commission rules], and the Public Utilities Code.

The dispute began last Spring when CPUC demanded that AT&T file a notice – an “advice letter” – detailing its terms for “Next Gen” 911 service, which will run over an Internet protocol connection, like other Internet data, rather than using legacy copper network switching and other 20th century technology.

AT&T first blew off the demand, and then said it’s none of your business

[Mark Berry, AT&T regulatory director] spoke with [CPUC] staff and relayed the following in response to the question of why AT&T had not filed an advice letter:

  1. AT&T does not offer the services referred to in the letter and even if it did offer these services, AT&T does not agree that the CPUC can require a tariff because under [a now expired public utilities code section], the CPUC does not have authority to regulate IP-enabled services.
  2. If AT&T offers Next Gen 911services in the future, it will not file tariffs because the CPUC does not have authority over these services.

The CPUC and AT&T exchanged more such pleasantries, until AT&T finally filed some paperwork, without answering the questions asked. So AT&T executives were ordered to appear at a hearing later this month to explain themselves.

This kind of arm wrestling over filing and disclosure requirements is nothing new. Business as usual would be a good description, although it usually doesn’t get this far. This case is significant because the primary legal basis for AT&T’s refusal expired at the end of 2019. It was a law enacted in 2012 that banned the CPUC from regulating Voice over Internet Protocol (VoIP) or other “Internet protocol enabled” services. Back then, VoIP was still a developing technology, and telcos and cable companies hadn’t gone all in on it as a replacement for legacy copper service and as a way to get out from under the regulatory oversight that comes with it.

AT&T and other monopoly model telecoms companies tried to get the ban extended last year, but ran into a brick wall in Sacramento, also known as the Communications Workers of America. The betting is that they’ll try again this year – why spend billions on service quality when a few million in the pockets of lawmakers will get you off the hook?

So it’s up to the CPUC to figure out how VoIP fits into California’s regulatory ecosystem. One way the commission can do that (relatively) quickly is to litigate disputes like this one, and bake new case law into the resulting decision.

Ponderosa Telephone makes its case for blocking Comcast’s bid to cherrypick “high end” households

by Steve Blum • , , , , ,

Tesoro viejo construction 25aug2019

Ponderosa Telephone shot back at Comcast’s claims that no harm would come from its proposed cherry picking of affluent households in a new, high end development outside of Fresno. In comments filed with the California Public Utilities Commission last week, Ponderosa made its case for denying Comcast permission to offer telephone service in its territory. The company argued that if the CPUC wants to change its current policy of protecting small rural telcos from competition, it should do so on a top level basis, and not on case by case requests from a major telecoms company.

Particularly if that telecoms company’s request for special treatment is “disingenuously misleading”.

California has 13 rural telephone companies that serve remote communities. Or in some cases, communities that used to be reckoned as remote, before the arrival of suburban and exurban sprawl. Rural telecoms service can be expensive – miles and miles of lines are needed to reach scattered homes and businesses. Low population density means low revenue density, so to keep telephone service affordable both the CPUC and the Federal Communications Commission back fill rural telco’s budgets with subsidies from universal service funds. To keep the tab for taxpayers as low as possible, the CPUC doesn’t allow competitive telephone companies, or big incumbents who want to exert their monopoly model might, to carve off service areas where the revenue potential is the highest and the need for subsidies is the lowest. If there’s a need at all.

That policy is under review, in a CPUC proceeding that could take years to resolve. Meanwhile, Comcast wants permission to add telephone service – it already can offer broadband and TV service – to newcomers able to afford a home in the (relatively) pricey Tesoro Viejo development, just north of Fresno. That would be costly to taxpayers, Ponderosa said…

Comcast seeks to raid the most profitable consumers in Ponderosa’s service territory. This “cherry-picking” concern by [non-carrier of last resort telcos] operating in [rural telco] territories was a factor that led the Commission to conclude that wireline competition would “leave behind residential, small business, and community anchor institution customers in more scattered and harder to serve areas of the rural carrier’s territory”; “adversely affect the bulk of the hard-to-serve and high cost customers”; and “result in the [small rural telcos] losing revenue and needing to seek a larger draw from the [California High Cost Fund rural subsidy] program.”

Abandoning, or at least substantially modifying, decades-old rural telecoms policy might be necessary, as 21st century digital services replace legacy telephone technology and business models that, in some respects, date back to the 19th century. It needs to be done thoughtfully and carefully, and not on the basis of requests for case by case special treatment by telecoms giants.