Tag Archives: telecom

Rural broadband gaps are life and death issues, California wildfire study says

by Steve Blum • , , , ,

Paicines pole route

Ageing, inadequate infrastructure contributed to the destruction during last year’s Camp Fire in Butte County that killed 86 people and did billions of dollars worth of damage. Congested roads were a big part of the problem, but so was a lack of telecommunications service, either because it was knocked out by the fires or, in many cases, not there in the first place, according to a report by a “strike force” commissioned by California governor Gavin Newsom…

In a matter of hours, 52,000 people from rural Paradise and surrounding communities evacuated onto roads built for a fraction of that capacity and converged on Chico, overwhelming the recovery system. The scale and speed of catastrophic, wind-driven wildfires, like the Camp Fire, incapacitate existing emergency response systems, local infrastructure and planned recovery efforts. Many California communities designed their fire emergency response and recovery systems decades ago, using old technology and outdated fire modelling. A clear overhaul of the California emergency response systems and the underlying infrastructure is needed.

The lack of broadband in rural communities and access to cell service make it difficult to communicate clear emergency evacuation orders to residents or locate residents who are in trouble.

Broadband did not play a significant role in warning residents of massive fires sweeping through California’s wine country in 2017. The North Bay/North Coast Broadband Consortium surveyed nearly 1,600 residents of the fire stricken areas. Only 11 said they received warnings from online sources: five on Facebook, four from Nextdoor.com and two via notices on public agency websites.

Phone calls – including those from from family, friends, public agencies – played a bigger role. About a third of the respondents were alerted via either mobile or landline calls.

The big problem during the wine country fires was the damage done to telecommunications infrastructure. Nearly four-fifths of the people surveyed lost mobile connectivity, either partially or completely, and two-thirds lost landline connections. Overall, 69% were cut off from the Internet for at least some of the time during the disaster.

CPUC allows AT&T, Frontier to tap dance their way out of fines for bad service

by Steve Blum • , , , ,

AT&T and Frontier Communications were fined $2.2 million and $823,000, respectively, by California Public Utilities Commission, for “chronic” service failure, primarily in rural California. Sorta. Kinda.

Well, not really.

At its meeting in Fresno last week, the CPUC voted unanimously to allow Frontier and AT&T to skip the fines, which were mostly for taking too long to restore telephone service for customers who experienced outages. In return, the companies promised to make “incremental” investments in improving service quality. The amount of those supposedly incremental investments are claimed to be twice the amount of the fines. Which is allowed under a baffling and irregular decision made two years ago by the commission.

The resolution regarding the AT&T fines noted the difficulty in figuring out what’s incremental and what’s money that would have been spent anyway. CPUC staff asked AT&T for list of what it planned to spend on “construction and rehabilitation projects”. AT&T responded with a list of repair work it was doing, rather than “planned projects focused on the rehabilitation of poor performing central offices”. That was because AT&T claimed not to know what it would have spent normally, because “they do not budget for specific projects; all project work is identified on a rolling basis and reprioritised based on the ability to reduce high maintenance costs”.

Translation: we can tell you anything we want and you have to believe us because you can’t prove differently.

After some back and forth, CPUC staff accepted AT&T’s story and recommended that commissioners accept AT&T’s claims and give the company two years to demonstrate “the results of their proposed projects to measurably improve service quality in its network”.

Frontier’s explanations are similarly wooly.

After the vote, which was on a consent agenda – a bunch of resolutions bundled together – commissioner Clifford Rechtscaffen said the policy of letting telephone companies fine themselves and keep the money needs a second look…

These are a set of resolutions we just approved under a new program we initiated a couple of years ago as an amendment to general order 133-D, and in particular they allow telephone companies who have been found to have violated our service quality requirements to substitute paying a penalty by making investments to improve service quality that are at least twice the amount of the penalty. I understand that this investment option was a creative way to try to address longstanding service quality deficiencies.

At the time the resolution was adopted a couple of commissioners expressed concerns and reservations about it, including commissioner Randolph, who asked the question of whether or not we would really be able to tell whether or not the investments that were made were incremental to what telephone companies would be doing in the normal course of business.

We’ve approved these three investment alternatives, but in practice we can now see that it is difficult, in fact, to determine whether or not the investments are incremental. This required a lot of staff time working with the companies to figure out what was new, what was not new and where the best places for the investments would be. I commend the staff for reaching good conclusions here. I continue to have some reservations about this option. I think we want to look carefully and see whether or not the investments really do improve service quality and whether or not this option makes sense in terms of our larger enforcement objectives.

The option doesn’t make sense. There’s no indication that either AT&T or Frontier offered any two year budgets, which is the only way to even begin determining whether their proposed spending-in-lieu-of-fines is, in truth, extra money on top of what they’d spend anyway. Or whether they’re just shifting money from one line item to another.

Letting Frontier and AT&T pay fines to themselves was a bad decision in the first place, and the details behind last week’s commission vote proves it.

CPUC refuses to reconsider waiving AT&T, Frontier fines for bad rural service

by Steve Blum • , , , ,

AT&T, Frontier Communications and other telephone companies can continue to fine themselves and keep the money, if they fail to meet California’s service quality standards. The California Public Utilities Commission rejected an appeal by a group of consumer organisations, which claim that the bizarre 2016 decision allowing telcos to pay their own expenses instead of paying fines was made “without any support whatsoever in the record”.

The decision was rammed through by commission president Michael Picker, who refused to allow a vote on an alternative offered by then-commissioner Catherine Sandoval, contrary to usual procedure.

Four organisations – TURN, the Greenlining Institute the Center for Accessible Technology and the CPUC’s office of ratepayer advocates – asked commissioners to reconsider the decision, pointing out that the let telcos keep the money alternative was never on the table during the years while evidence was gathered. It first appeared in a draft decision, after the record was closed.

The commission’s response, adopted in a closed door meeting last week, was to 1. argue that vague comments and complaints made by telcos early in the proceeding were a sufficient “factual record basis”, and 2. refuse to reconsider the decision.

Next week, commissioners are scheduled to vote on proposals to let Frontier and AT&T off the hook for hundreds of thousands of dollars worth of fines they’ve incurred because of poor telephone service in rural California. The 2016 decision says that’s okay if they spend at least twice the amount on “a project” that “improves service quality” in the next two years. That doesn’t necessarily mean building new infrastructure. It could go toward salaries or other operating expenses too. It just has to be an “incremental expenditure”.

There’s no way to know if they would have spent that money anyway. Corporate budgets shift year to year and quarter to quarter. We won’t even know what they’re spending the money on, or what service quality improvements to expect: the CPUC plans to keep that information secret, because of its “sensitive nature”.

Broadband service failed in 2017 California firestorm, mobile hit worst

by Steve Blum • , , ,

One of the big questions to answer about the mega fire still tearing through Shasta County this morning is how do you warn people? Broadband and other high tech tools failed in last year’s fires. Instead, people were saved the old fashioned way: a knock on the door or the smell of smoke.

Mobile service went down more often than any other kind of broadband service during 2017’s northern California firestorm, but cable, telco and fixed wireless systems also took a severe beating. Satellite Internet service had the highest survivability rate. That’s one conclusion drawn from a wide survey of people in and around Mendocino, Napa and Sonoma counties during an horrific wildfire in October 2017.

Mobile broadband carriers had an 88% failure rate – i.e. people “lost some to all service”. Broadband service from Comcast was down for 73% of customers, while 69% of AT&T and Frontier broadband subscribers experienced outages.

One caveat: the survey did not break AT&T subscribers out by type of service. Comparing total numbers, and also the voice telephone service data collected, it’s a fair conclusion that most of the 700 people who ticked the AT&T Internet service box are landline customers. So that’s how I categorised them, but I’m sure that some mobile customers are mixed in.

DSL resellers – companies that lease copper lines from AT&T or Frontier, add some equipment and provide semi-independent Internet access – only had a 43% failure rate. But that doesn’t mean they did better than the big guys – resellers concentrate their service in cities and towns, and most of the fire damage was in exurban and rural areas. If rural residents can’t buy the service, then they can’t lose it either.

One common problem that all the terrestrial companies in the counties have is that AT&T is pretty much the only middle mile game in town, according to the report. So if an AT&T inter-city line is damaged, all the ISPs using it suffer.

Satellite Internet service, which had a 26% failure rate, isn’t affected by AT&T’s backhaul outages. But it does share one weak link with all the others: electricity. If your power (or your provider’s) goes out and there’s no back up, you don’t have Internet service either. Smoke shouldn’t have been a problem for satellite or other wireless services – any smoke that’s thick enough to block signals is too thick to breath. No one should have been sitting at home trying to get on the Internet at that point.

The data was collected by the North Bay/North Coast Broadband Consortium. More than 3,700 people filled out an online survey. It wasn’t a scientifically selected sample, but just taken for what it is, it’s a significant number of responses.

North Bay/North Coast Broadband Consortium Telecommunications Outage Report: Northern California Firestorm 2017, released 10 May 2018.
Report Appendices, released 10 May 2018.
More information from the North Bay/North Coast Broadband Consortium.

CPUC tags Frontier’s service as “chronic failure”, proposes to let it slap itself on the wrist

by Steve Blum • , , ,

Frontier Communications failed to meet California phone service repair standards in 2017. It’s supposed to restore service within a certain amount of time 90% of the time in any given month, in every one of its Californian service territories. According to two draft resolutions on the table at the California Public Utilities Commission, two of Frontier’s three subsidiaries missed the mark every single month.

Of the 24 reports, the worst performance was 22% in January 2017, the best was 87% in December 2017. The company does show significant improvement over the course of the year, but not enough to escape a “chronic failure status” finding, and nominally face fines totalling $823,000.

Now, the CPUC is proposing to let Frontier keep the money. Under the terms of a 2016 decision rammed through by commission president Michael Picker, if Frontier agrees to pay for “incremental actions directed at improving compliance with the service quality standard that led to the fine in an amount that is no less than two times the incurred fine”, then it doesn’t have to pay it.

Frontier convinced CPUC staff that it’ll spend $2.1 million on service improvements over the next two years, that it wasn’t planning to make. Given the obligations that Frontier has to improve or extend service for more than 800,000 Californian homes – obligations imposed on it by the CPUC when it purchased Verizon’s systems in 2015 – and the company’s reluctance to offer details about its plans, there’s good reason to be skeptical about its latest promise.

There will be no opportunity to see what improvements Frontier proposes. It convinced the CPUC to keep the details of its plea bargain secret, because of its “sensitive nature”.

At the time, I described the CPUC’s 2016 vote as “the most cynical decision I’ve ever seen” it make, and pointed out that “concocting an ‘incremental expenditure’ two years down the road, based on last year’s budget, is a trivial accounting exercise”.


Life and death alerts are low tech/no tech, California firestorm study shows

by Steve Blum • , , ,

Social media and other online services were not the way people received lifesaving warnings when a firestorm tore through three northern California counties last year. Nearly all were alerted to evacuate via phone or personal contact, or by their own eyes, ears and noses.

That’s a top line read of the data from a study just published by the North Bay/North Coast Broadband Consortium. They ran an online survey and 3,700 people responded, nearly all of them from the hardest hit counties of Mendocino, Napa and Sonoma. The number is significant as it stands, even if it wasn’t a scientifically selected sample.

Given that some people had “literally seconds” to leave their homes in the face of a wall of fire that moved at speeds up to 50 miles per hour, the most important question is “how did you receive warning/notice to evacuate?” About a quarter got no warning at all – they figured it out themselves – and about a third were alerted by phone calls of various kinds. Nearly a quarter got a boots-on-the-ground warning – someone banged on their door or shouted out to them. Most of the rest checked other, and closer inspection shows nearly all of those responses are also variations on phone, physical or smell-the-smoke alerts.

The Internet accounted for very few of the time-critical messages. Of the more than 1,600 evacuees who responded, only 11 credited online sources for the warning: five people mentioned Facebook, four said Nextdoor and two saw notices on public agency websites. Even in those cases, most already knew something was going on. Radio and TV – ancient compared to the Internet, but newer tech than phones or feet – were almost as insignificant.

Across those three counties, 78% of respondents said they lost “some to all” of their cellular voice and data service, while 66% reported the same for wireline phones. Just looking at Internet connectivity, of whatever sort, 69% said they lost “some to all” of it.

There’s no mystery about the cause. The fires burned countless utility poles (some argue blazes were started by electric lines mounted on those poles) and more than “340 cell sites were completely destroyed or damaged”. When infrastructure goes up in smoke, service disappears too.

The report is largely quantitative and doesn’t point any particular fingers of blame, noting “the severity of the 2017 wildfires was unpredictable”.

North Bay/North Coast Broadband Consortium Telecommunications Outage Report: Northern California Firestorm 2017, released 10 May 2018.
Report Appendices, released 10 May 2018.
More information from the North Bay/North Coast Broadband Consortium.

Telecom Deficit Slows Angola’s Development

Steve Blum, Tellus Venture
Associates, doing project
due diligence in Huambo
Telecommunications and transportation make the difference between subsistence farming and sustainable commercial agriculture in Angola’s Huambo province, where Tellus Venture Associates is supporting a development project through Rotary International. The physical infrastructure was obliterated during nearly 30 years of civil war, but mobile phone applications could soon provide a life-saving solution.

Potatoes sold in Huambo might earn $175 per ton, but could fetch $500 or more per ton in coastal markets, hundreds of kilometers away. Using fertilizer and improved seed varieties, a smallholder farmer might produce 2.5 tons per crop. But that seed and fertilizer costs about $375, which means selling it locally will net little more than $60, while selling it on the coast nets $875. With two crops a years, that’s the difference between trying, often failing, to survive on pennies day, and earning enough to buy a family basic necessities, including a level of education for the children.

Landmines and destroyed military equipment litter the Angolan countryside
Destroyed military equipment
litters the countryside
It’s possible to stagger production within and among villages so that some produce can be sold on an annual contract basis. But agriculture is still largely a seasonal business, which makes up to date market information absolutely vital. The Gates Foundation and World Vision are working on filling that gap in Huambo. Lack of telecommunications infrastructure makes it very difficult, though.

The skeletal remains of utility transmission poles and towers are scattered throughout the province. Communications facilities were fought over and heavily mined during the war, and even today Huambo has one of the highest concentrations of landmines and active minefields on the planet.

Microwave trunking
Trunking within
and out of
Huambo is
based on
point to point
Chinese, Brazilian and Portuguese companies are rebuilding roads and rail lines, and in a couple of places underground fiber optic lines are reportedly going in at the same time. Angola Telecom, the government-owned PTT, provides landlines and phone service. Private carriers are beginning to appear, but it’s hard to tell if claims of progress are supported by facts on the ground. It’s possible to get a POTS line in some towns — after a long wait or a quick, under the table payment — but reliability is low. The bulk of day to day communication in Angola generally, and Huambo province in particular, is wireless.

There are two privately-owned mobile carriers, Movicel (CDMA) and Unitel (GSM). Coverage for both is relatively good in the capital of Luanda, although both regularly experience outages and have significant gaps. As a result, it’s common in urban areas for people to subscribe to both and carry two mobile phones.

In rural Huambo, where the agricultural development project is continuing, mobile network coverage is spotty. But spotty beats nothing at all. The next step in the project is to set up a commodity price reporting service from coastal markets, via SMS and MMS.

Luanda, the capital,
has the consumers,
Huambo has the produce
The Gates Foundation is funding market development programs, which establish contractual relationships with coastal supermarkets, restaurants and other wholesale buyers. Mobile phones could provide the essential link between those markets and smallholders in Huambo.

Longer term the hope is to extend Internet access into villages, to enable ongoing education and technical assistance from agricultural and marketing experts, in addition to current market data. Right now, the focus is on finding a sustainable way of paying for VSAT terminals, but a terrestrial solution – WiMAX, say – is likelier to succeed. Given the ample high ground around the agricultural valley, a handful of base stations, maybe as few as one or two, could service all the current project areas. Backhaul could be terrestrial or satellite-based, although much of the project-related traffic would be local and could be handled through what would be, in effect, a WAN.

Operating expenses have to be tighly controlled. Even taking full advantage of coastal market opportunities, cash flow will be at low levels relative to the cost of international VSAT service. The cost of bandwidth has to be proportional to the actual value added by any given application, which favors keeping as much traffic as possible on a local network.