Connect with us


AWS for Industry, but Better: The Railroad Investment Case



AWS for Industry, but Better: The Railroad Investment Case

Welcome to the weekly free edition of The Diff! Subscribers-only posts you missed this week include what J.P. Morgan is seeing in the broader economy (and why it matters), how demographics will reverse a financial law of nature everywhere except Japan, and thinking of your research time as a portfolio.

Coming attractions include a look back at the “productivity paradox,” a profile of YouTube as a business, and a look at how continued improvements in search have changed the way we look for information (and made us truly spoiled relative to our data-impoverished ancestors).

This newsletter goes out to 32,107 readers, up 430 since last week. In this issue:

  • AWS for Industry, But Better: The Railroad Investment Case

  • Smart Margin

  • Peak College?

  • Two-Sided Markets

  • Bonds

  • Diff Jobs

AWS for Industry, But Better: The Railroad Investment Case

If you’re choosing where to work, you probably want to aim for secular growth. Growing industries have more employees, fewer responsibilities and better economies of scale. This is good news for your career prospects, regardless of whether you are looking to become a manager or a contributor. In a shrinking industry, you suffer the tyranny of the long generation, where organizations get more risk-averse as the median employee ages, and start to promote more based on seniority at exactly the time when the people who already work there are accumulating a lot of it.

For investors, the picture is more complicated. Growing industries can make fortunes. The biggest net worths are those that result from growth. However, growing industries face more competition and the expected returns on capital will decrease if it is not available faster than the opportunity. Railroads are, in relative terms at least, the most non-growth industry imaginable, since they used to be most of the market and went through a roughly century-long decline featuring massive losses, painful consolidation, and the largest bankruptcy in history up to that point from the Penn Central. 1 This decline had several causes:

  • Passenger rail was an important revenue source early in railroads’ existence, but total passengers peaked in 1920 and had declined by more than half by the 50s thanks to the rise of the automobile. Rail-based passenger transportation can work in some contexts, like dense cities (the MTA carried 5.5m riders each weekday in 2019, above the entire US rail system’s passenger traffic at its peak, but short hauls have different economics than longer ones, and the MTA still needs heavy subsidies).

  • Trucking devoured market share in freight transportation in the mid-twentieth century. The Interstate Highway System played a major role in this. In the ten years that followed, trucks surpassed railroads by 7 points, making them the largest single type of freight.

  • Railroads benefit when more supply chains are domestic, and containerization made it easier for low value-added parts of the manufacturing process to move overseas. For example, a car manufactured in the USA might require trainloads full of steel. A steel mill will need trainloads full of iron and coal. Railroads are impacted by the movement of parts of the supply chain overseas. They earn revenue from each step that takes place on land.

  • Railroads were especially well-adapted to bulk transportation. As they lost share in other kinds of transportation, they remained dominant for coal, but coal has been in terminal decline since 2008.

  • Unit economics for railroads look best when each locomotive is pulling lots of cars. Railroads that wanted to optimize their unit profitability traded-off on scheduling: if one customer was a day behind for a big shipment, they’d wait, but that meant that the next customer would be behind, too. (The popularity of just-in-time manufacturing is not compatible with a delivery date where the error bar is 20% on either side.) These scheduling issues were problematic for a unionized workforce that had many rules about work hours. A shipment that arrives a few hours late might arrive at the end of its last shift, rather than in the middle. This would force the railroad to hire a new crew, even though it had already paid the former one.

  • As a legacy of the industry’s growth, there were many cases where multiple companies could service the same routes. Their total volume decreased, so they had to compete more on price. Fixed costs were always the same regardless of how much they carried. It was better to make a small profit than have idle tracks, equipment and workers.

  • Because of all these negative trends, smart managers generally avoided the industry. This only adds to the problems. This makes it more difficult for an industry to adapt when workers are trying to keep their pensions. It won’t accept many risks when there have been only minor downsides over the years.

So every secular trend possible has been conspiring against the train industry for an astonishingly long time. This is just one side of the story. The other side is that the US has the best freight rail infrastructure in the world, and rail transportation is about 80% cheaper than trucking per ton-mile. And trains and trucks can be combined to achieve rail’s cost savings for one part of the route and trucking’s flexibility for the rest. 2 If the industry got its act together, that cost advantage and irreplaceable route network could produce some good returns for investors.

The industry did, in fact, get its act together, and attracted interest from smart investors. Warren Buffett bought BNSF in 2008, Pershing Square won a tough proxy fight with Canadian Pacific in 2012, which turned into one of his most successful investments ever by 2016, and this interview with Eric Mandelblatt at Soroban is a good overview of the current bull case on railroads. This interview with Eric Mandelblatt at Soroban is a great overview of the current bull case on railroads I’ve actually spent a lot of time thinking about railroad economics due to a high school addiction to Railroad Tycoon II, but hadn’t looked seriously at the industry until recently. 3)

The story of how that happened is partly about a bad situation that ran out of ways to get worse, and partly a story about how the assets these companies have were valuable all along. The bear case against railroads has a part to do with environmental issues. Coal is still a large source of revenue and spillage can prove extremely dangerous. But the bull case, too, is an environmental one: drawing a line on a map, flattening it, and ramming steel into it at scale is just not something the US will really allow; all it takes is one recalcitrant landowner or one potentially-endangered species anywhere between point A and point B to nix the project. Railroads had to find a way that would fit into a just in time environment. They did this by rethinking the unit economics of their operations and leveraging their cost advantage.

Why American Rail Infrastructure is Great

The economic model for railroads is not too far off that of big tech: there’s a fixed asset that leads to a hard-to-match competitive advantage, and variance in outcomes is driven by how well companies exploit that asset. Although it might seem strange to think that someone who enjoys investing in software companies would also like to invest in an industry that emits smoke, it is true. Cascade Investment Group was the family office of Bill Gates and used some of its Microsoft sales proceeds to become Canada’s largest shareholder.

Understanding the railroads starts with understanding that asset base.

America in the present, is very fortunate to have been very sloppy and irresponsible in the distant past. Over-optimism and market manipulation are the reasons America has such an impressive railroad network. Railroads were about two thirds of the market in 1900, and other big chunks like banks, steel, and telegraphs were intimately tied to the railroad industry’s fortunes. The market has never been dominated by any industry, and it will not again. 4 Right now there are basically three publicly-traded railroads in the US, with a collective market cap of around $250bn, worth about as much as Coca-Cola. In 1900, they had a million employees, and total revenue that was three times Federal tax receipts. Today, they employ around 135,000 people, and total revenues for North American Class I railroads were $73bn in 2021. (Back to the soft drink comparisons, that’s a bit smaller than PepsiCo’s $79bn. )

There were a few big drivers for the 19th century growth of the North American railroad industry:

  • Early on, railroads had a cost advantage relative to other forms of transportation.

  • They were also capital-intensive enough that they had to raise money through capital markets rather than banks. They were also capital-intensive enough that they had to raise money through capital markets rather than banks. A British investor might be skeptical about making a loan to an American company. However, he could purchase American railroad bonds or shares and know that there were American banks watching the situation and that there was a secondary market. 5

  • Railroads, especially transcontinental ones, were heavily subsidized through government loans and land grants. As generous as these subsidies were, the railroads found ways to extract more value from them: the Union Pacific’s lawyers found a poorly-written clause in a subsidy law, allowing them to reinterpret their interest obligations from subsidized loans in a way that added $43m in subsidized interest to an intended $77m loan.

Meanwhile, railroads operating before the advent of modern accounting, not to mention modern data storage and analysis, had no idea what their unit economics looked like. They couldn’t figure out the breakeven price of a load or the incremental return from adding a branch.

This doesn’t mean that there weren’t reliable ways to make money from the railroad system. There were two popular ones:

  1. Directly skimming money from construction, by creating a captive construction company owned by railroad insiders and politicians that overcharged the railroad and then paid dividends to supporters.

  2. Insider speculation in land based on foreknowledge about what route the railroad chose.

Both of these were harmful to the railroads’ owners, and didn’t do anything for customers, either. It did provide a strong incentive to companies to continue building as many as they could. The new track brought more subsidies, land grants, opportunities to earn money, and the possibility to flip acreage along the route.

Since railroad investors weren’t fully aware of how much they were being ripped off, they tended to overestimate profits. Every incentive was available to increase track mileage, including subsidies, speculation and theft. Rail mileage peaked at over 250,000 miles in 1916, by which point the industry had already gone through several cycles of boom and bust (a quarter of the rail system by mileage was in bankruptcy by 1893).

Since then, there’s been a long retrenchment; total railroad mileage is around 92,000 today, with the pace of decline leveling out in the last fifteen years or so.

Transcontinental railroads shaped the US economy and culture (and politics, leading to the abrogation of several treaties with tribes whose land was along routes railroads wanted). This is even more true in Canada where the inclusion of British Columbia was contingent on the completion the Canadian Pacific Railway. This was a strong argument for industrial policy, or against the corporate welfare state. Your discount rate is a key factor in determining whether this proved to be a strong argument in favor of the power of industrial policies. The railroads created the largest financial market in the world, created the US manufacturing base and a dense network that provides cheap over-land transportation.

The Railroads Today

This map is a good look at where things stand today. Every major port has a railroad connecting it. Although they no longer serve a primary purpose of transporting people, the cost to transport goods is very competitive.

Whether that cost advantage accrues mostly to railroads’ customers (and their customers’ customers, i.e. The extent to which the industry is consolidated and how it’s managed will determine whether this cost advantage accrues primarily to railroad customers (and their customers, i.e. American consumers) as well as railroad shareholders. Consolidation was painful and long-lasting, as were the management changes.

It’s always tricky to attribute industry-wide changes to specific individuals, but two stand out. The first was Hunter Harrison, a former CEO of the Illinois Central and Canadian National Railways, Canadian Pacific, Canadian Pacific and CSX. He was a busy manager and job-hopping caused serious problems for all parties. Harley Staggers was the other, sponsoring a bill to de-regulate railroads. This would allow them to set their own rates as long as there are competitors. It also allows railroads to establish contracts with other shippers and reduce railroads’ influence on each other’s rates.

One reason Harrison was so effective was the Long Generation effect mentioned above: he joined the St. Louis & San Francisco in 1963, when the department he worked for had made just one or two hires since the end of the Second World War. Harrison was also an exemplar of working as an individual contributor well past the point where it’s directly economically rational: at one point, as a CEO, he booked a hotel room with a view of a rail yard, spotted an idle train, and called the station to demand an explanation. He was also a CEO and noticed congestion on his network status dashboard. He called the dispatcher at the station and worked for the whole night as a dispatcher.

This kind of obsession about detail is somewhat unreasonable, and could be quite expensive; Harrison was making tens of millions of dollars a year, so in terms of opportunity cost he spent around a dispatcher’s annual salary by working as one for a single night. It can have a multiplicative effect if the CEO is good at negative feedback (e.g. yelling at people and firing them) is more effective than positive feedback.

The more repeatable element of Harrison’s impact was that he rethought railroad unit economics. Railroads have always aimed to build the longest train possible. Since every incremental car costs less, you can measure P&L when trains are moving. But railroads are also a capital-intensive business; CSX’s $13.1bn of annual revenue requires a $40.5bn asset base, and that asset base ignores the value of the long-since-depreciated-to-zero rights-of-way that make railroads so valuable in the first place.

If the sole priority is longer trains, something has to give–many things, in fact. For example, if a customer arrives late, but can fill more cars in a few days, schedules will fall. It is possible for idle equipment to multiply and train can end up with suboptimal route. Customers have an understanding of railroad economics and can demand lower prices as well as more delays.

All this means that a railroad that doesn’t adhere to a relentless fixed schedule is more capital-intensive than one that does. Harrison was determined to correct this problem, focusing on the schedule and allowing customers to be available when they were not. This decreased idle time and made the whole system more predictable. Customers who relied on rail transportation now have an alternative, but more costly, in trucks. This gave them an economic incentive to continue to pay.

Harrison’s manifesto is actually available on Amazon (for $400), but this book is a good summary, albeit one highly favorable to the Harrison view of things. Two strong data points support Precision Scheduled Railroading. First, when Harrison agreed to join CSX it added $10bn to the company’s market value in a few days; CSX was worth 28% more with Harrison as probable CEO than it had been before. And second, other railroads ended up adopting the same methodology; if you Google “precision scheduled railroading” today, the top result is from the blog of Union Pacific, which adopted it in 2018.

Railroads still face challenges. As of 2019, around 14% of their traffic still consisted of coal, which has been declining at around 6% each year. Over the past decade, railroad traffic has remained flat. However, traffic excluding coal has been increasing in the low single digits each year; as coal declines further, the secular trend toward more railroad shipping will drive traffic up.

And that’s just a general bet on economic growth. The interesting part is looking at what happens when supply chains change. The US is a country with high wages, low-cost capital, flexible capital, relatively cheap energy and land, compared to other countries. Manufacturing becomes more capital-intensive and possibly more energy-intensive as it matures. It also tends to become less labor- and resource-intensive. One way to view the global macro trend in manufacturing is to see that the US’s relative disadvantage in manufacturing begins to decrease as it becomes less sensitive to the cost of manufacturing and more dependent on the price of manufacturing.

This doesn’t mean that we’ll run the story of globalization in reverse, and that the US will once again be the world’s dominant manufacturer; you generally don’t want your country to be making lots of shoes, lawn furniture, and toys domestically, unless the alternative is subsistence farming or starvation. However, it does mean that outsourcing some types of manufacturing becomes less cost-effective over time. This is especially true when it becomes harder to trust long, distant supply chains, regardless of whether they are due to American or Chinese foreign policies. The US is not capable of imposing the kind of severe Covid lockdowns that have been enforced by the Chinese Communist Party. Therefore, the US manufacturing base can be harder to disrupt.

Because of their cost advantage in intra-US transportation, and because of monopolistic economics that are politically and technologically infeasible for anyone else to duplicate, the railroad industry represents a sort of royalty on the growth rate of domestic manufacturing (and, to a lesser extent, consumption of bulky products like cars, lumber for houses, grain for food, oil, etc.). Just like how cloud computing affects the growth of software or ads in consumption, spending more on the end product results in higher spending and higher margins on hard-to-replace intermediate products.

One thought experiment to use on the railroad industry is to imagine that Congress decides to give Elon Musk rights-of-way on a 92,000-mile network connecting all major US ports and manufacturing centers. This would make The Boring Company’s ambitious plans look boring and lead to a revolution in domestic manufacturing. Now that the railroad industry is on board with higher operating standards that fit well into a just-in-time world and a trend away from offshoring, they start to approach that hypothetical Musk asymptote. Thanks to errors of commission in the 19th century and errors of omission throughout the 20th, North American railroads have gotten massive overinvestment, leading to a wonderfully valuable asset that can keep on throwing off cash for years to come.

Paying subscribers can read Part 2 tomorrow, where we’ll do a deep dive on a single railroad and look at what drives their economics and how to value them.


A Word From Our Sponsors

Here’s a dirty secret: part of equity research consists of being one of the world’s best-paid data-entry professionals. It’s a pain–and a rite of passage–to build a financial model by painstakingly transcribing information from 10-Qs, 10-Ks, presentations, and transcripts. Or, at least, it was: Daloopa uses machine learning and human validation to automatically parse financial statements and other disclosures, creating a continuously-updated, detailed, and accurate model.

If you’ve ever fired up Excel at 8pm and realized you’ll be doing ctrl-c alt-tab alt-e-es-v until well past midnight, you owe it to yourself to check this out.


Smart Margin

At a CFTC roundtable discussing a new futures margin proposal from FTX, a representative of futures exchange ICE claimed that at least one major trading firm technically defaulted early in the Covid pandemic, but was given a pass and ultimately did not collapse ($, FT). Although the comment didn’t give any details about what had happened, it did leave out the warning that “that person sitting in this room knows exactly who I am talking about”). “)

It’s not immediately obvious which is better for financial stability: a system where someone can slightly default and get away with it, or one where market chaos is compounded by the collapse of a trading firm, which would probably take down other firms and perhaps exchanges with it. There are anecdotes here and there about similar situations; in his memoir, Jim Cramer claims that at the peak of the 1998 meltdown, his hedge fund should have gotten a margin call from its brokers, but they didn’t get around to it in time and prices recovered. This happened to Archegos as well: they refused to provide more margin even though the collateral value was continuing to decline.

There may be a load-bearing level of forbearance within the industry, especially on the part of exchanges that may know their counterparties are hours away from fixing whatever put them in default. This proposal by FTX is intended to eliminate this discretion by automatically adjusting margin requirements and liquidating positions in reaction to market movements. This proposal is more elegant than the existing system because it’s easier to read, but it could be that the legacy version’s illegibility was what allowed it to continue running.

Peak College?

College enrollments are down for a fifth straight semester, with community colleges slowing more. Although this is not a Covid trend, it has been slowing down more than others. Students who went to college online realized that they could have a similar experience for a fraction of the cost through coding bootcamps or other options. The ROI is still there, even if it doesn’t compare to college.

Two-Sided Markets

While EVs are still rising in popularity, with US sales up 85% Y/Y in 2021, there’s a shortage of charging stations ($, WSJ). While cars and gas stations had their own economic models, it was difficult to create the same type of drivetrain with electric vehicles. The gas station business depends on foot traffic, which is a good thing. State grants for charging stations have been snapped up mostly by established companies; in Texas, 85% of a $21m grant went to Shell and Buc-ees. 6


China is allowing foreign investors to access its onshore bond market ($, FT) starting at the end of next month. China’s bonds are popular due to its large size and ability to offer better real returns than other sovereigns. It’s not directly comparable because the Chinese financial system is not compatible with fully open financial flows. I’ve previously written about how this closed system limits China’s prospects as a reserve currency issuer. Any move that opens up their markets further improves their chances of achieving reserve currency status. It also indicates the government’s intentions in this direction.

Diff Jobs

Diff Jobs is our service matching readers to job opportunities in the Diff network. Our clients include a variety of venture-funded companies. We focus on fintech and software, but we also work with other industries.

If you’re interested in pursuing a role, please reach out–if trhere’s a potential match, we start with an introductory call to see if we have a good fit, and then a more in-depth discussion of what you’ve worked on. Depending on the role this could be focused on work or on side projects. Job applicants can apply for Diff Jobs at no cost.

  • A company trying to build something wildly ambitious in decentralized computing is looking for a Principal Product Designer. (US, remote)

  • A company solving a major time sink for developers is looking for a variety of engineering candidates (ML research, devops, backend, frontend etc.) Washington D.C. area.

  • A company helping financial institutions, governments and exchanges to track financial crime on blockchains is looking for a Snr. Product Manager. Experience as a PM in the data industry is required. (US, remote)

  • A company making working capital loans to a traditionally underbanked group of small businesses is looking to add a bizops and strategy lead. (SF)

  • A company that is being incubated out of a top VC fund is using web3 tech to improve the way brands can acquire and retain customers. The company is looking for a senior engineer leader without any web3 experience. (US, remote)

  • We’re looking for data engineer roles for multiple companies, including a senior role at an edtech company that would build out the business’s entire data team, and a position at a rapidly-growing e-commerce business that’s helping expand a novel market. (US, remote)

Find a Role With Diff Jobs

Hire With Diff Jobs

Read More

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published.


Google Pixel 7 and 7 Pro are getting a built-in VPN at no extra cost




Google Pixel 7 and 7 Pro are getting a built-in VPN at no extra cost
Google Pixel 7 Pro hands on front Snow

(Image credit: Future / Lance Ulanoff)

Users of the Google Pixel 7 and 7 Pro devices will be able to secure their data without the need to pay for an additional Android VPN after the company said it would be including its Google One VPN service at no extra cost. 

The move will make the Pixel 7 and 7 Pro the first smartphones to include a free VPN connection. 

The offer is restricted to just some countries, though – and what’s more, some data won’t be secured inside the VPN tunnel.  

Peace of mind when you connect online ✨Later this year, #Pixel7 and 7 Pro will be the only phones with a VPN by Google One—at no extra cost.¹#MadeByGoogle¹See image for more info 6, 2022

See more

Google Pixel 7 VPN

Despite the aforementioned limits, the big tech giant assures that the VPN software won’t associate users’ app and browsing data with users’ accounts. 

Google One VPN typically costs around $10 per month as part of the company’s Premium One plan, which also comes with a 2TB of cloud storage on top. 

This decision is the latest move to bring Google’s mobile data security to the next level. Not too long ago, the company made Google One VPN available also for iOS devices, and also introduced the option of having an always-on VPN across its latest smartphones. 

Google promises that its secure VPN software will shield your phone against hackers on unsecure networks, like public Wi-Fi. It will also hide your IP address so that third parties won’t be able to track your location.

Shorter for virtual private network, a VPN is exactly the tool you want to shield your sensitive data as it masks your real location and encrypts all your data in transit. Beside privacy, it can allow you to bypass geo-restrictions and other online blocks. 

Chiara is a multimedia journalist, with a special eye for latest trends and issues in cybersecurity. She is a Staff Writer at Future with a focus on VPNs. She mainly writes news and features about data privacy, online censorship and digital rights for TechRadar, Tom’s Guide and T3. With a passion for digital storytelling in all its forms, she also loves photography, video making and podcasting. Originally from Milan in Italy, she is now based in Bristol, UK, since 2018.

Read More

Continue Reading


The Steam Deck dock is finally here and will ship faster than you think




The Steam Deck dock is finally here and will ship faster than you think
a steam deck placed in a steam deck dock

(Image credit: Valve)

After months of waiting and delays, Valve has finally announced that the Steam Deck dock is available for purchase on its official site.

Not only that but, according to Valve, the dock will ship out in an incredibly fast one to two weeks, which pairs with the fact that the Steam Deck itself is now shipping with no wait time (not to mention that it’s incredibly easy to set up). The port selection is pretty solid as well, with the dock featuring three USB-A 3.1 gen 1 ports, one Ethernet port, a DisplayPort 1.4, and an HDMI 2.0 port. And for its power supply, it uses a USB-C passthrough delivery.

A Steam Deck dock will run you $90 (around £81 / AU$140), which is a bit steeper than most third-party options on the market right now. But for those waiting it out for an official product until now, price most likely will not be an issue.

Is it worth buying? 

Considering that even Steam Decks themselves are shipping without a queue and that the dock has such a quick turnaround to delivery, it seems that the supply chain issues that had been gripping Valve are loosening considerably.

However, the deck itself is far from perfect. Because of the fact that it uses USB-C for the display port, a third-party USB-C dock that uses its own power supply and video out will output the display of the official dock. 

And as mentioned before, the price of the official Steam Deck dock is steeper than many third-party options on the market, meaning that those who are on a budget might pass this product up in favor of a lower-priced one.

There are also some bugs that Valve is working on fixing at this time, including one involving compatibility with LG displays. According to the FAQ, if the “Docking Station is connected via HDMI, sleep/wake can result in visual noise.”

It might be worth waiting for Valve to work out the kinks of its dock before investing in one. And while you’re waiting, research other options that might better suit your needs.

Allisa has been freelancing at TechRadar for nine months before joining as a Computing Staff Writer. She mainly covers breaking news and rumors in the computing industry, and does reviews and featured articles for the site. In her spare time you can find her chatting it up on her two podcasts, Megaten Marathon and Combo Chain, as well as playing any JRPGs she can get her hands on.

Read More

Continue Reading


Why doesn’t Bash’s `set -e` do what I expected?




Why doesn’t set -e (or set -o errexit, or trap ERR) do what I expected?

set -e was an attempt to add “automatic error detection” to the shell. Its goal was to cause the shell to abort any time an error occurred, so you don’t have to put || exit 1 after each important command. This does not work well in practice.

The goal of automatic error detection is a noble one, but it requires the ability to tell when an error actually occurred. In modern high-level languages, most tasks are performed by using the language’s builtin commands or features. The language knows whether (for example) you tried to divide by zero, or open a file that you can’t open, and so on. It can take action based on this knowledge.

But in the shell, most of the tasks you actually care about are done by external programs. The shell can’t tell whether an external program encountered something that it considers an error — and even if it could, it wouldn’t know whether the error is an important one, worthy of aborting the entire program, or whether it should carry on.

The only information conveyed to the shell by the external program is an exit status — by convention, 0 for success, and non-zero for “some kind of error”. The developers of the original Bourne shell decided that they would create a feature that would allow the shell to check the exit status of every command that it runs, and abort if one of them returns non-zero. Thus, set -e was born.

But many commands return non-zero even when there wasn’t an error. For example,

if [ -d /foo ]; then ...; else ...; fi

If the directory doesn’t exist, the [ command returns non-zero. Clearly we don’t want to abort when that happens — our script wants to handle that in the else part. So the shell implementors made a bunch of special rules, like “commands that are part of an if test are immune”, and “commands in a pipeline, other than the last one, are immune”.

These rules are extremely convoluted, and they still fail to catch even some remarkably simple cases. Even worse, the rules change from one Bash version to another, as Bash attempts to track the extremely slippery POSIX definition of this “feature”. When a SubShell is involved, it gets worse still — the behavior changes depending on whether Bash is invoked in POSIX mode. Another wiki has a page that covers this in more detail. Be sure to check the caveats.

A reference comparing behavior across various historical shells also exists.

Story time

Consider this allegory, originally posted to bug-bash:

Once upon a time, a man with a dirty lab coat and long, uncombed hair
showed up at the town police station, demanding to see the chief of
police.  "I've done it!" he exclaimed.  "I've built the perfect
criminal-catching robot!"

The police chief was skeptical, but decided that it might be worth
the time to see what the man had invented.  Also, he secretly thought,
it might be a somewhat unwise move to completely alienate the mad
scientist and his army of hunter robots.

So, the man explained to the police chief how his invention could tell
the difference between a criminal and law-abiding citizen using a
series of heuristics.  "It's especially good at spotting recently
escaped prisoners!" he said.  "Guaranteed non-lethal restraints!"

Frowning and increasingly skeptical, the police chief nevertheless
allowed the man to demonstrate one robot for a week.  They decided that
the robot should patrol around the jail.  Sure enough, there was a
jailbreak a few days later, and an inmate digging up through the
ground outside of the prison facility was grabbed by the robot and
carried back inside the prison.

The surprised police chief allowed the robot to patrol a wider area.
The next day, the chief received an angry call from the zookeeper.
It seems the robot had cut through the bars of one of the animal cages,
grabbed the animal, and delivered it to the prison.

The chief confronted the robot's inventor, who asked what animal it
was.  "A zebra," replied the police chief.  The man slapped his head and
exclaimed, "Curses!  It was fooled by the black and white stripes!
I shall have to recalibrate!"  And so the man set about rewriting the
robot's code.  Black and white stripes would indicate an escaped
inmate UNLESS the inmate had more than two legs.  Then it should be
left alone.

The robot was redeployed with the updated code, and seemed to be
operating well enough for a few days.  Then on Saturday, a mob of
children in soccer clothing, followed by their parents, descended
on the police station.  After the chaos subsided, the chief was told
that the robot had absconded with the referee right in the middle of
a soccer game.

Scowling, the chief reported this to the scientist, who performed a
second calibration.  Black and white stripes would indicate an escaped
inmate UNLESS the inmate had more than two legs OR had a whistle on
a necklace.

Despite the second calibration, the police chief declared that the robot
would no longer be allowed to operate in his town.  However, the news
of the robot had spread, and requests from many larger cities were
pouring in.  The inventor made dozens more robots, and shipped them off
to eager police stations around the nation.  Every time a robot grabbed
something that wasn't an escaped inmate, the scientist was consulted,
and the robot was recalibrated.

Unfortunately, the inventor was just one man, and he didn't have the
time or the resources to recalibrate EVERY robot whenever one of them
went awry.  The robot in Shangri-La was recalibrated not to grab a
grave-digger working on a cold winter night while wearing a ski mask,
and the robot in Xanadu was recalibrated not to capture a black and
white television set that showed a movie about a prison break, and so
on.  But the robot in Xanadu would still grab grave-diggers with ski
masks (which it turns out was not common due to Xanadu's warmer climate),
and the robot in Shangri-La was still a menace to old televisions (of
which there were very few, the people of Shangri-La being on the average
more wealthy than those of Xanadu).

So, after a few years, there were different revisions of the
criminal-catching robot in most of the major cities.  In some places,
a clever criminal could avoid capture by wearing a whistle on a string
around the neck.  In others, one would be well-advised not to wear orange
clothing in certain rural areas, no matter how close to the Harvest
Festival it was, unless one also wore the traditional black triangular
eye-paint of the Pumpkin King.

Many people thought, "This is lunacy!"  But others thought the robots
did more good than harm, all things considered, and so in some places
the robots are used, while in other places they are shunned.

The end.


Or, “so you think set -e is OK, huh?”

Exercise 1: why doesn’t this example print anything?

   2 set -e
   3 i=0
   4 let i++
   5 echo "i is $i"

Exercise 2: why does this one sometimes appear to work? In which versions of bash does it work, and in which versions does it fail?

   2 set -e
   3 i=0
   4 ((i++))
   5 echo "i is $i"

Exercise 3: why aren’t these two scripts identical?

   2 set -e
   3 test -d nosuchdir && echo no dir
   4 echo survived
   2 set -e
   3 f() { test -d nosuchdir && echo no dir; }
   4 f
   5 echo survived

Exercise 4: why aren’t these two scripts identical?

   1 set -e
   2 f() { test -d nosuchdir && echo no dir; }
   3 f
   4 echo survived
   1 set -e
   2 f() { if test -d nosuchdir; then echo no dir; fi; }
   3 f
   4 echo survived

Exercise 5: under what conditions will this fail?

   1 set -e
   2 read -r foo < configfile


But wait, there’s more!

Even if you use expr(1) (which we do not recommend — use arithmetic expressions instead), you still run into the same problem:

   1 set -e
   2 foo=$(expr 1 - 1)
   4 echo survived

Subshells from command substitution unset set -e, however (unless inherit_errexit is set with Bash 4.4):

   1 set -e
   2 foo=$(expr 1 - 1; true)
   4 echo survived

Note that set -e is not unset for commands that are run asynchronously, for example with process substitution:

   1 set -e
   2 mapfile foo < <(true; echo foo)
   3 echo ${foo[-1]} 
   4 mapfile foo < <(false; echo foo)
   5 echo ${foo[-1]} 

Another pitfall associated with set -e occurs when you use commands that look like assignments but aren’t, such as export, declare, typeset or local.

   1 set -e
   2 f() { local var=$(somecommand that fails); }
   3 f    
   5 g() { local var; var=$(somecommand that fails); }
   6 g    

In function f, the exit status of somecommand is discarded. It won’t trigger the set -e because the exit status of local masks it (the assignment to the variable succeeds, so local returns status 0). In function g, the set -e is triggered because it uses a real assignment which returns the exit status of somecommand.

A particularly dangerous pitfall with set -e is combining functions with conditionals. The following snippets will not behave the same way:

   1 set -e
   2 f() { false; echo "This won't run, right?"; }
   3 f
   4 echo survived
   1 set -e
   2 f() { false; echo "This won't run, right?"; }
   3 if f; then  
   4     echo survived
   5 fi

As soon as a function is used as a conditional (in a list or with a conditional test or loop) set -e stops being applied within the function. This may not only cause code to unexpectedly start executing in the function but also change its return status!

Using Process substitution, the exit code is also discarded as it is not visible from the main script:

   1 set -e
   2 cat <(somecommand that fails)
   3 echo survived

Using a pipe makes no difference, as only the rightmost process is considered:

   1 set -e
   2 somecommand that fails | cat -
   3 echo survived

set -o pipefail is a workaround by returning the exit code of the first failed process:

   1 set -e -o pipefail
   2 failcmd1 | failcmd2 | cat -
   4 echo survived

though with pipefail in effect, code like this will sometimes cause an error, depending on whether the output of somecmd exceeds the size of the pipe buffer or not:

   1 set -e -o pipefail
   2 somecmd | head -n1
   4 echo survived

So-called strict mode

In the mid 2010s, some people decided that the combination of set -e, set -u and set -o pipefail should be used by default in all new shell scripts. They call this unofficial bash strict mode, and they claim that it “makes many classes of subtle bugs impossible” and that if you follow this policy, you will “spend much less time debugging, and also avoid having unexpected complications in production”.

As we’ve already seen in the exercises above, these claims are dubious at best. The behavior of set -e is quite unpredictable. If you choose to use it, you will have to be hyper-aware of all the false positives that can cause it to trigger, and work around them by “marking” every line that’s allowed to fail with something like ||true.


GreyCat‘s personal recommendation is simple: don’t use set -e. Add your own error checking instead.

rking’s personal recommendation is to go ahead and use set -e, but beware of possible gotchas. It has useful semantics, so to exclude it from the toolbox is to give into FUD.

geirha’s personal recommendation is to handle errors properly and not rely on the unreliable set -e.

Read More

Continue Reading


Copyright © 2022 Xanatan