Waymo’s Robots Drove More Miles Than Everyone Else Combined
But in newly released numbers from California regulators, the challengers in the self-driving-car race are looking better.
Self-driving cars promise to change cities, mint billionaires, and push robots into the everyday lives of millions of people. The only problem is, no one knows quite when or how. And with all the research and development locked up inside private companies, the public has little information to judge the progress of the technology, aside from the occasional PR reveal or disaster.
We have one (imperfect) yardstick, however: the numbers that the California Department of Motor Vehicles requires that any company testing an autonomous vehicle in the state file every month. Those are rolled up and released in January of each year. Though people in the industry don’t like what they see as the uneven comparisons between companies, this is the best we’ve got. The data include two primary numbers: the number of autonomous miles driven, which gives a rough indication of the scale of a program in the state, and the number of disengagements, or when a human driver takes over for the computer.
For every year of these disclosures, Waymo, the self-driving-car project within Google’s parent company, Alphabet, has been the leader by a wide margin.
The year 2018 was no different. Waymo drove 1.2 million miles in the state, which is not even its primary testing ground. Its cars disengaged 114 times, for a rate of 0.09 disengagements per 1,000 miles. That’s down from 0.18 in 2017. GM Cruise cemented its position as the key challenger to Waymo supremacy, logging nearly 448,000 miles with 162 disengagements, for a rate of 0.19 per 1,000 miles, and that’s on San Francisco’s difficult streets, a fact that GM Cruise’s Kyle Vogt is fond of pointing out. Together, the two companies’ cars drove 86 percent of the autonomous miles in the state.
Apple, whose self-driving program is less high-profile, came in at No. 3 in miles driven, with nearly 80,000 autonomous miles. However, the company’s disengagement rate was 871.65 disengagements per 1,000 miles, according to the DMV methodology—the highest of the 27 companies that submitted data.
Apple’s cover letter to the DMV indicates that the company changed its reporting methodology halfway through the year, and that after July 2018, the company’s rate of “important disengagements” would land it in the realm of 0.5 disengagements per 1,000 miles, which would be in the top tier of performances.
That enormous discrepancy highlights what the various companies don’t like about the reporting processes. They don’t have a true standard for what must count as a “disengagement,” leaving room for companies to make their numbers look better (or even worse) than they might otherwise be. Where the miles are driven obviously matters, too: It’s harder to drive in Manhattan than Palo Alto. And these cars are trying to learn, which means you don’t necessarily want to encourage drives on empty highways, where the learning rate per mile is low.
Nonetheless, no other state requires any kind of disclosure about miles driven or disengagements. These numbers are all we have, in large part because none of these companies want to self-report, nor do their interactions with the DMV seem to indicate that they’d like more stringent standards.
Huge, huge, huge money is at stake in the race to build autonomous vehicles of all kinds, and no company wants to give away more secrets than it has to.
This year, the field of real competitors has grown. More companies have attained the basic ability to run a self-driving car on the streets of California for fairly extended periods of time. The most advanced ones have expanded their driving greatly. And it’s worth noting that much of the action occurs in special training facilities outside California (whether that’s in Arizona or the traditional seat of the car industry, Michigan), or in simulated worlds filled with real data.
In just the past week, two new players received $1.4 billion of funding: $940 million to Nuro, a driverless delivery company, and $500 million to Aurora. Amazingly, both companies can trace their lineage to the Google self-driving-car project that eventually spun off into Waymo, a company that financial analysts value at tens of billions of dollars. In 2018, self-driving-car company Zoox also raised $500 million. Many others have secured or are eyeing tens of millions, a hundred million, or even a billion dollars.
Nuro, Aurora, and Zoox all appear on the DMV list. Zoox and Nuro both posted top-five disengagement numbers and ranked fifth and sixth in miles driven (30,000 and 25,000, respectively). Aurora drove almost 33,000 miles (fourth in the pack), but showed a very high disengagement rate of 10.01 per 1,000 miles. That could be consistent with the kind of long-term-oriented program that founder Chris Urmson has outlined.
Nuro’s regulatory filing was also unusually detailed in its description of the problems that its vehicles encountered, making concrete the general problems that self-driving cars can encounter. Among them: Cars can have trouble identifying objects; the mapping information they rely on to function can be out-of-date or inaccurate; they can have problems with their sensor inputs. And they can simply make bad decisions with the information they have; the filing includes entries such as “planned trajectory failed to leave adequate room for parked car on narrow road” and “planned trajectory resulted in erroneous sharp braking, recklessly tailgating motorist may have been unable to stop.”
What about Tesla, which has talked a big self-driving game? Like last year, the company used the necessity of filing a letter with the DMV to lobby for its heterodox mode of self-driving-car development, which included huge numbers of real-world miles logged by human drivers and its Autopilot mode, albeit with a more limited range of sensor data than other self-driving cars. Tesla drivers have logged 1 billion miles in Autopilot mode, according to the company’s filing. If that turns out to be the winning strategy, clearly Tesla will suddenly come into focus as the leader in autonomous driving. But until that day, we don’t really know how far up the ladder toward true autonomy their approach can take them.