How do you get a car to “see” a funny hat? Training autonomous vehicles is harder than you think

While self-driving cars are years away, Canada’s tech ecosystem is already leading the space from the front.

PHOTO ILLUSTRATION BY MONICA GUAN

Teaching a car to drive is hard — there is just so much for it to learn. Case in point: In August 2018, two engineers from the University of Waterloo went on a ride in the “Autonomoose,” the school’s partly self-driving test vehicle. As they tooled around the suburbs of Waterloo, the vehicle — the first developed by a Canadian university — collected data from a suite of on-board sensors about the environment, road conditions and potential obstacles. It slowed and signalled, accelerated and merged, all with minimal intervention from the engineers.

Until, that is, the Autonomoose came up behind a stopped car with a trailer full of lawnmowers.

“The Autonomoose had no idea what it was,” recalls Steven Waslander, then director of the Waterloo Autonomous Vehicles Laboratory. “It just ignored it and kept driving. It had never been trained on that.” An engineer had to tap the brakes to avoid rear-ending the gardening equipment.   

It sounds like a joke — but it’s a perfect illustration of the stubborn technological challenges that have put the brakes on some of the sunnier predictions about how soon we’ll see fully self-driving technology on our streets and highways.

Today, that goal of completely autonomous vehicles (A.V.s) looks a little farther away than it did just a couple of years ago. “There’s been a universal realization as to how much effort automation is going to take. There’s a huge difference between demonstrating that the technology can work, and proving that it works in all circumstances, and that it’s safe,” Waslander says.

As the timeline for full autonomy is pushed back, some may wonder what the point of pursuing it is at all. For Waslander, accessibility is key to the self-driving value proposition: “It means that you don’t have to be a driver to use a vehicle.” It would give the young, the very old and people with intellectual or physical disabilities more convenient access to personal transportation.

And of course, there’s safety. In 2020 alone, 1,745 people died on Canadian roads, most due to driver error. “Vehicular accidents are in the top 10 global killers in the world,” says Colin Earp, KPMG’s Canadian national transport leader and global chair in infrastructure technology. “We have somehow come to accept this as normal. Not only could safe autonomous systems dramatically reduce that, but the social costs associated with accidents and injury as well.”

Those benefits are pushing the research forward to help make A.V.s a reality, and Canada is home to some of the most ambitious efforts of all.

“Canada is the world-leading hub of A.I. There is a talent capability in A.I., decision-making software and sensors at a density found nowhere else in the world,” says Colin Earp. “All the conditions should be right for Canada to be forging ahead in A.V.s.”

Giving autonomous cars more road experience — virtually

In 2020, KPMG released its newest Autonomous Vehicles Readiness Index, measuring 30 countries on their A.V.-friendliness, based on metrics ranging from policy to technology to infrastructure. It showed Canada slipping to 12th position from seventh in 2019 (there was no 2021 ranking, due to the pandemic).

“There are naturally challenges and barriers that Canada has to face,” says Earp. “The sheer geographical size is a challenge.” (The top jurisdiction in KPMG’s index was tiny Singapore.)

But the report also cited, as strengths, Canada’s wide variety of urban and rural testing environments, its strong automotive sector and its role as a world-leading hub of A.I. research.

For evidence of the latter, look no further than the founding in 2021 of Waabi, an A.V. startup led by Raquel Urtasun, a University of Toronto professor who is also the former head of Uber’s self-driving research group.

The time she spent at Uber — where self-driving vehicles experienced multiple accidents, including a fatal pedestrian collision in Arizona — “really imprinted the importance of safety and having safety first,” she says.

Nothing is safer, of course, then running your tests in a computer rather than the real world. With the help of $83.5 million in venture funding, the company is creating “Waabi World,” which aims to build “digital twins” of the real world at greater scale and fidelity than ever before. Essentially, the idea is to put self-driving systems through their paces virtually, running through millions upon millions of simulated kilometres.

It could be a solution to Waslander’s lawnmower problem — or, as machine-learning experts call it, the open-world problem. “Every time you go driving, there’s a small chance you’ll see something you haven’t seen before,” explains Waslander. “Pedestrians get missed because they’re wearing funny hats or brightly coloured clothing, objects are missed because the computer doesn’t recognize them — there’s always something new out there.”

Mastering wintery conditions

Another problem that has so far stymied A.V.s: winter. But Canadian researchers are using extreme weather to their advantage. Waslander now leads a robotics and A.I. lab at the University of Toronto that is working on WinTOR, a project to teach self-driving systems how to navigate northerly conditions like icy roads, exhaust plumes clouding cameras, and landscapes of blinding snow. Based in warmer climes, the industry’s heavy hitters have yet to grapple with these issues.

“They’re all still trying to resolve rain,” says Waslander.

As with Waabi World, most of the WinTOR research will be done in a computer rather than the real world. Waslander is planning some road-testing next winter, however, including visits to northern Ontario on the “five or six days that are absolutely atrocious.” (Slippery conditions remain difficult to replicate on simulators.)

Winter-weather testing is also a selling point at Area X.O, a sprawling test facility operated by economic development agency Invest Ottawa. This 1,900-acre grid of roadways and low-slung industrial buildings, plunked into the agricultural landscape southeast of the national capital, houses what Invest Ottawa CEO Michael Tremblay calls “Canada’s most advanced transportation-focused futureplex.”

The site includes a grid of streets outfitted as test tracks for A.V.s; a high-speed, pre-commercial 5G network; and soon, satellite-provided internet courtesy of Telesat, the Canadian satellite operator developing a network of broadband-internet satellites. It all makes for a microcosm of the advanced communications systems that a world full of self-driving vehicles will need, so that vehicles can communicate near-instantaneously with one another and with the smart-city infrastructure around them.

Area X.O. is currently home to a multitude of startups, academic partners and governmental outfits at work developing, among other things, self-flying drones, “smart” agricultural machines and snowplows, and autonomous public-transit vehicles. The latter includes a low-speed public-transit shuttle trial launched in 2020 that saw real-world testing at a government campus in Ottawa, serving actual passengers.

Keeping things (and people) moving

The A.V.s that most of us will see in the real world will be in transport and transit fleets, says Earp. “I don’t think it’s the retail vehicles that will drive this going forward,” he says. “What you will see is large fleets start to change.” Think fleets of small delivery vehicles that could be dispatched to remote locations that don’t merit the cost of larger vehicles and drivers. These fleets could communicate with one another via futuristic communications networks, maximizing their efficiency to “create a logistics network that Canada has never seen before.”

Toronto-based NuPort Robotics, for example, is one of a growing number of autonomous trucking companies already using public roads, moving goods from warehouses to rail terminals in Ontario, albeit under operator supervision.

Those predictable, repetitive trips are also why A.V.s may be coming relatively soon to a public-transit system near you. The Canadian Urban Transit Research & Innovation Consortium is currently working with municipalities nationwide to advance the next-generation of electrified, autonomous, data-driven mass transit.

“You do not need fully autonomous buses that can navigate under any traffic circumstance or situation,” says the consortium’s CEO, Josipa Petrunic. “But you have to change the cityscape and streets; you need dedicated and semi-dedicated rights of way to make that feasible.”

Full autonomy will require even greater changes: the kind of ultra-connected, ultra-reliable digital infrastructure that can be glimpsed at Area X.O., but which is still in its infancy.

But the benefits, believes Earp, will be well worth the effort. “We have a tendency to think of A.V.s as just being about the self-driving car. But it’s really about mobility, about the way cities work and how we live. There’s just a massive opportunity to integrate all of these, and it will transform our communities.”

MaRS believes “innovation” means advancing Canadian technology for the benefit of all people. Join our mission.