Amazon is investing billions of dollars in robots to make its e-commerce business more efficient and profitable. This huge initiative started out a lot smaller.
A decade ago, the company launched a competition for university engineering teams called the Amazon Picking Challenge. It called on researchers to design robots for a common warehouse task: grabbing products from a shelf and putting them in a box.
As a tech reporter, I was intrigued by this quirky project. At the time, in early 2015, Google was testing self-driving cars, a technology that emerged from a similar academic competition known as the DARPA Grand Challenge. What if Amazon was trying to replicate this magic, but with robots rather than automobiles?
Then a funny thing happened. The Amazon Picking Challenge faded away. It was renamed, and it lasted only a few years. I chalked this up to another bad call and moved on.
I thought about this challenge again late last year. Amazon had unveiled a next-generation warehouse in Louisiana with 10 times the robots moving products around and, yes, picking them up with dexterity. The company has said the facility is designed to process orders faster and more efficiently, and it's likely to be the future of its e-commerce operation.
Ten years after the Amazon Picking Challenge, the fruits of this nerdy competition have finally emerged. It follows an uncannily similar timeline as the DARPA Grand Challenge, which started in 2004 and resulted in Google's driverless cars hitting the road roughly a decade later.
So, with the help of Business Insider's Eugene Kim, I investigated how Amazon's huge new fleet of picking robots came to be, and how this competition laid the foundation for a new wave of automation that's about to crash over the warehouse and logistics industry.
From pallets to picking
It started with an acquisition. In 2012, Amazon paid $775 million for Kiva Systems, which designed flat robots that zip around warehouse floors.
This helped move pallets of goods around, but humans still needed to pick items. Getting a robot to spot the correct product in a box, then grab it just hard enough to pick it up but not damage it — that's incredibly difficult.
This is where the Amazon Picking Challenge came in. Instead of hacking away at this problem itself, the company wanted to focus the broader academic community on it.
The risk was that any valuable inventions would be out in the public sphere and Amazon might not directly benefit from them. But the potential gains were much bigger, according to executives and roboticists.
"Amazon doesn't compete with robotics companies," said Brad Porter, a former Amazon Robotics chief who now runs a robotics startup called Cobot. "When facing an unsolved research problem in robotics AI, like bin picking, Amazon benefits if anyone solves that problem as long as Amazon can get access to the technology to improve their operations."
"The challenge Amazon was trying to solve was how to motivate researchers to focus on this problem," Porter added. "The Picking Challenge very much succeeded in doing that."
Oreos, Sharpies, and dog toys
The first competition took place over two days in late May 2015 in Seattle, with more than 25 teams from colleges including MIT, Duke, Rutgers, and Georgia Tech.
The contestants had to design a robot that could pick products from a typical shelf on a Kiva Systems warehouse pod and then put those items into containers. The picker had to be fully autonomous, and each robot was given 20 minutes to pick 12 target items from the shelves. Contestants had to open-source their creations.
Companies including ABB, Fanuc, and Rethink Robotics, founded by the industry pioneer Rodney Brooks, provided hardware for contestants to repurpose and tinker with.
The products were 25 items commonly sold on Amazon.com, including packs of Oreo cookies, boxes of Sharpie pens, and dog toys.
Some were easier to pick; there were simple cuboids, like a box of coffee stirrers or a whiteboard eraser. Others were trickier. For instance, a box of Cheez-Its couldn't be removed without first tilting it, adding another complex step for the robots. Smaller items, such as an individual spark plug, were more difficult to detect and grasp.
Vacuum arms and 'catastrophic failure'
A research paper analyzing the results said that among the 26 teams, a total of 36 correct items and seven incorrect items were picked. Another four were dropped by robots in the competition.
About half of the teams scored no points, and two teams couldn't get their robots working well enough to even attempt the challenge, the paper said.
Problems ranged from the highly technical to the mundane. Some items were packed in different ways, which made them even more difficult to pick. One team's machine had a vacuum hose that got accidentally wound around the robotic arm.
The researchers wrote that the failure of any one of a system's hundreds of components could lead to "catastrophic failure of the overall system — as witnessed during the competition."
The main finding from this first Amazon Picking Challenge was that human warehouse workers were a lot better than machines at picking products.
"A human is capable of performing a more complex version of the same task at a rate of ∼400 sorts/hour with minimal errors," the researchers wrote, "while the best robot in the APC achieved a rate of ∼30 sorts/hour with a 16% failure rate."
But the conclusion was hopeful, too: The contest showed that robotics could substantially increase warehouse automation and order fulfillment in the near future.
The following year the competition was renamed the Amazon Robotics Challenge, and the tasks evolved to be more complex.
Suction and other benefits
Tye Brady, the chief technologist at Amazon Robotics, was involved in the later Amazon Robotics Challenges.
In a recent interview with Business Insider, he said research on robotic manipulation exploded from 2016 through 2018, with many institutions publishing their results and insights. This helped spread valuable knowledge across the industry, speeding up progress.
At least two professors started graduate-level classes related to Amazon's challenge, and these programs are still churning out experts with valuable practical applied knowledge in robotics, Brady said.
"When you get a whole bunch of smart people together in a room and think about focused problems, some great things are going to happen, and that's really what happened," he said. "It inspired a lot of the work that we have today that we see in, for example, our Sparrow and Robin manipulation systems that are real-world products delivering packages inside of our fulfillment centers."
In that first competition in 2015, some robotics teams used grippers that mimicked the way a hand picks things up. Other teams tried suction, with some even strapping off-the-shelf vacuum cleaners to their robots.
Gripping proved more problematic because the robots didn't receive enough information to know when to release or add pressure at the right times. This could result in crushed products or dropped items.
Sucking the items up so they stuck to the end of robot arms was a more successful approach.
"The idea of high-flow suction was novel. Bring your favorite vacuum cleaner and start picking up objects. That was kind of clever," Brady said, adding: "We used suction inside of our Robin and our Sparrow arms. It's very good."
The boss has noticed
Amazon unveiled Robin, its first robotic arm, in 2021. This machine picks up packages from conveyor belts and places them on other mobile robots called Pegasus.
Sparrow followed in 2023. This was Amazon's first robotic arm to handle individual items rather than packages. It uses computer vision and AI to pick more than 200 million items from containers and place them in totes.
Amazon CEO Andy Jassy has taken notice. At the AWS re:Invent conference in December, he took time away from talking about cloud computing to wax lyrical about Sparrow.
"It has to discern which item is which. It has to know how to grasp that item, given the size of it and the materials and the flexibility of that material. And then it has to know where in the receiving bin it can put it," Jassy said. "These are all inventions that are critical to us changing the processing time and the cost to serve our customers."
Wall Street has noticed, too. Morgan Stanley recently estimated that Amazon's warehouse robots could save the company as much as $10 billion a year.
"The big story is we're just getting started," Brady said.
