Perception Systems Guide the Path to a Fully Autonomous Tractor
Automation is making its way into many applications, from the smallest of components and systems to large pieces of heavy-duty mobile equipment. It can take many forms such as controlling a single movement of a part or enabling an entire vehicle to drive without interaction of a human being.
The technologies which make automation possible are varied; sensors and software are key among them as well as perception systems.
For John Deere’s autonomous 8R tractor unveiled at the Consumer Electronics Show (CES) 2022, Noah Schwartz, lead perception engineer at Bear Flag Robotics—an autonomous agriculture technology company acquired by John Deere in 2021—said the company leveraged many of its existing automation technologies. These include StarFire GPS, which provides reliable localization to enable the tractor to know where it is at when traversing the field. Such technology is vital to ensuring safe operation of the fully autonomous tractor.
Robust, Reliable Technology Development
In addition to its StarFire GPS technology, John Deere utilizes a robust control stack for path planning. “It handles all of the navigation and obstacle intelligence [necessary] to get through the field in a safe and effective way,” says Schwartz.
The autonomous 8R’s perception system feeds into this control stack to ensure safe movement through a field. It is comprised of six pairs of stereo cameras which provide a 360-deg. view around the tractor. The cameras are able to capture the environment surrounding the tractor and send data about it to the control stack, ensuring the safest path is chosen.
Not only does the perception system need to monitor the tractor’s surroundings, but also the implement being pulled by the tractor, the health of the implement health as well as the field state. All of this information needs to be fed to the tractor so it can appropriately determine how best to do its job. As such, Schwartz says there are a lot of robust AI (artificial intelligence), models and algorithms built into the system to ensure the tractor has a constant feed of high-quality information about what is going on in its environment.
Schwartz goes on to say the company relied on the use of deep learning and computer vision for its perception system because of their ability to accurately take in the amount of information necessary for this application. “It’s really rich technology, and we built models specifically to work with the cameras,” which are collecting full color imagery as well as near infrared and depth information.
He notes there is a suite of sensors currently undergoing research and development which John Deere plans to bring to market as necessary and as it brings automation to more agricultural equipment.
Perception and autonomy in agriculture is less like autonomous driving through a city and more like perception on Mars or the moon, says Schwartz. “We have sparse environments where there is not a lot of information, or it is the same thing over and over again. How do we develop models that are sensitive to things, because once they do detect [an object] it is a big deal. There should not be anything [in a farm field] but if there is we need to deal with it correctly,” he said.
Agricultural environments are harsh, which can also present challenges when developing perception systems for autonomous equipment. Farmers spend about a third of the day driving into the horizon and into the sun; they may also be operating at night or in fog or dust and in a variety of temperatures. “We have to make sure the system is robust and reliable throughout all of it,” says Schwartz.
He says it has been a fun challenge for the design team to determine the best sensors, models and techniques to utilize to ensure a robust and accurate system is developed. “It has been an amazing journey, and I'm really proud of what's coming out in the fall.”
The autonomous 8R tractor is scheduled to be available to customers starting in fall 2022.
Autonomy’s Benefits in Agriculture
Like many industries, agriculture is facing a shortage of skilled workers entering the market. This is causing many challenges for farmers, many of whom are already working long days. And as many start to reach retirement age, there will need to be younger generations entering the field to ensure the growing global population can continue to be fed.
It is estimated the global population will reach close to 10 billion people by 2050, increasing the demand for food 50%. This will put a lot of strain on the farming industry.
Autonomy is seen as a way to help overcome these challenges. A fully autonomous machine could be completing a task in the field, freeing up a farmer’s time to focus on other necessary tasks. Use of autonomous equipment enables them to do more with less. “There are always competing priorities [on a farm],” said Deanna Kovar, VP of Production and Precision Ag Production Systems at John Deere, during the company’s press conference unveiling the autonomous 8R at CES 2022. “And if you don't manage them, they're likely to not get done well or not get done at all.”
The autonomous tractor can help with this as it can run 24 hours a day if necessary, greatly benefiting a farmer’s productivity. In addition, autonomy can help ensure work is done correctly and efficiently as the precision needed to do so is built into the machine and there is no chance of human error. And like an assembly line, the autonomous tractor is able to perform the same operation repeatedly and precisely; no matter the conditions, the operation is done the same every time says Schwartz.
To start, John Deere’s autonomous 8R will focus on tillage. This application helps prepare fields for planting which is integral to the growing season. “It is a core operation and the market suggested that was a great place for us to start,” says Schwartz.
He notes the company is working with its other divisions—specifically construction and forestry—to evaluate and develop autonomous systems for those industries as well, as they are also facing similar labor challenges and need for increased productivity.
In the near term, Schwartz says John Deere is focusing on getting the autonomous 8R into customers hands and scaling out the technology as much as possible. There is a roadmap for expanding the autonomous system out to other tillage implements and eventually other types of agricultural equipment.
Take a deeper dive into some of the technology utilized in the 8R tractor by watching Farming AI: Self-Driving Tractors from our colleagues at Electronic Design who spoke with Gaurav Bansal of John Deere's Blue River Technology group about the electronics, machine learning and other solutions used to enable fully autonomous operation of the tractor.
