It’s no coincidence that the pruning cuts successfully completed by a robot in a tree fruit orchard this spring were made in a UFO-trained cherry block — the structured system offers simple pruning rules for a robot to follow.
Pruning robots have been on growers’ wish lists for about as long as automated harvest has, but researchers pursuing such technology have found it to be a complex problem both in terms of the algorithms to analyze tree systems and make pruning decisions and the hardware that can navigate the canopy to make cuts. Simplifying the trees — as in a UFO system — makes a simpler first test for the robotic approach.
“This is the first time we feel like we’re in a good position to cut branches outside,” said Joe Davidson, an assistant professor of mechanical engineering and robotics at Oregon State University and co-leader of the effort.
At Washington State University’s Roza research orchard in Prosser in March, Good Fruit Grower watched the robotic arm’s cameras scan the trees, looking for a candidate branch that fits its cutting criteria. With a target branch in sight, the arm advances, holding the pruning shears open as its vision system determines the best path toward the target. Its slow, methodical movements are recorded by graduate students tasked with tracking iterations of improvement. Once a force-sensor inside the shears registers that the branch is in position: snap.
One seemingly simple cut represents years of work. Graduate students celebrated.
Teaching a robot how to prune a fruit tree turns out to be quite complicated. Compared to the specific goal of robotic apple harvest, the goals of pruning can be varied and complex.
“If everyone had a picking bag, we’d all do pretty much the same thing; but we have seen that in apple orchards, two expert pruners may do it totally differently,” said Matt Whiting, a WSU horticulture professor who developed the UFO training system to try to simplify and standardize pruning. The rules are simple for people and robots. “That’s the idea behind the architecture … to move from artwork to something scientifically based and repeatable.”
That approach provided the perfect training ground for the pruning robot WSU engineers and their collaborators at OSU and Carnegie Mellon University have been working on for years, said Manoj Karkee, a WSU professor of biological systems engineering. He started off using cameras to record people pruning, with the goal of developing pruning rules that artificial intelligence could learn from watching the people. However, he found that each person approached the task quite differently.
Karkee started to explore the concept of robotic pruning almost a decade ago, but the current collaboration started with grant support from the Washington Tree Fruit Research Commission in 2019. A U.S. Department of Agriculture grant in 2020 supported the next step. And starting this year, the creation of the AgAID Institute, a $20 million investment in ag technology research led by WSU, aims to accelerate the development of the pruning robot, among other research goals.
Across the country, engineering students in Long He’s lab at Penn State University are also working on aspects of robotic pruning, including how, in a complex canopy, to approach branches with a cutting tool and vision systems to count buds, but they have yet to put all the pieces together.
The science behind the shears
Previous approaches tried to create 3D models of tree architecture from imagery on which the robots would base pruning decisions, but those time-consuming computations make it difficult for a robot to work in real time. This is where the simplified UFO system becomes an advantage, as the robot just works its way down one leader and up the next.
“We don’t need to see the entire tree to make pruning decisions,” Davidson said.
The cameras have a 2-foot-square field of view, but in the UFO system, that narrow view is enough, Whiting said. The robot only has one pruning rule to follow: Take off the laterals to maintain the structure of the narrow, upright fruiting offshoots — for which the system is named. (The second rule, select new shoots for renewal, isn’t yet under the purview of robotics).
To make those decisions, the vision system must first discern the leader that’s in front from the branches in the rows behind. As the camera scans, the algorithm differentiates between the foreground and background by the way the pixels move and then subtracts out the background noise, Davidson said. Then, the image is segmented to define the key features such as the leader, the trellis wire and the laterals.
Once the system recognizes a branch that needs to be cut (in the UFO system, that’s any branch) it determines a path to approach it. The robot’s AI trained for this pathfinding in a custom-built virtual orchard in which the robot is “rewarded” for getting closer and closer to a target branch. Think of it like a game of Marco Polo, Davidson said, or like telling a child “warmer … warmer … warmer!” on a treasure hunt. Thanks to all that practice, the AI can decide how to approach each new target.
“There’s a million ways to reach a branch,” Karkee said, some more efficient than others. The new AI institute will support more computer scientists joining the effort, helping to optimize the computations for tasks such as this path planning — and speeding up the process.
Similarly, the Penn State team is working on more efficient ways to do the path planning necessary for their team’s robotic arm to reach the branch, which currently takes 20 or 30 seconds, so that it can move more quickly, He said.
Research potential
Watching the robot scan up and down the canopy, Davidson, Karkee and Whiting agreed that it’s a moment to both celebrate how far the technology has come — from idea to implementation — and how far it still must go to be a viable commercial tool.
“We’ve got a lot of pieces built,” Davidson said, referring to both the mechanical cutting arm and the algorithms that underlie it. “Our goal is to prove the algorithms and controllers and build out something more robust.”
Optimizing the computations should help with the speed, Karkee said, and eventually he envisions multiple pruning arms running on one machine. Fresh Fruit Robotics, an Israeli company developing an apple harvesting robot, has expressed interest in someday swapping its picking arms for pruning ones to get more utility out of the machine. But the research will offer value to the fruit industry long before the technology is commercialized, Whiting said.
“One of the exciting parts is the ability to evaluate pruning treatments,” he said.
People never prune perfectly — even in his own UFO block he pointed out mistakes and inconsistencies. Horticulturists would like to be able to better measure the effects of different pruning approaches, especially in apple systems, but “it’s hard to have a crew of 30 people make a small change consistently,” he said.
The robot can, and it will do so in orchards where Whiting is collecting data on the trees year after year to understand how the pruning decisions impact yield and quality. In this way, the technology collaboration will create a feedback loop to understand “how we should be pruning,” he said.
—by Kate Prengaman
Leave A Comment