A robotic apple picking system is beginning to take shape.

Scientists from Vision Robotics Corporation of San Diego, California, were in Washington State in October to test a prototype of what they call the "scout," a machine that detects and maps the apples on the trees so that a robotic harvester will be able to pick them.

The prototype scout has six pairs of cameras mounted on a vertical boom facing the trees, and the boom can be angled to suit the canopy. The cameras take still images of the canopy every four inches as the scout moves down the orchard row. The paired cameras allow the three-dimensional position of the fruit to be calculated and mapped. Ultimately, the data will be used by a robotic harvester attached to the scout to locate the fruit and plan the most efficient order in which to pick it.

The Washington Tree Fruit Research Commission is providing funding to develop the robotic harvester, though private investment will be needed to complete it. Vision Robotics is also working on a similar harvesting system for oranges, with funding from the California Citrus Research Board. Tim McConnell, Vision Robotics’s project manager for agricultural robots, said the orange scout uses the same software as the apple scout, but will physically be adapted to the larger orange trees.

One of the greatest challenges in developing the scout has been making it suitable for use outdoors and dealing with bright sunlight, dust, and uneven ground, McConnell said. "We’ve done a fair amount of robotic work for systems that work in an indoor, controlled environment."

The company’s first commercial product, a robotic vacuum cleaner, will go on sale next year.

Accuracy

The prototype apple scout was put to the test in several Washington orchards with different types of tree canopies—some angled and some upright—and different colors of apples, including red Fuji, poorly colored Fuji, and Winter Banana.

The scout took images of sections of a tree row while Karen Lewis, Washington State University Extension educator, and Tree Fruit Research Commission staff picked and counted apples from the same sections to check the accuracy of the scout. Was it missing any apples or counting some twice? Eventually, before the fruit can be picked robotically, the scout will also need to be able to identify branches and wires that might be obstacles to the picking arms as they reach into the canopy for fruit. It must also be able to identify the color and size of the fruit so that the crop can be picked selectively.

Vision Robotics is focusing first on developing the scout, which has the potential to do far more than just tell the harvester where the fruit is. It could, for example, count green fruit early in the season so the crop can be thinned appropriately. In the summer, it could count the fruit and calculate the size to generate an accurate crop estimate.

Lewis noted that currently the size of Washington’s apple crop is calculated by different people in different ways. The scout could help standardize crop forecasting industrywide, as well as help individual growers assess their crops.

"There’s tremendous interest on the part of the marketing organizations to improve crop load assessment in June and July," agreed Tom Auvil, research horticulturist with the Research Commission.

Tory Schmidt, horticultural associate with the Research Commission, foresees the scout also being used to look for mildew, codling moth strikes, or leafroller damage.

The scout will be connected to an outside monitoring or computer system so that the grower can use the information as a decision aid.

Granny Smith

One of the objectives of the tests in Washington this fall was to capture as much data as possible with the cameras and determine how many images are needed to find the fruit on the tree, including those that are partially obscured by branches or leaves.

Next year’s focus will be on ensuring that the scout can detect apples that are not so easily distinguishable from the foliage, such as Granny Smith.

The final version of the scout will have cameras mounted on each side and will have lighting so that it can operate day and night. It will be towed by a tractor, though eventually it could be self propelled and guided by a Global Positioning System.

Once the scout system is proven to work, the company will focus on the robotic harvester. A group of senior students at Olin College of Engineering in Boston, Massachusetts, is already working to develop the picking hand (known as the end effector). McConnell said inventing a picking hand that can pick the fruit efficiently without damaging it is one of the most challenging aspects of the project, which is the reason it’s being tackled first. "Mechanically, the hand is one of the hardest things," he said.

By next year, Vision Robotics hopes to be working on the picking arm and has great confidence that it will be able to develop an arm that can reach all the fruit, McConnell said.

The harvester will have multiple picking arms with picking hands on the ends. The arms will reach out towards the fruit, based on data from the scout, and cameras on the picking hands will guide the hands more precisely as they locate and pick the fruit.

Derek Morikawa, chief executive officer at Vision Robotics, said that in the initial design the harvester has eight arms and will be able to pick apples at a rate of about three per second. However, as the system is refined, it might have up to ten arms—possibly one for each wire of the trellis on each side.

Investors

Morikawa is interviewing companies interested in manufacturing the agricultural products that Vision Robotics is developing, which include a robotic pruner for wine grapes. He is putting together a consortium of investors from the wine industry to commercialize the robotic pruner and next year will be looking for private investors in the tree fruit industry to move the apple scout and harvesting system towards commercialization. Once it’s funded, the scout will take another year to complete as a field-ready product that can be used for crop monitoring and estimating. The harvester will take about three years to develop.

Morikawa believes the timing is right for such developments because technology is reaching the point where it is practical to build such robots, and agricultural producers need to address labor shortages.

In apples, there’s also a need for technology that can help growers improve productivity and fruit quality in order to be more competitive. "It will allow growers to adopt more progressive practices—things that they might be hesitant to do now because of the amount of labor it takes," he said.

The cost of a scout and harvesting system to the grower is likely to be about $500,000, McConnell estimated, but it should pay for itself within three years.

To be successful in the future, growers will need to invest in mechanical systems so they are not at the whim of the labor force and immigration issues, he said, and the systems are becoming more and more capable as time goes on.

"Mechanical systems are in the future," is his message to growers. "You really need to get on board."