Why is there almost no manipulation data for agriculture?

🤗Datasets
1

I farm, and I’ve been going through the robotics datasets on the Hub for a few months trying to find anything useful. Almost everything is indoor tabletop. Open X-Embodiment, DROID, most of what’s under LeRobot — pick up the mug, open the drawer, fold the cloth, put the block in the bowl. Kitchens and desks.

Which is strange, because the manipulation problems in food production are harder in ways that seem interesting rather than just annoying:

The target is deformable and every instance is different. A mug is a mug. No two strawberry plants have the same geometry, and the same plant is a different problem in week 2 versus week 5 of the season.

Damage is often invisible at grasp time. Bruising shows up days later in the packhouse or at the retailer. So your reward signal is delayed by days and measured somewhere else entirely, which I don’t think any current dataset format handles.

You have to decide whether to pick before you pick. Ripeness judgment is folded into the manipulation policy rather than being a separate upstream classifier. Same for pruning — most of the skill is choosing which cane to cut.

Heavy occlusion, and it’s not incidental. Fruit sits behind leaves by default. Half the skill is moving foliage without damaging the plant.

Lighting varies more in one afternoon in a field than across an entire indoor dataset.

So my question is really two questions. First, is this an absence of interest or an absence of data — i.e. are people trying and failing, or has nobody bothered? Second, if someone were to build an open ag manipulation dataset properly, what should it look like? Specifically:

  • Do you record failures, and how do you label a failure whose consequence appears three days later
  • Is per-episode metadata enough for season stage, cultivar, weather, or does that need to be per-frame
  • Teleop demonstrations from a skilled picker versus scripted trajectories, given that a good human picker is very fast and hard to imitate
  • Does anything transfer from the household datasets, or is deformable-occluded-outdoor different enough that it starts from zero

I have access to a working farm and a packhouse and I’m willing to put real collection effort behind this if people think the format is worth getting right. Would rather ask before building something nobody can use.

2

Hmm… From what I could find, it seems that the relevant work is scattered across several fields and is therefore hard to discover:

My read is that this is not simply an absence of interest. People are collecting agricultural manipulation demonstrations and testing learned policies, but the public work is fragmented into small, crop-specific, hardware-specific projects rather than forming an agricultural equivalent of DROID or Open X-Embodiment.

The second part of your question is also slightly different from “no current format can represent this.” LeRobotDataset v3 can already store synchronized video, state, action, timestamps and custom features. The missing piece appears to be a broadly used convention that links:

  1. the robot trajectory,
  2. the particular fruit/plant/container involved,
  3. later packhouse or storage events,
  4. a quality observation made hours or days later,
  5. and the uncertainty about what actually caused that outcome.

I would therefore use LeRobot as the trajectory layer, rather than trying to force the entire farm-to-packhouse history into one episode table.

A reasonable default design would be:

immutable raw recording
        ↓
LeRobot trajectory dataset
        │
        └── harvest_event_id / target_id
                    ↓
linked event and outcome table
        ├── harvest / container / packing events
        ├── inspection time
        ├── measured quality
        ├── inspection method and scale
        ├── missing-data reason
        └── causal attribution, if any, kept separate

Before collecting at scale, I think the highest-value route would be to take a one-crop, one-task pilot schema to the LeRobot Discord. That is probably the best place to check action representation, custom features, converter design and whether the resulting dataset will work with current policies.

The Hugging Science community also seems relevant, but for a different reason: it explicitly focuses on cross-disciplinary collaboration, data fragmentation, standardized formats and shared evaluation. That may be a better place to find plant science, postharvest, data-curation or benchmark collaborators.

The most important design question to settle first is probably:

What identity can remain attached to the harvest event through the packhouse?

What remains traceable? What the later label can safely mean
Individual fruit The later observation can be joined directly to one harvest event
Tray, bin or small lot The later result should remain an aggregate outcome, not be assigned to one grasp
Only a large mixed lot Useful for operational statistics, but weak supervision for individual trajectories
No persistent linkage Start with immediate outcomes, or modify the tagging workflow before collecting delayed labels

Everything else—including whether weather belongs per-frame or per-episode—depends somewhat on that identity boundary.

Closest examples I could find

RoboHarvest: a currently accessible public example

The clearest public agricultural manipulation dataset I found is RoboHarvest, for outdoor pepper harvesting.

It reports and releases:

  • 300 harvesting demonstrations,
  • an external camera and an in-hand camera,
  • a custom handheld shear-gripper,
  • preprocessing code,
  • RGB observations,
  • 6D gripper poses,
  • and gripper actuation.

This is quite close to what you describe, but it also illustrates the fragmentation problem: the data and action representation are tightly connected to one crop, one collection device and one cutting mechanism.

The associated policy achieved a relatively modest field success rate, which is also evidence that the bottleneck is not merely “nobody tried.” Real outdoor execution remains difficult even after collecting task-specific demonstrations.

SROI: probably the closest collection pipeline, but not a currently verifiable dataset release

The SROI paper describes a portable strawberry-operation interface for gathering expert demonstrations in real growing environments.

The current SROI data-processing repository is especially relevant because it implements much of a practical collection pipeline:

  • direct RealSense recording or legacy ROS bag ingestion,
  • camera calibration handling,
  • ORB-SLAM trajectory extraction,
  • gripper-state extraction,
  • visual QC,
  • conversion into LeRobot,
  • optional inverse-kinematics feasibility reports,
  • and dataset merge/provenance guidance.

That makes it the closest implementation reference I found for:

field capture → processing → QC → LeRobot

There is an important limitation, however. The paper reports an open dataset, but I could not locate a working public download for the actual demonstrations. There is currently an open issue reporting that the dataset link returns 404.

So I would treat SROI as a valuable hardware and processing reference, but not currently as an independently downloadable and auditable dataset.

The repository also correctly warns that an inverse-kinematics pass is only a necessary heuristic: it does not prove that a human-recorded trajectory is dynamically executable by the robot.

HarvestFlex: evidence for transfer and stage-based evaluation

A very recent paper, HarvestFlex, integrates LeRobot with a greenhouse strawberry harvester and collects 227 long-horizon VR-teleoperated episodes.

The study is relevant because it:

  • fine-tunes general VLA models on agricultural demonstrations,
  • evaluates full task execution rather than only reaching,
  • includes retries,
  • reports fruit damage as well as success,
  • and divides harvesting into stages such as target selection, approach, detachment, placement and retry/reset.

The strongest model reached a reported 74% success rate under its evaluation protocol after agricultural fine-tuning.

I could not confirm a public dataset or code release for HarvestFlex, so I would use it as evidence about transfer, evaluation and task decomposition, not as an available open dataset.

General robotics datasets explain part of the contrast

DROID contains roughly 76,000 demonstrations across hundreds of indoor scenes, but its authors also emphasize how much logistics, hardware, labour and safety work was required to collect diverse real-world robot data.

Open X-Embodiment shows that cross-robot data can produce positive transfer when large, heterogeneous datasets are standardized.

Together, they suggest that large-scale agricultural data is not obviously impossible, but agriculture adds several costs that indoor collection largely avoids:

  • seasonal collection windows,
  • biological change over time,
  • non-resettable or destructive actions,
  • crop-specific end effectors,
  • outdoor sensing variability,
  • and outcomes that may only become measurable after the robot episode has ended.

I do not know of a study that quantitatively separates the contribution of each of these factors, so I would treat this as a plausible explanation rather than a proven decomposition of the shortage.

How I would divide the data

I would avoid making every field either “per-frame” or “per-episode.” There are at least four useful timescales.

1. Frame-level trajectory data

These are synchronized with robot execution:

  • timestamps,
  • camera images,
  • robot state,
  • robot action,
  • gripper/cutter/suction command,
  • force, tactile or pressure signals if available,
  • target-relative pose estimates,
  • safety clipping or intervention indicators,
  • rapidly changing lighting or wind measurements, if actually sensed at useful frequency.

LeRobot is a natural fit for this layer.

2. Episode or collection-session context

These usually change slowly:

  • crop and cultivar,
  • farm, plot, row or greenhouse compartment,
  • plant identifier,
  • growth or season stage,
  • date and time,
  • robot and end-effector configuration,
  • operator or teleoperation mode,
  • camera and sensor calibration versions,
  • weather snapshot,
  • collection protocol version,
  • software and converter versions.

I would not duplicate static cultivar or calibration data into every frame unless a downstream tool absolutely requires it.

Weather is probably best represented as:

  • a session/episode snapshot for slow-changing conditions,
  • plus timestamped events or a separate sensor stream if conditions change materially during collection.

3. Subtask and physical-event annotations

A single success = true/false loses too much information for harvesting.

Possible stages include:

  • target considered,
  • target selected,
  • approach,
  • view adjustment,
  • foliage interaction,
  • grasp/envelop/cut,
  • detach,
  • retreat,
  • transport,
  • placement,
  • retry,
  • abandon target,
  • human recovery.

Current LeRobot documentation includes frame-level subtask support, although that page currently describes main-branch functionality, so I would pin a LeRobot version before building a permanent schema around its exact file layout.

The LeRobot annotation pipeline can also add language descriptions and automatically proposed subtasks from video. That may be useful for search or policy conditioning, but I would not use a VLM annotation as ground truth for physical events such as successful detachment, plant damage or later bruising. Those should come from logs, sensors or human inspection.

4. External postharvest events and outcomes

These may occur after the robot episode and in a different facility:

  • receiving at packhouse,
  • transfer into a tray, bin or lot,
  • sorting or grading,
  • cooling,
  • packing,
  • transport,
  • inspection after one day,
  • inspection after three or five days,
  • retailer rejection or quality downgrade.

This is where I would use a linked table rather than trying to stretch the episode timeline over several days.

A minimal record might look like:

harvest_event_id: harvest_2026_07_28_000184
tracked_unit_type: individual_fruit
tracked_unit_id: fruit_000184

observed_at: 2026-07-31T09:20:00+09:00
inspection_stage: packhouse_followup

trait: visible_bruising
method: visual_inspection_protocol_v1
scale: ordinal_0_to_3
value: 2

observation_status: observed
attribution: unknown
attribution_confidence: not_assessed

The important point is that these are different statements:

Bruising was observed three days later.

and:

The robot grasp caused the bruising.

The first is an observation. The second is a causal claim and may require controls for transport, temperature, fruit-to-fruit contact, sorting and pre-existing damage.

Existing agricultural and traceability vocabularies

I would not design every metadata field from scratch, although I also would not require a first pilot to implement several large standards completely.

There are useful pieces in adjacent fields.

MIAPPE

MIAPPE is a plant-phenotyping metadata standard. Its concepts include:

  • Study,
  • Biological Material,
  • Observation Unit,
  • and Observed Variable.

Its observed-variable model is especially useful because it separates:

  • what was measured,
  • how it was measured,
  • and the unit or scale used.

That is a better basis for a delayed quality label than an undocumented field such as bruise = 1.

Crop Ontology

Crop Ontology uses a similar:

trait + method + scale

model.

For example:

trait: visible fruit bruising
method: visual inspection after storage
scale: ordinal severity 0–3

This makes the observation interpretable even if different projects use different sensors or grading systems.

It does not solve causal attribution, timestamps or robot actions, but it can prevent the quality-label vocabulary from becoming project-specific.

GS1 produce traceability

The GS1 Fresh Fruit and Vegetable Traceability Guideline is useful for thinking about events outside the robot dataset.

It describes maintaining traceability across harvest, packing and later supply-chain events using identifiers and event records.

This is normally oriented toward trade items, logistics units, batches and lots—not necessarily a single strawberry—so it does not automatically solve fruit-level linkage. It does, however, provide useful language for:

  • tracked units,
  • harvest information,
  • packing events,
  • lot assignment,
  • and preserving traceability through the packhouse.

My preference would be to borrow the parts that help define the join, rather than making the robot dataset itself an implementation of a complete supply-chain standard.

Failures, teleoperation and recovery

Record failures, but separate several kinds of failure

I would record at least:

Execution outcome

  • target not reached,
  • collision or unsafe contact,
  • empty grasp,
  • cut missed,
  • fruit slipped,
  • detachment failed,
  • fruit dropped,
  • adjacent fruit contacted,
  • plant damaged visibly,
  • timeout,
  • human abort.

Task-decision outcome

  • unripe target selected,
  • ripe target missed,
  • unreachable target selected,
  • wrong cane or stem selected,
  • target abandoned appropriately,
  • target switched.

Delayed outcome

  • bruising observed,
  • skin damage,
  • leakage,
  • decay,
  • grade reduction,
  • shelf-life reduction.

Missingness

  • item not inspected,
  • identifier lost,
  • item mixed into another lot,
  • inspection skipped,
  • image unusable,
  • outcome unavailable.

Not inspected must not become equivalent to no damage.

Skilled-picker demonstrations versus scripted trajectories

I do not think this needs to be a single choice.

Source Main value Main limitation
Skilled handheld demonstration Captures fast human target selection and approach strategy Embodiment and dynamic-feasibility gap
Robot teleoperation Produces actions the robot can actually execute Slower and may not reproduce expert human motion
Scripted trajectories Useful for calibration, coverage and controlled comparisons Does not capture much of expert crop-handling skill
Autonomous rollout Shows the policy’s actual state distribution Includes policy-specific failures and biases
Human intervention/recovery Captures how to escape failure states Requires a deployable initial policy and safe takeover

The LeRobot human-in-the-loop workflow is relevant here. It records autonomous execution together with human recovery and correction inside the same continuous episode.

That seems especially valuable for agriculture, where an autonomous policy may:

  • choose a bad approach angle,
  • become occluded near the fruit,
  • partially detach a target,
  • disturb foliage,
  • or enter a state that does not appear in clean expert demonstrations.

A sensible progression might be:

expert demonstrations
        ↓
initial policy
        ↓
autonomous field rollouts
        ↓
human intervention and recovery data
        ↓
fine-tune and repeat

Scripted data can remain useful as a control and for testing the collection pipeline, but I would not treat it as a substitute for skilled interaction data.

Does household or generalist robot data transfer?

I would not assume either extreme.

Open X-Embodiment provides evidence that heterogeneous robot data can produce positive cross-robot transfer. However, that does not show that a household-trained policy will perform agricultural harvesting without adaptation.

HarvestFlex is a more direct clue: general VLA policies could be adapted to strawberry harvesting, but the study still required real agricultural demonstrations and fine-tuning. Its remaining failures included severe occlusion, close-range observability loss and contact-dynamics mismatch.

So I would frame the question experimentally:

Baseline What it tests
General pretrained model + agricultural fine-tuning Whether broad pretraining improves sample efficiency
Same architecture trained only on agricultural data How much the prior contributes
Frozen backbone or LoRA adaptation Whether cheap adaptation is enough
Full fine-tuning Upper practical adaptation baseline
Crop-specific conventional policy Whether the generalist approach is actually competitive

The held-out evaluation conditions may matter more than the headline training loss.

Useful splits might include:

  • unseen plant or row,
  • unseen collection date,
  • unseen growth stage,
  • unseen cultivar,
  • unseen farm,
  • unseen lighting/weather condition,
  • or unseen operator/collection device.

A random frame split would probably leak nearly identical plants, backgrounds and episodes between training and test.

A low-risk pilot before large-scale collection

I would make the first pilot deliberately small enough to discard and rebuild.

Phase 1: verify the data contract

Collect a small number of episodes for:

  • one crop,
  • one task,
  • one robot or collection device,
  • one action representation,
  • and one clearly documented protocol.

Keep the raw recordings immutable.

For each collection run, preserve:

  • calibration,
  • clock source,
  • sensor configuration,
  • robot and end-effector versions,
  • raw-to-LeRobot converter commit,
  • LeRobot version,
  • and dataset revision.

Phase 2: verify the complete round trip

Do not stop when the files have been written.

A useful end-to-end check would be:

  1. record raw data;
  2. convert to LeRobot;
  3. finalize the dataset;
  4. visually inspect complete episodes;
  5. run automatic timestamp, shape and range checks;
  6. load it through LeRobotDataset;
  7. construct one training batch with the intended policy;
  8. run a small training smoke test;
  9. push it to the Hub;
  10. pull it back using a pinned revision or commit;
  11. join the external outcome table by stable ID;
  12. reproduce the intended train/test split.

This distinguishes three different claims:

The data can be stored.
The data can be loaded by LeRobot.
The data can actually train the intended policy.

Those are not always equivalent when a dataset uses custom state/action fields.

Phase 3: test delayed linkage

Include a small number of tracked units with:

  • immediate inspection,
  • one later inspection,
  • at least one deliberately missing outcome,
  • and at least one tray/bin/lot-level case if those occur operationally.

That should expose most identity and missing-data problems before a full season of collection is committed.

Phase 4: ask the community to review an example, not only a schema

A very small public sample is likely to produce better feedback than an abstract field list.

The most useful package to show the LeRobot community would probably contain:

  • one or two complete episodes,
  • the raw schema,
  • the converted LeRobot schema,
  • an example delayed-outcome row,
  • the intended action representation,
  • the intended baseline policy,
  • and a short note describing what can and cannot remain traceable in the packhouse.
A compact default schema

This is only a starting point, not a proposed universal standard.

dataset:
  schema_version: ag_manipulation_pilot_v0
  lerobot_version: pinned_version
  converter_commit: git_commit
  collection_protocol: protocol_v1

session:
  farm_id: pseudonymous_id
  plot_or_row_id: row_12
  crop: strawberry
  cultivar: cultivar_id
  growth_stage: stage_term
  collection_datetime: timestamp
  weather_snapshot_id: weather_004
  robot_config_id: robot_config_v2
  end_effector_config_id: gripper_v3
  calibration_id: calibration_2026_07_28

episode:
  episode_id: ep_000184
  harvest_event_id: harvest_000184
  plant_id: plant_032
  target_id: fruit_184
  task: select_and_harvest_ripe_fruit
  control_source: robot_teleoperation
  immediate_outcome: detached_and_placed
  intervention_count: 1

frame:
  timestamp: seconds
  observation.images.scene_left: image
  observation.images.scene_right: image
  observation.images.wrist: image
  observation.state: vector
  action: vector
  gripper_state: value
  subtask: approach
  controller_source: autonomous
  safety_event: none

external_event:
  harvest_event_id: harvest_000184
  tracked_unit_type: individual_fruit
  tracked_unit_id: fruit_184
  event_type: quality_inspection
  observed_at: timestamp
  trait: visible_bruising
  method: inspection_protocol_v1
  scale: ordinal_0_to_3
  value: 1
  observation_status: observed
  attribution: unknown
  attribution_confidence: not_assessed

The core idea is that harvest_event_id remains the bridge, while each layer retains its own timescale and semantics.

Practical recommendation

If I were starting from your position, my default path would be:

  1. choose one crop and one narrowly defined task;
  2. determine whether fruit-, container- or lot-level identity can survive through the packhouse;
  3. record the robot trajectory in LeRobot;
  4. keep delayed outcomes in a linked event table;
  5. separate observation, missingness and causal attribution;
  6. retain successful demonstrations, immediate failures and later recovery data;
  7. validate the complete Hub/training/join round trip on a small pilot;
  8. show that pilot to the LeRobot Discord before scaling collection;
  9. use Hugging Science or the relevant plant/postharvest communities to review the biological metadata and quality protocol.

The combination of farm and packhouse access is particularly valuable because it could capture a linkage that most current robot datasets cannot: not merely whether the robot completed the motion, but what happened to the biological product afterward.

Even a modest dataset that preserves that linkage cleanly could be more reusable than a much larger collection of trajectories with only a binary success label.