Chili Ripeness Model with YOLOv11

Every image comes from a camera pointed at real plants. 526 were shot in a home garden where the plants were raised, from straight-on and angled-down viewpoints in the morning, midday and afternoon. 208 came from the gantry's own top-down camera in the greenhouse, including evening shots. Healthy plants rarely have rotten fruit, so 195 images from a public Roboflow set of diseased chilies filled in the broken class.

Footage was recorded as 1080p video at 30 fps and sampled at two frames per second, then cleaned of blurry, duplicate and chili-free frames. That left 929 images, labelled by hand in Roboflow with 28,263 bounding boxes: about 30 fruits per image. The classes are far from balanced (16,907 unripe, 8,209 ripe, 2,767 turning, 380 broken), which is simply what a plant looks like mid-season.

The images were split 70/20/10 into 650 training, 186 validation and 93 test images, and the training set was tripled to 1,950 with 90° rotations, light noise and horizontal flips.

Run 1 was a baseline with Ultralytics defaults. Its mAP@50 and mAP@50-95 sat 0.285 apart, a sign of sloppy boxes: at 640 px a single frame packs 40-plus fruits into very few pixels each. Raising the resolution to 960 px in Run 2 lifted both numbers, but the curves plateaued by epoch 123 with the learning rate stuck where it started.

Run 3 swapped in AdamW at a lower learning rate with cosine decay and a longer warmup. Training stayed stable to the last epoch, but gained only 0.012. That pointed at the nano model's capacity, not the optimiser, as the limit.

Run 4 moved to YOLOv11s at 1280 px and tightened regularisation (more weight decay, mixup, a heavier classification loss, a longer warmup and close-mosaic window) so the bigger network wouldn't overfit while learning from scratch. It peaked between epochs 108 and 127 and early-stopped at 177, after about 4 hours on two Tesla T4s on Kaggle.

Three classes are solid. Turning isn't. The confusion matrix gets it right 71% of the time, with the rest mostly lost to background or called unripe: a fruit that's half green and half orange sits between two classes and, from above, looks a lot like the leaves around it. The background row shows the same thing in reverse, with leaves, stems and shadows mistaken for unripe fruit.

The F1-confidence curve sets the deployed threshold. F1 peaks at 0.80 at a confidence of 0.489 and stays flat between about 0.1 and 0.7. Precision only reaches 1.0 at 0.959, by which point recall has collapsed, so 0.489 is what the Pi uses.

YOLOv8s and Faster R-CNN, trained on the same data, both show the same weakness, at 0.726 and 0.600 mAP@50 for turning. That points at the data, not the architecture.

Before going near the hardware, the model was checked image by image, with every fruit counted by hand. Six scenes covered a top-down view in the greenhouse, a side-on view at plant height, a close-up, a plant outside the greenhouse, a dense frame from the gantry camera and a frame built around rotten fruit.

Across 257 detections it scored 94.9% precision and 87.5% recall. Side-on views hurt recall most (82.1%), since leaves hide more fruit. The gantry view caught every fruit but over-detected a little (85.3% precision), and the outside-the-greenhouse plant scored 95.8% on both, so the model isn't tied to one setting.

All three were trained from scratch on the same data and hardware, with hyperparameters matched as closely as each architecture allowed. YOLOv8s came close: slightly higher recall (0.799), slightly lower mAP across the board, and 25 ms slower on the Pi despite having more parameters (about 11.2M against 9.4M).

Faster R-CNN was the outlier. Each epoch took about 372 seconds against roughly 80 for the YOLO models, and the run hit Kaggle's 12-hour session cap at epoch 106. Its precision of 0.560 against a recall of 0.796 shows a model that over-predicts, and at 1.65 s per image on the Pi it's too slow for a scan that already spends its time moving.

The real test was a week of scans in the greenhouse, 1 to 7 July 2026. Before a scan, every fruit on all 27 plants was counted by hand: 950 fruits in total. The model found 863 of them, with 52 false detections and 87 misses, for 94.3% precision and 90.8% recall. Misses outnumber false alarms, which fits what the camera sees: fruit tucked behind leaves.

Ripe and unripe both held above 95% precision and 91% recall. Turning dropped to 75.4% and 77.8%, the same weak spot as on the test set. Over six complete sessions the class mix barely moved from day to day, with unripe making up about two thirds of detections and ripe about 30%.

The harvest rule (more than five ripe fruits means pick it) matched the hand count on 25 of 27 plants. Both misses were false alarms on plants sitting right at the threshold, where one wrong ripe call flips the decision.

Turning needs more examples, plus augmentation aimed at that colour range, such as targeted colour jitter and hard-negative mining on the fruit it confuses with ripe and unripe. The broken class should be rebuilt from real greenhouse photos instead of a borrowed dataset. And midday light, which washes out the colours and is why every scan ran in the late afternoon, needs either training data from those hours or a diffuser on the camera.