Nº 04of 12Featured

ClinVision

AI that grades dental practice from a photo: a YOLO pose model trained only on Blender synthetic data, millimeter measurements from an ArUco marker, and a 2D→3D correction before the grade

Year
2026
Status
IN PROGRESS
Role
ML pipeline, synthetic data and API · pair project
Stack
Python · FastAPI · OpenCV · Ultralytics YOLO11 +5
clinvision
ClinVision — Student result — the bonding grade, each tooth marked on the synthetic photo, and what to fix first
4 screens

1 of 4

ClinVision — Student result — the bonding grade, each tooth marked on the synthetic photo, and what to fix first

Student result — the bonding grade, each tooth marked on the synthetic photo, and what to fix first

01The project

Automatic grading for a dental practice exercise: bonding brackets on a training model (a typodont). The student photographs the model with a 10 mm ArUco marker in frame; the marker gives the pixel-to-millimeter scale, a YOLO pose model finds each tooth and five keypoints on it, a second-stage refiner repositions the keypoints on a 320 px crop, and geometry turns them into height, angulation and centering in mm and degrees — corrected from 2D to 3D before the rubric assigns the grade, with up to three views per submission. There is no annotated real photo, so the detector is trained only on synthetic photos rendered in Blender from 3D scans of real dental arches (sim-to-real). It is a pair project with Nicolas Volf: 95 of the 140 commits are mine — the ML pipeline, the synthetic generator, the experiments and most of the API; sign-in, classes and most of the front end are his. Current state: the detector has not yet been tested on real photos graded by a professor and the rubric's thresholds are provisional, so nothing here is validated in real use.

02What was built

  1. A 7-stage grading pipeline, ~0.9 s per photo on CPU: ArUco marker (px→mm scale), YOLO11 pose, a refiner on 320 px crops, measurements, a calibrated 2D→3D correction, the rubric, and fusion of up to 3 views

  2. 9 dated experiments, each with a question, data, result and decision; the first one showed that the frontal photo alone is off by ~0.5 mm even with perfect keypoints — 1 in 4 pass/fail calls wrong —, and that is what the 2D→3D correction goes after

  3. Synthetic data from Blender with the right answer by construction: brackets, resin, wire, lips, retractors and phone-camera defects randomized (domain randomization), and unlabeled real photos used only as a thermometer

  4. A human review queue: an unsure or incomplete grading goes to the professor first, and the model's grade stays hidden from the student until the review

  5. FastAPI API on Supabase (Postgres, Auth and Storage) with 7 migrations, photos stored with EXIF stripped, and 330 pytest tests; the JWT sign-in verified against the public keys (JWKS) and the student, professor and monitor roles are Nicolas's

03Decisions

Decision 01

Question: Why train the detector on synthetic photos only?

Answer: Because there is no annotated real photo, and none in sight. The Blender render comes out with the right answer by construction — each bracket's exact keypoints and the true measurement —, which separates the detector's error from the geometry's. That is how it became clear the bottleneck was not YOLO but measuring in 2D on a frontal photo. The price is the sim-to-real gap: an unlabeled real photo only works as a thermometer, which is why the results do not count as validation yet.

Decision 02

Question: Why inverse calibration and not direct regression to correct the measurement?

Answer: Direct regression won on average kappa, but it shrank every measurement toward the mean: with the detector, it failed 0% of teeth on angulation against 46% in the ground truth — it would hide a crooked bracket. Inverse calibration (2D measurement = a + b·truth, inverted) uses only the measurement itself, is fitted on the perfect keypoints of 80% of the scans, and does not depend on the detector.

Decision 03

Question: Why is the default model the one that finds fewer teeth?

Answer: On real photos, v5m with the refiner found slightly fewer teeth than v5s, but it did not invent a single one. A missed tooth goes to the professor's review; an invented tooth can turn into a wrong grade nobody notices.

04Screens

  • captura.webp

    Three-view submission — center, right and left, each facing its own side of the ArUco marker card

  • revisao.webp

    Professor review — the student's request, the AI's assessment, the tooth zoom and the measurements in mm and degrees

  • fila-revisao.webp

    The activity's review queue — student requests and unsure gradings go to the professor first

1 of 4

ClinVision — Student result — the bonding grade, each tooth marked on the synthetic photo, and what to fix first

Student result — the bonding grade, each tooth marked on the synthetic photo, and what to fix first

05Stack

  • Python
  • FastAPI
  • OpenCV
  • Ultralytics YOLO11
  • Blender
  • Supabase
  • PostgreSQL
  • Next.js 16
  • TypeScript
Next project →Nº 05

Home prices served by a FastAPI API on AWS: 9.32% MAPE against actual 2010 sales, every request logged, continuous deploy with a healthcheck