Project Diary

A running log of what happened on RoboPrompt, day by day.

2026-07-02 - Project kickoff

• Set up the initial Next.js project.

2026-07-08 - Found the real product, rebuilt it

• Defined RoboPrompt as an AI assistant that turns a photo of a robotic arm into a working control plan.
• Rebuilt the whole app from scratch - a proper agent system prompt, streaming classify/chat API routes, image upload, an interview UI, and a rendered plan view.
• Turned the single chat page into a multi-page product site and deployed it to Vercel.
• Fixed a production photo-upload crash (a native-binary packaging issue with pnpm and sharp).
• Reworked the interview from free-text chat into real structured forms, and fixed loading-state gaps so nothing looks frozen while waiting.

2026-07-09 - Overnight hardening and code generation, then merged with multilingual support

• Fixed a dark-mode UI bug, added a password gate protecting the whole tool so token usage can't be abused, and deepened the agent's questions so it gathers everything needed for a real build, not just a rough idea.
• Added the ability to attach reference files (datasheets, existing code, CAD/URDF) alongside the photo.
• Added a "Download code (.zip)" option that generates a real, working project (firmware, control panel, README) from the confirmed plan.
• Ran a full regression pass and a live production self-test end-to-end with a real robot arm photo - confirmed everything works.
• Fixed security issues involving zip bombs, unsafe file paths, and insecure comparisons.
• Added photo thumbnails so users can see what they uploaded.
• Added multilingual interface and AI-response support for English, Spanish, French, and Chinese.
• Rebranded the site to a fixed black-and-pink theme with a new robot logo.
• Made the tool feel like a real product: CI now checks every push, an interview survives a page refresh, and a finished plan can be revised with feedback (e.g. "switch to ESP32") instead of starting over.

2026-07-10 - Faster, cheaper, and two production bugs squashed

• Fixed the intermittent 504 when downloading code: generation regularly runs past two minutes, but the serverless function was capped at exactly that — raised the limit and added a friendly timeout message.
• Switched the interview to a faster model (Haiku): photo analysis and questions now respond in 2-5 seconds instead of 8-11, at a tenth of the cost, while code generation stays on Sonnet for quality. Live-testing the switch surfaced three protocol weaknesses (prose drift, a fenced plan marker breaking detection, endless follow-up questions after confirmation) — all fixed for every model.
• Fixed a crash on "Download code" ("e.forEach is not a function"): the model occasionally double-encodes its file list as a JSON string — output is now normalized and recovered, covered by unit tests and a real-browser click test.
• Added a clear start-over path, refined the interview button hierarchy, and rebuilt the waiting indicator around the robot logo.
• Hardened the product with safe redirects, API rate limits, broader automated tests, stronger CI, and a corrected pnpm setup for reliable Vercel deploys.
• Replaced the finished plan's text wall with themed section cards.

2026-07-11 - More natural multilingual conversations

• Improved adaptive language matching so replies follow the user's actual message while still respecting the selected interface language.

2026-07-13 - Friendlier onboarding for beginners

• Reworked the upload, interview, and plan flow into three clearly labeled steps, with photo tips, plain-language guidance, a privacy note, and complete copy in all four supported languages.

2026-07-15 - Removed the Members page, faster AI responses

• Removed the Members section from the site: the page, individual profile pages, member photos, the nav link, and all related translations in the four languages.
• Enabled prompt caching on all three AI routes (photo analysis, interview chat, code generation): the large system prompt and the conversation history are now cached between turns and shared across users — the cached portion costs ~90% less and responses start faster. Verified live: ~5,500 tokens served from cache per turn.
• Made code generation ~6x faster for the most common setup: the standard Arduino-USB-serial scaffold (firmware sketch + Web Serial control panel) now ships as a pre-built, tested template inside the app, and the AI only generates the small per-project files (pin/limit config, README). Verified live: 22.6 seconds and ~1,900 output tokens, down from 2-3 minutes and ~25,000. Non-standard setups (ESP32, ROS2, vendor SDKs) still get fully generated code.

2026-07-16 - Notes and files at upload time

• The upload step now has a free-text box for extra info or instructions ("I already have firmware", "only ask about the gripper") — it goes straight to the AI with the photo and also sets the response language.
• The reference-file picker now advertises everything the backend already reads (JSON configs, code files like .ino/.py/.cpp, URDF, PDFs, zips), and the upload widget is translated in all four languages instead of English-only.

2026-07-16 - Kinova Pick-and-Place studio

• New "Pick & Place" page: upload a Kortex sequence JSON that ran successfully on a real Kinova arm, and the app extracts a workspace profile from the demonstration — table height, safe travel height, tool orientation, gripper open/close, speed — each with its source action, reasoning, and confidence, and each requiring explicit confirmation before use.
• From confirmed parameters plus new pick/place coordinates it plans the standard 9-step pick-and-place, shows a color-coded motion timeline and 2D path preview (top + side view against the demonstrated workspace), runs safety validation (below-table Z, clearance, out-of-demonstrated-range, gripper values), and generates a new Kortex JSON that clones the uploaded file's exact structure and metadata.
• Everything runs in the browser; the tool never controls the arm, and every export carries "verify in Kortex before running". Verified end-to-end against a real Kortex export.

2026-07-17 - Robot Agent: vision + language control

• New "Robot Agent" page: point a camera (or upload a test image in clearly-labeled Demo Mode) at the workspace, type "Put the apple in the blue box", and the app detects objects with real AI vision (numbered when duplicated, never auto-guessing), parses the command into a structured task, asks follow-up questions when information is missing, and plans the 9-step pick-and-place on top of a confirmed Kortex skill template.
• Coordinates stay honest: without a depth camera or calibration, pixels never become robot coordinates - pick/place positions are entered manually and confirmed. A local Python Robot Bridge (reference simulation included in the repo) executes plans only after Preview Plan and an explicit Confirm & Run, with pause/stop, live progress, and a hard rule that Demo Mode and any red safety item disable execution.
• Adapter seams for the future: RobotAdapter (Kortex Python API / ROS 2 later) and PolicyAdapter (SmolVLA/OpenVLA later); today's control mode is honestly labeled "Verified Skill Template". 14 new tests (49 total).