Kal AI
Building a Smart Calorie Tracker: AI Vision, Verified Nutrition, and a Homelab Pipeline
I got tired of calorie tracking apps that make you search through endless databases, guess portion sizes, or pay monthly subscriptions for basic features. So I built my own — Kal AI, a PWA that lets you snap a photo of your food and get verified nutritional data in seconds. It runs on my homelab, supports Portuguese and English, and feels like a native iPhone app.
This article breaks down how I built it, the AI pipeline behind the analysis, and the lessons learned shipping it to real users.
1. The Problem
Most calorie trackers fall into one of two camps:
- Manual search apps — you type "chicken breast" and pick from 40 results with wildly different values. Nobody has time for that.
- AI-only apps — they estimate calories from a photo, but the data is unreliable. AI hallucinations in nutrition data can mean a 300 kcal error on a single meal.
I wanted something in between: AI identifies the food, then verified databases confirm the numbers. The best of both worlds.
2. The Tech Stack
| Layer | Technology |
|---|---|
| Framework | Next.js 16 (App Router) |
| Styling | Tailwind CSS v4 |
| Database & Auth | Supabase (PostgreSQL + Auth) |
| AI Vision | Google Gemini 2.5 Flash |
| Nutrition Data | FatSecret API + OpenFoodFacts |
| Recipe Search | Spoonacular API |
| Charts | Recharts |
| PWA | next-pwa |
| Deployment | Docker + GitHub Actions → GHCR → Homelab |
| Language | TypeScript |
The entire app is a Next.js 16 PWA with the App Router. Styling is Tailwind CSS v4 with an iOS-inspired design system — SF Pro font stack, Apple Health color palette (--ios-blue, --ios-green, --ios-red), frosted-glass navigation bar, and safe-area support for iPhone notches.
Authentication runs through Supabase Auth with email/password and Google OAuth. The database is PostgreSQL on Supabase with Row Level Security on every table — users can only read and write their own data.
3. The AI Pipeline: From Photo to Verified Nutrition
This is the core of the app. When you log a meal, it goes through a 3-step pipeline:
📸 Step 1 — Gemini Vision Identifies the Food
The photo (or ingredient list) is sent to Google Gemini 2.5 Flash. The AI acts as a "Digital Nutritionist" — it identifies individual food items, estimates portion sizes based on plate and cutlery proportions, considers cooking methods (oil sheen = fried = more calories), and returns structured JSON with each item's name, weight, and estimated macros.
User snaps photo of lunch plate
│
▼
┌─────────────────────────┐
│ Gemini 2.5 Flash │
│ "I see: │
│ - 200g white rice │
│ - 150g chicken breast │
│ - 80g broccoli" │
└─────────────────────────┘
🔍 Step 2 — Database Verification
Gemini's estimates are good, but not trustworthy enough on their own. Each identified item is cross-referenced against real nutrition databases:
- FatSecret API — the primary source, with per-100g verified data
- OpenFoodFacts — free fallback if FatSecret has no match
- Gemini Estimate — last resort, only used when both databases miss
Each item in the final result shows a source badge so the user knows exactly where the data came from. If multiple sources were consulted, all values are shown for transparency.
✅ Step 3 — Confidence Score
The system calculates a confidence score based on how many items were verified by databases vs. AI-only estimates. A meal where 3/3 items matched FatSecret gets 90%+ confidence. A meal where everything fell back to Gemini gets ~65%.
The user sees this score alongside the results — full transparency about data quality.
4. Features Deep Dive
🥩 Ingredient Builder
Not every meal needs a photo. The app has a categorized ingredient builder with 9 food categories: Grains & Carbs, Meat & Poultry, Fish & Seafood, Vegetables, Legumes, Dairy & Eggs, Fruits, Fats & Oils, and Sauces & Extras.
Each food has English and Portuguese names. Some items use unit-based input instead of grams — eggs, for example, let you type "2" instead of "100g", and the app calculates the weight automatically (2 × 50g = 100g).
There's also an "Other / Custom" category where users can type any food name freely — this was added based on user feedback requesting foods not yet in the database.
The food database includes Portuguese and Brazilian foods like panados (breaded cutlets), feijoada, farofa, and beijinho — all added based on real user requests through the in-app feedback system.
🍪 Common Snacks
Packaged snacks are a special case. A protein bar or a pack of cookies has exact nutrition data printed on the label — no AI estimation needed.
The flow:
- Take a photo of the nutrition label on the packaging
- Gemini reads the label and extracts per-serving macros (not per 100g, not per pack — the actual single serving)
- Save it once — from then on, just tap the snack, pick how many you ate, done
You can optionally add context like "pack of 4 cookies" to help the AI identify what one serving actually is.
🍳 Pantry Recipes
"I have chicken, rice, and broccoli — what can I make?" The pantry feature answers this using the Spoonacular API.
Users type ingredients they have at home. The app supports Portuguese input with auto-translation — type "frango" and it gets translated to "chicken" before calling the API. This is powered by a built-in bilingual dictionary with 40+ ingredient mappings.
Recipe results show:
- Nutrition per serving (calories, protein, carbs, fats)
- Which ingredients you already have vs. what you still need
- Step-by-step cooking instructions
- A link to the full recipe
📊 Dashboard & Charts
The daily view shows a circular calorie ring (SVG), macro progress bars, and a list of today's meals with meal type icons and timestamps.
The weekly view is an area chart showing calorie trends over 7 days. Tap any day to see that day's meals in detail.
The monthly view is a bar chart aggregating calories by week.
All charts are built with Recharts and are fully responsive.
🌍 Bilingual Support
The entire app is available in English and Portuguese. This isn't just UI labels — the AI responses, food names, ingredient categories, error messages, and even the feedback system all respect the selected language.
The translation system is a custom lightweight i18n implementation (no heavy library needed) with a single i18n.ts file containing all translation keys.
5. The Deployment Pipeline
The app is self-hosted on my homelab with a fully automated CI/CD pipeline:
git push to master
│
▼
┌──────────────────────┐
│ GitHub Actions │
│ - Build Docker image │
│ - Inject secrets as │
│ build-args │
│ - Push to GHCR │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ GitHub Container │
│ Registry (ghcr.io) │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Homelab Server │
│ - Watchtower auto- │
│ pulls new images │
│ - Docker Compose │
│ restarts container │
└──────────────────────┘
The Dockerfile uses a 3-stage multi-stage build:
- deps — installs
node_modules - builder — builds the Next.js app with all env vars injected as build-args
- runner — minimal Alpine image with only the standalone output
A key lesson learned: environment variables set in the builder stage don't carry over to the runner stage in multi-stage Docker builds. Server-side API keys (Gemini, FatSecret, Spoonacular) had to be re-declared as ARG + ENV in the runner stage, or the API routes would silently fail at runtime with no keys available.
The GitHub Actions workflow injects all secrets as --build-args during the Docker build. On the homelab, Watchtower monitors GHCR and automatically pulls new images when they're pushed — zero manual deployment.
6. The Admin Debug Mode
When something breaks in production, you need visibility. The app has a hidden debug mode gated by the admin email address and a localStorage flag.
When enabled, the analysis flow logs every step in real-time:
- Whether the API key is present
- What Gemini identified and estimated
- Which nutrition sources were consulted for each item
- Where lookups failed and what the fallback was
- Final calorie totals and data source labels
This was instrumental in diagnosing a production bug where the Spoonacular API key wasn't reaching the Docker runtime — the debug panel immediately showed API key present: false, pointing directly to the multi-stage Dockerfile env var issue.
7. User Feedback Loop
The app includes a built-in feedback system — users can submit bug reports or feature requests with screenshot attachments directly from the app. Reports are stored in Supabase with status tracking (open, in progress, closed).
This has driven real improvements:
- A user reported that expired email confirmation links showed a raw error URL instead of a friendly message → fixed with bilingual error handling
- Users requested specific foods (panados, feijoada, farofa) → added to the ingredient database
- A user wanted to type custom food names when selecting "Other" → added a free-text input field
Building the feedback loop into the app itself (instead of relying on email or GitHub issues) dramatically lowered the barrier for non-technical users to report problems.
8. Design Decisions
Why PWA instead of native?
The target audience uses iPhones. Building a native iOS app would require an Apple Developer account ($99/year), App Store review, and maintaining Swift/SwiftUI code. A PWA installed via Safari's "Add to Home Screen" gives 95% of the native experience — standalone mode, no browser UI, app icon on the home screen — with zero App Store friction.
Why not just use MyFitnessPal?
Three reasons: the AI photo analysis (snap and done), the multi-source verification (don't trust a single source), and the Portuguese language support (most nutrition apps are English-only, which is friction for Portuguese-speaking users).
Why self-host?
The app handles food photos and personal health data. Self-hosting on the homelab means the data stays on infrastructure I control. Plus, running on Docker with Watchtower auto-updates makes deployment trivially simple — push to GitHub and it's live within minutes.
9. FAQ
Q: How accurate is the AI photo analysis?
A: The AI identification is surprisingly good for common meals — it correctly identifies rice, chicken, vegetables, eggs, etc. from photos. The accuracy of the nutrition data depends on the verification step: items matched against FatSecret or OpenFoodFacts are highly accurate (within 5-10%), while AI-only estimates can be off by 15-20%. The source badges make this transparent.
Q: Does it work offline?
A: The PWA shell caches for offline access, but meal analysis requires an internet connection (Gemini API + nutrition lookups). Previously logged meals are viewable offline.
Q: Can I add my own foods?
A: Yes. The "Other / Custom" category lets you type any food name, and the AI will estimate its nutrition. For packaged snacks, you can photograph the nutrition label once and quick-log it forever.
Q: What about privacy?
A: All data is stored in Supabase with Row Level Security — users can only access their own data. Meal photos are stored in Supabase Storage. The app is self-hosted, so no third-party analytics or tracking.
Conclusion
Kal AI started as a personal project to solve a real annoyance — calorie tracking that's both fast and accurate. The key insight was combining AI vision (fast) with database verification (accurate) into a single pipeline.
The most rewarding part has been watching real users submit feedback and shape the app. Features like Portuguese ingredient translation, the custom "Other" input, and the snack label reader all came from actual user requests.
The stack — Next.js + Supabase + Gemini + Docker on a homelab — turned out to be a sweet spot for a small-scale production app: fast to develop, cheap to run, and fully under my control.
Source code: github.com/MonoBL/Kal-AI
Vibecoded and designed by Nuno