SaaS / RAGDebugger

RAGDebugger

bootstrapped
Déboguez vos pipelines RAG en quelques minutes, pas en quelques jours
🌐 show-hn-ragdebugger-observabil.eymdey-network.com
👥 Voir les clients
MRR
0€/m
Clients payants
0
Trials
0
Churn
0%
Fiche produit
ICP
Équipes ML/IA qui déploient des applications RAG en production
Problème résolu
Les développeurs et data scientists perdent des heures à diagnostiquer pourquoi leur pipeline RAG retourne des réponses incorrectes, hallucine ou récu
Pricing
RAGDebugger
49€
Pro
149€
Business
399€
Trial : 14 jours gratuits
Mettre à jour les métriques
Docs générés automatiquement
SPEC.mdDESIGN.mdMARKETING.md

RAGDebugger — Spécification Technique MVP

1. Vue d'ensemble

Concept : RAGDebugger est une plateforme d'observabilité SaaS pour pipelines RAG (Retrieval-Augmented Generation). Elle instrumente automatiquement LangChain, LlamaIndex et pipelines custom via un SDK léger, trace chaque étape (embedding → retrieval → reranking → génération) et expose des métriques de qualité avec alertes. Valeur ajoutée principale : Remplacer les print() et logs épars par une interface de débogage centralisée qui révèle pourquoi un pipeline RAG hallucine ou retourne des résultats non pertinents — en quelques minutes, pas en jours. Ce qu'on build en MVP (48h max) :
  • SDK Python avec décorateur @ragdebugger.trace() qui envoie des traces via HTTP
  • Ingestion des traces via API REST sécurisée par clé API
  • Dashboard avec liste/détail de traces, métriques de base (latence, scores similarité)
  • Alertes email sur seuils configurables (hallucination score, latence)
  • Auth + billing Stripe (Free / Pro)

2. Stack technique

Core

| Couche | Technologie |

|--------|-------------|

| Framework | Next.js 15 App Router + TypeScript strict |

| Base de données | Supabase (PostgreSQL + Auth + Realtime) |

| Billing | Stripe (Checkout + Webhooks + Customer Portal) |

| Emails | Resend + React Email |

| Déploiement | Vercel |

Dépendances NPM spécifiques

{

  "dependencies": {


    "@supabase/supabase-js": "^2.45.0",


    "@supabase/ssr": "^0.5.0",


    "stripe": "^16.0.0",


    "@stripe/stripe-js": "^4.0.0",


    "resend": "^4.0.0",


    "@react-email/components": "^0.0.22",


    "recharts": "^2.12.0",


    "date-fns": "^3.6.0",


    "zod": "^3.23.0",


    "zustand": "^4.5.0",


    "@tanstack/react-query": "^5.51.0",


    "@tanstack/react-table": "^8.19.0",


    "react-json-view-lite": "^1.4.0",


    "nanoid": "^5.0.0",


    "clsx": "^2.1.0",


    "tailwind-merge": "^2.4.0",


    "lucide-react": "^0.400.0",


    "@radix-ui/react-dialog": "^1.1.0",


    "@radix-ui/react-tabs": "^1.1.0",


    "@radix-ui/react-select": "^2.1.0",


    "@radix-ui/react-tooltip": "^1.1.0",


    "@radix-ui/react-badge": "^1.0.0",


    "@radix-ui/react-switch": "^1.1.0",


    "sonner": "^1.5.0",


    "next-themes": "^0.3.0"


  },


  "devDependencies": {


    "@types/node": "^20.0.0",


    "tsx": "^4.16.0"


  }


}

> Note SDK Python (hors Next.js) : Le SDK Python (ragdebugger-sdk) est un package PyPI minimal avec requests ou httpx. Il est livré séparément dans /sdk/python/. La spec en détaille l'interface mais pas l'implémentation interne.

3. Modèle de données Supabase

Tables existantes (présupposées)

  • profiles — reliée à auth.users, contient id, email, full_name, avatar_url, created_at
  • subscription_events — log des événements Stripe

Nouvelles tables

-- ============================================================

-- ORGANISATIONS (multi-tenant)


-- ============================================================


CREATE TABLE organizations (


  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  name         TEXT NOT NULL,


  slug         TEXT NOT NULL UNIQUE,


  owner_id     UUID NOT NULL REFERENCES profiles(id) ON DELETE CASCADE,


  plan         TEXT NOT NULL DEFAULT 'free' CHECK (plan IN ('free', 'pro', 'enterprise')),


  stripe_customer_id TEXT UNIQUE,


  stripe_subscription_id TEXT UNIQUE,


  trial_ends_at TIMESTAMPTZ DEFAULT (NOW() + INTERVAL '14 days'),


  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  updated_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_organizations_owner_id ON organizations(owner_id);


CREATE INDEX idx_organizations_slug ON organizations(slug);


-- ============================================================


-- MEMBRES D'ORGANISATION


-- ============================================================


CREATE TABLE organization_members (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  user_id         UUID NOT NULL REFERENCES profiles(id) ON DELETE CASCADE,


  role            TEXT NOT NULL DEFAULT 'member' CHECK (role IN ('owner', 'admin', 'member')),


  invited_email   TEXT,


  accepted_at     TIMESTAMPTZ,


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  UNIQUE(organization_id, user_id)


);


CREATE INDEX idx_org_members_user_id ON organization_members(user_id);


CREATE INDEX idx_org_members_org_id ON organization_members(organization_id);


-- ============================================================


-- PROJETS (un org peut avoir plusieurs pipelines RAG)


-- ============================================================


CREATE TABLE projects (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  name            TEXT NOT NULL,


  description     TEXT,


  framework       TEXT CHECK (framework IN ('langchain', 'llamaindex', 'custom', 'other')),


  created_by      UUID NOT NULL REFERENCES profiles(id),


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_projects_org_id ON projects(organization_id);


-- ============================================================


-- CLÉS API


-- ============================================================


CREATE TABLE api_keys (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  project_id      UUID REFERENCES projects(id) ON DELETE SET NULL,


  name            TEXT NOT NULL,


  key_hash        TEXT NOT NULL UNIQUE,   -- SHA-256 de la clé réelle


  key_prefix      TEXT NOT NULL,          -- ex: "rdbg_live_xxxx" (8 premiers chars)


  last_used_at    TIMESTAMPTZ,


  expires_at      TIMESTAMPTZ,


  revoked_at      TIMESTAMPTZ,


  created_by      UUID NOT NULL REFERENCES profiles(id),


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_api_keys_org_id ON api_keys(organization_id);


CREATE INDEX idx_api_keys_key_hash ON api_keys(key_hash);


-- ============================================================


-- TRACES (enregistrement principal d'une exécution RAG)


-- ============================================================


CREATE TABLE traces (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  project_id      UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  -- Identification


  trace_id        TEXT NOT NULL,          -- ID côté SDK (pour corrélation)


  session_id      TEXT,                   -- grouper plusieurs turns d'une conversation


  user_id_external TEXT,                  -- ID utilisateur final (côté client)


  tags            TEXT[] DEFAULT '{}',


  -- Requête


  query           TEXT NOT NULL,          -- question posée par l'utilisateur


  query_embedding_model TEXT,             -- ex: "text-embedding-3-small"


  -- Résultat


  final_answer    TEXT,                   -- réponse générée par le LLM


  llm_model       TEXT,                   -- ex: "gpt-4o-mini"


  -- Métriques globales


  total_latency_ms INTEGER,


  retrieval_latency_ms INTEGER,


  llm_latency_ms  INTEGER,


  total_tokens    INTEGER,


  prompt_tokens   INTEGER,


  completion_tokens INTEGER,


  estimated_cost_usd NUMERIC(10, 6),


  -- Scores calculés


  faithfulness_score  NUMERIC(4,3),       -- 0.000 à 1.000 (réponse fondée sur les chunks ?)


  answer_relevance_score NUMERIC(4,3),    -- réponse pertinente par rapport à la question ?


  context_precision   NUMERIC(4,3),       -- chunks récupérés pertinents ?


  context_recall      NUMERIC(4,3),


  -- État


  status          TEXT NOT NULL DEFAULT 'success' CHECK (status IN ('success', 'error', 'timeout', 'partial')),


  error_message   TEXT,


  has_hallucination_flag BOOLEAN DEFAULT FALSE,


  -- Metadata


  sdk_version     TEXT,


  environment     TEXT DEFAULT 'production' CHECK (environment IN ('production', 'staging', 'development')),


  metadata        JSONB DEFAULT '{}',


  ingested_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()  -- timestamp côté SDK


);


CREATE INDEX idx_traces_project_id ON traces(project_id);


CREATE INDEX idx_traces_org_id ON traces(organization_id);


CREATE INDEX idx_traces_ingested_at ON traces(ingested_at DESC);


CREATE INDEX idx_traces_trace_id ON traces(trace_id);


CREATE INDEX idx_traces_session_id ON traces(session_id) WHERE session_id IS NOT NULL;


CREATE INDEX idx_traces_status ON traces(status);


CREATE INDEX idx_traces_has_hallucination ON traces(has_hallucination_flag) WHERE has_hallucination_flag = TRUE;


CREATE INDEX idx_traces_tags ON traces USING GIN(tags);


-- ============================================================


-- SPANS (étapes individuelles dans une trace)


-- ============================================================


CREATE TABLE spans (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  trace_id_fk     UUID NOT NULL REFERENCES traces(id) ON DELETE CASCADE,


  parent_span_id  UUID REFERENCES spans(id) ON DELETE SET NULL,  -- arbre de spans


  span_type       TEXT NOT NULL CHECK (span_type IN (


    'retrieval', 'embedding', 'reranking', 'llm_call',


    'prompt_template', 'document_loader', 'custom'


  )),


  name            TEXT NOT NULL,          -- ex: "ChromaDB retrieval", "OpenAI embedding"


  start_time      TIMESTAMPTZ NOT NULL,


  end_time        TIMESTAMPTZ,


  latency_ms      INTEGER,


  status          TEXT DEFAULT 'success' CHECK (status IN ('success', 'error', 'skipped')),


  error_message   TEXT,


  -- Données spécifiques au type (flexibilité maximale en JSONB)


  input_data      JSONB,                  -- requête/input de ce span


  output_data     JSONB,                  -- résultat de ce span


  metadata        JSONB DEFAULT '{}',


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_spans_trace_id ON spans(trace_id_fk);


CREATE INDEX idx_spans_type ON spans(span_type);


CREATE INDEX idx_spans_parent ON spans(parent_span_id) WHERE parent_span_id IS NOT NULL;


-- ============================================================


-- CHUNKS RÉCUPÉRÉS (détail du retrieval par trace)


-- ============================================================


CREATE TABLE retrieved_chunks (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  trace_id_fk     UUID NOT NULL REFERENCES traces(id) ON DELETE CASCADE,


  span_id         UUID REFERENCES spans(id) ON DELETE SET NULL,


  rank_position   INTEGER NOT NULL,       -- position dans les résultats (1-based)


  chunk_id_external TEXT,                -- ID du chunk dans le vector store


  content         TEXT NOT NULL,


  source_document TEXT,                  -- nom/URL du document source


  source_page     INTEGER,


  similarity_score NUMERIC(6,5),         -- score cosine (0 à 1)


  rerank_score    NUMERIC(6,5),          -- score après reranking


  is_used_in_prompt BOOLEAN DEFAULT FALSE, -- ce chunk a-t-il été inclus dans le prompt final ?


  metadata        JSONB DEFAULT '{}',


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_chunks_trace_id ON retrieved_chunks(trace_id_fk);


CREATE INDEX idx_chunks_rank ON retrieved_chunks(trace_id_fk, rank_position);


-- ============================================================


-- ÉVALUATIONS (annotations manuelles ou auto)


-- ============================================================


CREATE TABLE evaluations (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  trace_id_fk     UUID NOT NULL REFERENCES traces(id) ON DELETE CASCADE,


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  evaluator_type  TEXT NOT NULL CHECK (evaluator_type IN ('human', 'llm_judge', 'rule_based')),


  evaluator_id    UUID REFERENCES profiles(id),  -- NULL si auto


  metric_name     TEXT NOT NULL,          -- ex: "faithfulness", "relevance", "custom_correctness"


  score           NUMERIC(4,3),           -- 0 à 1


  label           TEXT,                   -- ex: "correct", "hallucinated", "partial"


  reasoning       TEXT,                   -- explication du score


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_evaluations_trace_id ON evaluations(trace_id_fk);


CREATE INDEX idx_evaluations_metric ON evaluations(metric_name);


CREATE INDEX idx_evaluations_org_id ON evaluations(organization_id);


-- ============================================================


-- ALERTES (règles configurées par les utilisateurs)


-- ============================================================


CREATE TABLE alert_rules (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  project_id      UUID REFERENCES projects(id) ON DELETE CASCADE,


  name            TEXT NOT NULL,


  is_active       BOOLEAN DEFAULT TRUE,


  metric          TEXT NOT NULL,          -- ex: "faithfulness_score", "total_latency_ms", "error_rate"


  operator        TEXT NOT NULL CHECK (operator IN ('lt', 'gt', 'lte', 'gte', 'eq')),


  threshold       NUMERIC NOT NULL,


  window_minutes  INTEGER DEFAULT 60,     -- fenêtre d'évaluation


  channel         TEXT NOT NULL CHECK (channel IN ('email', 'slack', 'webhook')),


  destination     TEXT NOT NULL,          -- email, webhook URL, slack channel


  cooldown_minutes INTEGER DEFAULT 30,    -- anti-spam


  last_triggered_at TIMESTAMPTZ,


  created_by      UUID NOT NULL REFERENCES profiles(id),


  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()


);


CREATE INDEX idx_alert_rules_org_id ON alert_rules(organization_id);


CREATE INDEX idx_alert_rules_project_id ON alert_rules(project_id) WHERE project_id IS NOT NULL;


CREATE INDEX idx_alert_rules_active ON alert_rules(is_active) WHERE is_active = TRUE;


-- ============================================================


-- HISTORIQUE DES ALERTES DÉCLENCHÉES


-- ============================================================


CREATE TABLE alert_events (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  alert_rule_id   UUID NOT NULL REFERENCES alert_rules(id) ON DELETE CASCADE,


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  triggered_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  metric_value    NUMERIC NOT NULL,


  threshold_value NUMERIC NOT NULL,


  traces_affected INTEGER DEFAULT 0,


  notification_sent BOOLEAN DEFAULT FALSE,


  notification_error TEXT


);


CREATE INDEX idx_alert_events_rule_id ON alert_events(alert_rule_id);


CREATE INDEX idx_alert_events_triggered_at ON alert_events(triggered_at DESC);


-- ============================================================


-- USAGE QUOTAS (tracking pour les limites de plan)


-- ============================================================


CREATE TABLE usage_counters (


  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),


  organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,


  period_start    DATE NOT NULL,          -- début du mois de facturation


  traces_count    INTEGER DEFAULT 0,


  spans_count     INTEGER DEFAULT 0,


  api_calls_count INTEGER DEFAULT 0,


  updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),


  UNIQUE(organization_id, period_start)


);


CREATE INDEX idx_usage_org_period ON usage_counters(organization_id, period_start DESC);

RLS Policies

-- ============================================================

-- Activer RLS sur toutes les tables


-- ============================================================


ALTER TABLE organizations ENABLE ROW LEVEL SECURITY;


ALTER TABLE organization_members ENABLE ROW LEVEL SECURITY;


ALTER TABLE projects ENABLE ROW LEVEL SECURITY;


ALTER TABLE api_keys ENABLE ROW LEVEL SECURITY;


ALTER TABLE traces ENABLE ROW LEVEL SECURITY;


ALTER TABLE spans ENABLE ROW LEVEL SECURITY;


ALTER TABLE retrieved_chunks ENABLE ROW LEVEL SECURITY;


ALTER TABLE evaluations ENABLE ROW LEVEL SECURITY;


ALTER TABLE alert_rules ENABLE ROW LEVEL SECURITY;


ALTER TABLE alert_events ENABLE ROW LEVEL SECURITY;


ALTER TABLE usage_counters ENABLE ROW LEVEL SECURITY;


-- ============================================================


-- Helper function : obtenir les org IDs accessibles pour un user


-- ============================================================


CREATE OR REPLACE FUNCTION get_user_org_ids(user_uuid UUID)


RETURNS UUID[] AS $$


  SELECT ARRAY(


    SELECT organization_id


    FROM organization_members


    WHERE user_id = user_uuid AND accepted_at IS NOT NULL


  );


$$ LANGUAGE sql STABLE SECURITY DEFINER;


-- ============================================================


-- ORGANIZATIONS


-- ============================================================


CREATE POLICY "org_select" ON organizations


  FOR SELECT USING (id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "org_insert" ON organizations


  FOR INSERT WITH CHECK (owner_id = auth.uid());


CREATE POLICY "org_update" ON organizations


  FOR UPDATE USING (owner_id = auth.uid());


-- ============================================================


-- ORGANIZATION_MEMBERS


-- ============================================================


CREATE POLICY "members_select" ON organization_members


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "members_insert_owner" ON organization_members


  FOR INSERT WITH CHECK (


    EXISTS (


      SELECT 1 FROM organizations


      WHERE id = organization_id AND owner_id = auth.uid()


    )


  );


CREATE POLICY "members_delete" ON organization_members


  FOR DELETE USING (


    user_id = auth.uid() OR


    EXISTS (


      SELECT 1 FROM organizations


      WHERE id = organization_id AND owner_id = auth.uid()


    )


  );


-- ============================================================


-- PROJECTS


-- ============================================================


CREATE POLICY "projects_select" ON projects


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "projects_insert" ON projects


  FOR INSERT WITH CHECK (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "projects_update" ON projects


  FOR UPDATE USING (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "projects_delete" ON projects


  FOR DELETE USING (


    EXISTS (


      SELECT 1 FROM organizations


      WHERE id = organization_id AND owner_id = auth.uid()


    )


  );


-- ============================================================


-- API_KEYS (lecture seulement via dashboard, écriture via service role)


-- ============================================================


CREATE POLICY "apikeys_select" ON api_keys


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "apikeys_insert" ON api_keys


  FOR INSERT WITH CHECK (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "apikeys_update_revoke" ON api_keys


  FOR UPDATE USING (organization_id = ANY(get_user_org_ids(auth.uid())));


-- ============================================================


-- TRACES (ingestion via service role, lecture via dashboard)


-- ============================================================


CREATE POLICY "traces_select" ON traces


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


-- INSERT/UPDATE via service_role uniquement (API d'ingestion)


-- ============================================================


-- SPANS


-- ============================================================


CREATE POLICY "spans_select" ON spans


  FOR SELECT USING (


    EXISTS (


      SELECT 1 FROM traces t


      WHERE t.id = trace_id_fk


        AND t.organization_id = ANY(get_user_org_ids(auth.uid()))


    )


  );


-- ============================================================


-- RETRIEVED_CHUNKS


-- ============================================================


CREATE POLICY "chunks_select" ON retrieved_chunks


  FOR SELECT USING (


    EXISTS (


      SELECT 1 FROM traces t


      WHERE t.id = trace_id_fk


        AND t.organization_id = ANY(get_user_org_ids(auth.uid()))


    )


  );


-- ============================================================


-- EVALUATIONS


-- ============================================================


CREATE POLICY "evals_select" ON evaluations


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


CREATE POLICY "evals_insert" ON evaluations


  FOR INSERT WITH CHECK (organization_id = ANY(get_user_org_ids(auth.uid())));


-- ============================================================


-- ALERT_RULES


-- ============================================================


CREATE POLICY "alerts_all" ON alert_rules


  FOR ALL USING (organization_id = ANY(get_user_org_ids(auth.uid())));


-- ============================================================


-- ALERT_EVENTS


-- ============================================================


CREATE POLICY "alert_events_select" ON alert_events


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));


-- ============================================================


-- USAGE_COUNTERS


-- ============================================================


CREATE POLICY "usage_select" ON usage_counters


  FOR SELECT USING (organization_id = ANY(get_user_org_ids(auth.uid())));

4. Routes API

Ingestion (authentification par clé API dans le header X-API-Key)

| Method | Route | Description | Auth |

|--------|-------|-------------|------|

| POST | /api/ingest/trace | Ingérer une trace complète (query + spans + chunks + métriques) | API Key |

| POST | /api/ingest/trace/batch | Ingérer un batch de traces (max 50) | API Key |

| POST | /api/ingest/span | Ajouter un span à une trace existante (streaming progressif) | API Key |

|