AI agent skill · by @andrewyng

New Project

Build a new AI agent with Olakai monitoring from scratch — project setup, SDK integration, KPI configuration, and end-to-end validation

Archived 2026-10-23 · Source updated 2026-05-31 · License: MIT. Check the original repository for the latest version.

SKILL.md instructions

# Build a New AI Agent Project with Olakai

This skill guides you through creating a new AI agent that is fully integrated with Olakai for analytics, KPI tracking, and governance.

## Prerequisites

Before starting, ensure:
1. Olakai CLI installed: `npm install -g olakai-cli`
2. CLI authenticated: `olakai login`
3. API key for SDK (generated per-agent via CLI — see Step 2.2)

## Why Custom KPIs Are Essential

Olakai's core value is **tracking business-specific KPIs for your AI agents**. Without KPIs, you're tracking events without gaining actionable insights.

**What you can measure with KPIs:**
- Business outcomes (items processed, success rates, revenue impact)
- Operational data (step counts, retry rates, execution time)
- Quality indicators (error rates, user satisfaction signals)

**Without KPIs configured:**
- No dashboard KPIs beyond basic token counts
- No aggregated performance views
- No alerting thresholds
- No ROI calculations

> **Every agent should have 2-4 KPIs that answer: "How do I know this agent is performing well?"**

> **KPIs created here belong to this specific agent only.** If you later create additional agents, each one needs its own KPI definitions — KPIs cannot be shared or reused across agents.

## Understanding the customData to KPI Pipeline

Before diving into implementation, understand how data flows through Olakai:

```
SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula → kpiData
```

### How It Works

1. **customData** (SDK): Raw JSON you send with each event
2. **CustomDataConfig** (Platform): Schema defining which fields are processed
3. **Context Variables**: CustomDataConfig fields become available for formulas
4. **KPI Formula**: Expression that computes a value (e.g., `SuccessRate * 100`)
5. **kpiData** (Response): Computed KPI values returned with each event

### Critical Rules

| Rule | Consequence |
|------|-------------|
| Only CustomDataConfig fields become variables | Unregistered customData fields are NOT usable in KPIs |
| Formula evaluation is case-insensitive | `stepCount`, `STEPCOUNT`, `StepCount` all work in formulas |
| NUMBER configs need numeric values | Don't send `"5"` (string), send `5` (number) |
| KPIs are unique per agent | Each KPI belongs to exactly one agent — create separately for each |

### Built-in Context Variables (Always Available)

| Variable | Type | Description |
|----------|------|-------------|
| `Prompt` | string | The prompt text sent to the LLM |
| `Response` | string | The LLM response text |
| `Documents count` | number | Number of attached documents |
| `PII detected` | boolean | Whether PII was detected |
| `PHI detected` | boolean | Whether PHI was detected |
| `CODE detected` | boolean | Whether code was detected |
| `SECRET detected` | boolean | Whether secrets were detected |

## Step 1: Design the Agent Architecture

### 1.1 Determine Agent Type

**Agentic AI** (Multi-step autonomous workflows):
- Research agents, document processors, data pipelines
- Track as SINGLE events aggregating all internal LLM calls
- Focus on workflow-level KPIs (total tokens, total time, success/failure)

**Assistive AI** (Interactive chatbots/copilots):
- Customer support agents, coding assistants, Q&A systems
- Track EACH interaction as separate events
- Focus on conversation-level KPIs (per-message tokens, response quality)

### 1.2 Design Your KPI Schema (CRITICAL)

**Design your KPIs BEFORE writing any SDK code.** This ensures only meaningful data is sent and tracked.

#### Step A: Identify Business Questions

What do stakeholders need to know about this agent?
- "How many items does it process per run?"
- "What's the success/failure rate?"
- "How efficient is each execution?"

#### Step B: Map Questions to Data Fields

| Business Question | Field Name | Type | KPI Formula | Aggregation |
|-------------------|------------|------|-------------|-------------|
| Throughput | ItemsProcessed | NUMBER | `ItemsProcessed` | SUM |
| Reliability | SuccessRate | NUMBER | `SuccessRate * 100` | AVERAGE |
| Error count | SuccessRate | NUMBER | `IF(SuccessRate < 1, 1, 0)` | SUM |
| Correlation | ExecutionId | STRING | (for filtering only) | - |

#### Step C: Plan Your customData Structure

```typescript
// ONLY include fields you'll register as CustomDataConfigs
customData: {
  // Business KPIs
  ItemsProcessed: number,  // Count of items handled
  SuccessRate: number,     // 0-1 success ratio

  // Performance KPIs
  StepCount: number,       // Number of workflow steps

  // Identification (for filtering, not KPIs)
  ExecutionId: string,     // Correlation ID
}
```

> **IMPORTANT**: Only include fields you will register as CustomDataConfigs. Unregistered fields are stored but **cannot be used in KPIs**.

### What NOT to Include in customData

The Olakai platform automatically tracks these — do NOT duplicate them:

| Already Tracked | Where | Don't Send As customData |
|-----------------|-------|--------------------------|
| Session ID | Main payload | `sessionId` |
| Agent ID | API key association | `agentId` |
| User email | `userEmail` parameter | `email`, `userEmail` |
| Timestamp | Event metadata | `timestamp`, `createdAt` |
| Request time | `requestTime` parameter | `duration`, `latency` |
| Token count | `tokens` parameter | `tokenCount` |
| Model | Auto-detected | `model`, `modelName` |
| Provider | Client config | `provider` |

**customData is ONLY for:**
1. **KPI variables** — Fields you'll use in formula calculations
2. **Tagging/filtering** — Fields you'll filter by in queries

## Step 2: Configure Olakai Platform

### 2.1 Create a Workflow (Required)

> **Every agent MUST belong to a workflow**, even if it's the only agent in that workflow.

```bash
olakai workflows create --name "Your Workflow Name" --json
# Output: { "id": "wfl_xxx...", "name": "Your Workflow Name" }
```

### 2.2 Create the Agent in Olakai

```bash
olakai agents create \
  --name "Your Agent Name" \
  --description "What this agent does" \
  --workflow WORKFLOW_ID \
  --with-api-key \
  --json

# Returns agent details including apiKey:
# {
#   "id": "cmkbteqn501kyjy4yu6p6xrrx",
#   "name": "Your Agent Name",
#   "workflowId": "wfl_xxx...",
#   "apiKey": "sk_agent_xxxxx..."   <-- Use this in your SDK
# }
```

**Agent-Workflow Hierarchy:**
```
Workflow: "Customer Support Pipeline"
├── Agent: "Ticket Classifier"
├── Agent: "Response Generator"
└── Agent: "Quality Checker"

Workflow: "Document Processing"
└── Agent: "Document Summarizer"  ← single-agent workflows are valid
```

### 2.3 Create Custom Data Configurations (BEFORE Writing SDK Code)

> **This step MUST be completed before Step 3 (SDK Integration).** Only fields registered here can be used in KPI formulas.

> **ONLY create configs for data you'll use in KPIs or for filtering.** Don't create configs for data already tracked automatically.

```bash
# For numeric fields (can be used in KPI calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ItemsProcessed" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "SuccessRate" --type NUMBER
olakai custom-data create --agent-id YOUR_AGENT_ID --name "StepCount" --type NUMBER

# For string fields (for filtering/grouping, not calculations)
olakai custom-data create --agent-id YOUR_AGENT_ID --name "ExecutionId" --type STRING

# Verify all configs are created
olakai custom-data list --agent-id YOUR_AGENT_ID
```

### 2.4 Create KPI Definitions

#### Quick Start with Templates

Instead of writing formulas from scratch, use predefined classifier templates:

```bash
# List available templates
olakai kpis templates

# Create a classifier KPI from a template
olakai kpis create --name "User Satisfaction" \
  --calculator-id classifier --template-id sentiment_scorer \
  --scope CHAT --agent-id $AGENT_ID

# Create a time-saved estimator
olakai kpis create --name "Time Saved" \
  --calculator-id classifier --template-id time_saved_estimator \
  --scope CHAT --agent-id $AGENT_ID
```

#### Custom Formula KPIs

```bash
# Variable passthrough
olakai kpis create \
  --name "Items Processed" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "ItemsProcessed" \
  --unit "items" \
  --aggregation SUM

# Percentage calculation
olakai kpis create \
  --name "Success Rate" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "SuccessRate * 100" \
  --unit "%" \
  --aggregation AVERAGE

# Conditional counting
olakai kpis create \
  --name "Error Count" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "IF(SuccessRate < 1, 1, 0)" \
  --unit "errors" \
  --aggregation SUM

# Validate formulas before creating
olakai kpis validate --formula "ItemsProcessed" --agent-id YOUR_AGENT_ID
```

## Step 3: Implement SDK Integration

### 3.1 TypeScript Implementation

**Install dependencies:**
```bash
npm install @olakai/sdk openai
```

**Initialize and track:**
```typescript
import { olakaiConfig, olakai } from "@olakai/sdk";
import OpenAI from "openai";

// Initialize Olakai
olakaiConfig({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: process.env.NODE_ENV === "development",
});

// Create LLM client
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// Use wrapped client — monitoring happens automatically
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: userPrompt }],
});
```

**Agentic workflow with manual event tracking:**

> **`taskExecutionId` — Cross-Agent Task Correlation.** Generate ONE `taskExecutionId` per task and share it across all agents in a multi-agent workflow. This links events from different agents into a single logical task for analytics.

```typescript
async function runAgent(input: string): Promise<string> {
  const startTime = Date.now();
  const executionId = crypto.randomUUID();
  const taskExecutionId = crypto.randomUUID();
  let totalTokens = 0;
  let stepCount = 0;
  let itemsProcessed = 0;

  try {
    // Step 1: Planning
    stepCount++;
    const plan = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: `Plan: ${input}` }],
    });
    totalTokens += plan.usage?.total_tokens ?? 0;

    // Step 2: Process items
    const items = parseItems(plan.choices[0].message.content);
    for (const item of items) {
      stepCount++;
      const result = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: `Process: ${item}` }],
      });
      totalTokens += result.usage?.total_tokens ?? 0;
      itemsProcessed++;
    }

    // Step 3: Summarize
    stepCount++;
    const summary = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [{ role: "user", content: "Summarize results" }],
    });
    totalTokens += summary.usage?.total_tokens ?? 0;

    const finalResponse = summary.choices[0].message.content ?? "";

    // Track the complete workflow as a single event
    // Only send fields that have CustomDataConfigs (from Step 2.3)
    olakai("event", "ai_activity", {
      prompt: input,
      response: finalResponse,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 1.0,
      },
    });

    return finalResponse;
  } catch (error) {
    // Track failed execution — same fields, different values
    olakai("event", "ai_activity", {
      prompt: input,
      response: `Error: ${error instanceof Error ? error.message : "Unknown"}`,
      tokens: totalTokens,
      requestTime: Date.now() - startTime,
      taskExecutionId,
      task: "Data Processing & Analysis",
      customData: {
        ExecutionId: executionId,
        StepCount: stepCount,
        ItemsProcessed: itemsProcessed,
        SuccessRate: 0,
      },
    });
    throw error;
  }
}
```

### 3.2 Python Implementation

**Install dependencies:**
```bash
pip install olakai-sdk openai
```

**Initialize and track:**
```python
import os
from olakaisdk import olakai_config, olakai, OlakaiEventParams
from openai import OpenAI

# Initialize Olakai
olakai_config(os.getenv("OLAKAI_API_KEY"))

# Create OpenAI client
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
```

**Agentic workflow:**
```python
import time
import uuid

def run_agent(input_text: str) -> str:
    start_time = time.time()
    execution_id = str(uuid.uuid4())
    task_execution_id = str(uuid.uuid4())
    total_tokens = 0
    step_count = 0
    items_processed = 0

    try:
        # Your workflow steps here...
        step_count += 1
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": input_text}]
        )
        total_tokens += response.usage.total_tokens
        final_response = response.choices[0].message.content

        # Track successful execution
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=final_response,
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 1.0,
            }
        ))

        return final_response

    except Exception as e:
        olakai("event", "ai_activity", OlakaiEventParams(
            prompt=input_text,
            response=f"Error: {str(e)}",
            tokens=total_tokens,
            requestTime=int((time.time() - start_time) * 1000),
            taskExecutionId=task_execution_id,
            task="Data Processing & Analysis",
            customData={
                "ExecutionId": execution_id,
                "StepCount": step_count,
                "ItemsProcessed": items_processed,
                "SuccessRate": 0,
            }
        ))
        raise
```

### 3.3 REST API Direct Integration

For other languages or custom integrations:

```bash
curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "prompt": "User input here",
    "response": "Agent response here",
    "app": "your-agent-name",
    "task": "Data Processing & Analysis",
    "tokens": 1500,
    "requestTime": 5000,
    "customData": {
      "ExecutionId": "abc-123",
      "StepCount": 5,
      "ItemsProcessed": 10,
      "SuccessRate": 1.0
    }
  }'
```

## Step 4: Test-Validate-Iterate Cycle

**Always validate your implementation by running a test and inspecting the actual event data.**

### 4.1 Run Your Agent

Execute your agent with test data to generate at least one event.

### 4.2 Fetch and Inspect the Event

```bash
olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json
olakai activity get EVENT_ID --json
```

### 4.3 Validate Each Component

**Check customData is present and correct:**
```bash
olakai activity get EVENT_ID --json | jq '.customData'
```

**Check KPIs are numeric (not strings):**
```bash
olakai activity get EVENT_ID --json | jq '.kpiData'
```

**CORRECT** — numeric values:
```json
{
  "Items Processed": 10,
  "Success Rate": 100
}
```

**WRONG** — string values (broken formula):
```json
{
  "Items Processed": "itemsProcessed"
}
```
Fix: `olakai kpis update KPI_ID --formula "YourVariable"`

**WRONG** — null values:
Fix by verifying:
1. CustomDataConfig exists: `olakai custom-data list --agent-id ID`
2. Field name case matches exactly (case-sensitive)
3. SDK actually sends the field in customData

### 4.4 Validation Flow

```
1. Run agent (generate event)
           ↓
2. Fetch event: olakai activity get ID --json
           ↓
3. Check customData present? NO → Fix SDK code
           ↓
4. Check kpiData numeric? NO → Fix formula
           ↓
5. Check kpiData not null? NO → Create CustomDataConfig or fix field name
           ↓
✅ All validations pass — implementation complete
```

## Step 5: Production Checklist

Before deploying to production:

- [ ] API key stored securely in environment variables
- [ ] Error handling wraps all LLM calls
- [ ] Failed executions still report events (with SuccessRate: 0)
- [ ] All custom data fields have corresponding CustomDataConfig entries
- [ ] KPI formulas validated and showing numeric values (not strings)
- [ ] SDK configured with appropriate retries and timeouts
- [ ] Sensitive data redaction enabled if needed

## KPI Formula Reference

### Supported Operators

| Category | Operators |
|----------|-----------|
| Arithmetic | `+`, `-`, `*`, `/` |
| Comparison | `<`, `<=`, `=`, `<>`, `>=`, `>` |
| Logical | `AND`, `OR`, `NOT` |
| Conditional | `IF(condition, true_val, false_val)`, `MAP(value, match1, out1, default)` |
| Math | `ABS`, `MAX`, `MIN`, `AVERAGE`, `TRUNC` |
| Null handling | `ISNA(value)`, `ISDEFINED(value)`, `NA()` |

### Common Formula Patterns

```bash
--formula "ItemsProcessed"                              # passthrough
--formula "SuccessRate * 100"                            # percentage (0-1 to 0-100)
--formula "IF(SuccessRate < 1, 1, 0)"                    # conditional counting
--formula "IF(PII detected, 1, 0)"                       # built-in variable
--formula "IF(ISDEFINED(MyField), MyField, 0)"           # null-safe
--formula "IF(AND(StepCount > 5, SuccessRate < 0.9), 1, 0)"  # compound conditions
```

### Aggregation Types

| Aggregation | Use For | Example |
|-------------|---------|---------|
| `SUM` | Totals, counts | Total items processed across all runs |
| `AVERAGE` | Rates, percentages | Average success rate |

## Task Categories Reference

Use these predefined task categories for the `task` field:

| Category | Example Use |
|----------|-------------|
| Research & Intelligence | Competitive intelligence, market research |
| Data Processing & Analysis | Data extraction, statistical analysis |
| Content Development | Blog writing, technical documentation |
| Content Refinement | Editing, proofreading |
| Customer Experience | Complaint resolution, ticket triage |
| Software Development | Code generation, code review, debugging |
| Strategic Planning | Roadmap development, scenario planning |

## Quick Reference

```bash
# CLI Commands
olakai login                                              # Authenticate
olakai workflows create --name "Name" --json              # Create workflow
olakai agents create --name "Name" --workflow ID --with-api-key  # Register agent
olakai custom-data create --agent-id ID --name X --type NUMBER   # Create custom field
olakai kpis create --formula "X" --agent-id ID            # Create KPI
olakai activity list --agent-id ID                        # View events
```

```typescript
// TypeScript SDK
import { olakaiConfig, olakai } from "@olakai/sdk";
olakaiConfig({ apiKey: process.env.OLAKAI_API_KEY });

olakai("event", "ai_activity", {
  prompt: "input",
  response: "output",
  tokens: 1500,
  task: "Data Processing & Analysis",
  customData: { StepCount: 3, Success: 1 },
});
```

```python
# Python SDK
from olakaisdk import olakai_config, olakai, OlakaiEventParams
olakai_config(os.getenv("OLAKAI_API_KEY"))

olakai("event", "ai_activity", OlakaiEventParams(
    prompt="input",
    response="output",
    tokens=1500,
    task="Data Processing & Analysis",
    customData={"StepCount": 3, "Success": 1},
))
```