AI agent skill · by @kubesphere

Vector

Use when installing or configuring the WizTelemetry Data Pipeline (vector) extension for KubeSphere, which provides data collection, transformation, and routing for observability data including logs, auditing, events, and notifications

Archived 2026-10-23 · Source updated 2026-07-15. Check the original repository for the latest version.

SKILL.md instructions

# WizTelemetry Data Pipeline (Vector)

## Overview

WizTelemetry Data Pipeline is an extension based on vector (https://vector.dev/) that provides the ability to collect, transform, and route observability data. It is a core dependency for other WizTelemetry extensions like Logging, Auditing, Events, and Notification.

## When to Use

- Installing or configuring the WizTelemetry Data Pipeline extension
- Setting up data collection for logs, auditing, events, and notifications
- Configuring Vector sinks (OpenSearch)
- Managing Vector agent components

## Installation

### Prerequisites

**REQUIRED: Complete all steps in order before generating InstallPlan.**

#### Step 1: Get Available Clusters and Confirm Target

**⚠️ CRITICAL: DO NOT proceed until target clusters are determined.**

**Step 1.1: Get available clusters**

```bash
kubectl get clusters -o jsonpath='{.items[*].metadata.name}'
```

**Step 1.2: Determine target clusters**

- If user **explicitly specified** target clusters in the request → Use those clusters directly, proceed to Step 2
- If user **did NOT specify** target clusters → Ask user to confirm which clusters to deploy to, then proceed to Step 2

**Ask user (if not specified):**
```
Available clusters: host, dev
Which clusters do you want to deploy Vector to?
```

#### Step 2: Get OpenSearch Endpoint and Credentials (MUST DO)

- If user **already provided** OpenSearch endpoint and credentials in the request → Use those directly, proceed to Step 3
- If user **did NOT provide** → **You MUST ask user** for OpenSearch endpoint and credentials

**Ask user for (if not provided):**

1. **OpenSearch endpoint URL** (required)
   - Example: `http://<node-ip>:30920` or `https://opensearch.example.com:9200`

2. **OpenSearch credentials** (required)
   - Username (default: `admin`)
   - Password

**DO NOT proceed to Step 3 until user provides both endpoint and credentials.**

#### Step 3: Get Latest Vector Version (if not provided by user)

**MUST do this to get the latest version:**

```bash
kubectl get extensionversions -l kubesphere.io/extension-ref=vector -o jsonpath='{range .items[*]}{.spec.version}{"\n"}{end}' | sort -V | tail -1
```

This outputs the latest version (e.g., `1.1.4`). Note this down - you'll use it in the InstallPlan.

### Install Vector Extension

**⚠️ IMPORTANT: Complete prerequisite steps (1-3) BEFORE this step.**

**⚠️ CRITICAL: InstallPlan `metadata.name` MUST be `vector`. DO NOT use any other name.**

Based on your selections:
- **Target clusters**: Use the user-confirmed cluster names
- **OpenSearch endpoint**: User-provided endpoint
- **OpenSearch credentials**: User-provided username and password

**⚠️ CRITICAL: `config` field is YAML format. You MUST:**
- Use the config structure exactly as shown in the template
- **DO NOT** add configuration fields that are not shown in the template
- **DO NOT** modify the structure or hierarchy

**⚠️ CRITICAL: All placeholders MUST be replaced with actual values. DO NOT leave them as placeholders.**

#### Template

```yaml
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>  # From Step 3
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>
```

**Replace placeholders:**
- `<VECTOR_VERSION>`: From Step 2 (e.g., `1.1.4`)
- `<OPENSEARCH_ENDPOINT>`: User-provided endpoint (e.g., `http://<node-ip>:30920`)
- `<OPENSEARCH_USER>`: User-provided username (default: `admin`)
- `<OPENSEARCH_PASSWORD>`: User-provided password
- `<TARGET_CLUSTERS>`: User-confirmed cluster names

**⚠️ DO NOT generate InstallPlan until all placeholders have real values.**

### Wait for Deployment

**After applying InstallPlan, you MUST wait for deployment to complete:**

```bash
# Wait for Vector pods to be ready (on each cluster)
kubectl wait --for=condition=Ready pods -n kubesphere-logging-system -l app.kubernetes.io/instance=vector --timeout=300s

# Verify deployment status
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/instance=vector
```

**Show deployment summary to user:**
- Which clusters Vector was deployed to
- OpenSearch endpoint used
- Pod status (Ready/Total)

#### Enable Metrics Export

```yaml
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
      exportMetrics:
        enabled: true
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>
```

## Configuration Parameters

### Agent Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `agent.role` | string | "Agent" | Role identifier |
| `agent.image.tag` | string | "0.53.0-debian" | Vector image tag |
| `agent.resources.requests.cpu` | string | "100m" | CPU request |
| `agent.resources.requests.memory` | string | "100Mi" | Memory request |
| `agent.resources.limits.cpu` | string | "2000m" | CPU limit |
| `agent.resources.limits.memory` | string | "2000Mi" | Memory limit |
| `agent.service.ports` | list | see values.yaml | Service ports |
| `agent.exportMetrics.enabled` | bool | false | Enable metrics export |

### Agent Sinks OpenSearch Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `agent.sinks.opensearch.endpoints` | list | Yes | OpenSearch endpoint URLs |
| `agent.sinks.opensearch.auth.strategy` | string | Yes | Authentication strategy (set to `basic`) |
| `agent.sinks.opensearch.auth.user` | string | Yes | Username for authentication |
| `agent.sinks.opensearch.auth.password` | string | Yes | Password for authentication |
| `agent.sinks.opensearch.tls.verify` | bool | No | Enable TLS verification (default: false) |

**Example:**

```yaml
agent:
  sinks:
    opensearch:
      endpoints:
        - http://<node-ip>:30920
      auth:
        strategy: basic
        user: admin
        password: admin
      tls:
        verify: false
```

### Docker Root Directory Configuration

If Docker root directory is not `/var/lib`:

```yaml
agent:
  extraVolumes:
    - name: docker-root
      hostPath:
        path: /path/to/docker
        type: ''
  extraVolumeMounts:
    - name: docker-root
      mountPath: /path/to/docker
```

## Extension Operations

### Check Extension Status

```bash
# View extension installation status
kubectl get installplan vector

# View extension version
kubectl get extensionversions -l kubesphere.io/extension-ref=vector
```

### Check Pod Status

```bash
# View all Vector pods
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/name=vector

# View agent pods
kubectl get pods -n kubesphere-logging-system -l app.kubernetes.io/name=vector,app.kubernetes.io/component=agent
```

### View Logs

```bash
# View agent logs
kubectl logs -n kubesphere-logging-system -l app.kubernetes.io/name=vector,app.kubernetes.io/component=agent --tail=100
```

### Update Configuration

```yaml
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>
```

### Uninstall Extension

**Uninstall from all clusters:**

```bash
kubectl delete installplan vector
```

**Uninstall from specific cluster:**

To remove Vector from a specific cluster, update the InstallPlan by removing that cluster from `clusterScheduling.placement.clusters`:

```yaml
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: vector
spec:
  extension:
    name: vector
    version: <VECTOR_VERSION>
  enabled: true
  upgradeStrategy: Manual
  config: |
    agent:
      sinks:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
  clusterScheduling:
    placement:
      clusters:
        - <REMAINING_CLUSTERS>  # Remove the cluster you want to uninstall from
```

## Important Notes

1. **Dependency**: Vector is a core dependency for WizTelemetry extensions. Install it first before installing Logging, Auditing, Events, or Notification.
2. **OpenSearch Required**: User must provide OpenSearch endpoint and credentials.
3. **Multicluster**: The extension uses `installationMode: Multicluster`:
   - `agent` (tag: agent) is deployed to all selected member clusters
4. **Agent Scheduling**: Agent pods have affinity to avoid edge nodes and tolerate all taints.
5. **Cross-cluster Access**: Ensure OpenSearch endpoint is accessible from all Vector clusters.

## Troubleshooting

### Check Vector Configuration

```bash
# View Vector configmap
kubectl get configmap -n kubesphere-logging-system -l app.kubernetes.io/name=vector

# View specific config
kubectl get configmap -n kubesphere-logging-system vector-config -o yaml
```

### Verify Sinks

```bash
# Check if sinks are configured correctly
kubectl get secret -n kubesphere-logging-system vector-sinks -o yaml
```

### Common Issues

| Issue | Solution |
|-------|----------|
| Pods not starting | Check if OpenSearch is accessible |
| Data not flowing | Verify sink configuration and network connectivity |
| Agent not on member cluster | Check multicluster installation settings |
| Out of memory | Increase resource limits in configuration |