Monitoring & Analytics

Monitoring & Analytics

This guide covers how to effectively monitor and analyze your AI usage with UniCraft’s comprehensive monitoring and analytics features.

Overview

UniCraft provides detailed monitoring and analytics to help you:

• Track usage patterns and costs
• Monitor performance metrics
• Identify optimization opportunities
• Set up alerts and notifications
• Generate reports and insights

Dashboard Overview

Key Metrics Dashboard

The main dashboard displays essential metrics at a glance:

// Get dashboard metrics
const metrics = await unicraft.analytics.getDashboardMetrics({
  time_range: "7d",
  group_by: "day",
});

console.log("Total Requests:", metrics.total_requests);
console.log("Total Cost:", metrics.total_cost);
console.log("Average Response Time:", metrics.avg_response_time);
console.log("Success Rate:", metrics.success_rate);

Real-time Monitoring

Monitor your AI usage in real-time:

// Subscribe to real-time metrics
const subscription = unicraft.analytics.subscribe({
  metrics: ["requests", "cost", "response_time", "errors"],
  callback: (data) => {
    console.log("Real-time update:", data);
  },
});

Usage Analytics

Request Patterns

Analyze your request patterns to understand usage:

// Get usage patterns
const patterns = await unicraft.analytics.getUsagePatterns({
  start_date: "2024-01-01",
  end_date: "2024-01-31",
  group_by: "hour",
  metrics: ["requests", "tokens", "cost"],
});

// Analyze peak usage times
const peakHours = patterns.filter((p) => p.requests > average_requests * 1.5);
console.log("Peak usage hours:", peakHours);

Provider Comparison

Compare performance across different providers:

// Compare providers
const comparison = await unicraft.analytics.compareProviders({
  time_range: "30d",
  metrics: ["response_time", "cost", "success_rate", "availability"],
});

console.log("Provider comparison:", comparison);

Model Performance

Track performance metrics for each model:

// Get model performance
const modelPerformance = await unicraft.analytics.getModelPerformance({
  time_range: "7d",
  models: ["gpt-4", "gpt-3.5-turbo", "claude-3-sonnet"],
});

modelPerformance.forEach((model) => {
  console.log(`${model.name}:`);
  console.log(`  Avg Response Time: ${model.avg_response_time}ms`);
  console.log(`  Cost per Request: $${model.avg_cost}`);
  console.log(`  Success Rate: ${model.success_rate}%`);
});

Cost Analytics

Spending Trends

Monitor your spending patterns over time:

// Get cost trends
const costTrends = await unicraft.analytics.getCostTrends({
  start_date: "2024-01-01",
  end_date: "2024-01-31",
  group_by: "day",
  breakdown: ["provider", "model"],
});

// Calculate daily averages
const dailyAverage =
  costTrends.reduce((sum, day) => sum + day.cost, 0) / costTrends.length;
console.log("Daily average cost:", dailyAverage);

Cost Optimization Insights

Get recommendations for cost optimization:

// Get optimization recommendations
const recommendations = await unicraft.analytics.getOptimizationRecommendations(
  {
    time_range: "30d",
  }
);

recommendations.forEach((rec) => {
  console.log(`Recommendation: ${rec.title}`);
  console.log(`Potential savings: $${rec.potential_savings}`);
  console.log(`Description: ${rec.description}`);
});

Budget Management

Set up and monitor spending budgets:

// Create budget
const budget = await unicraft.budgets.create({
  name: "Monthly AI Budget",
  amount: 1000,
  period: "monthly",
  alerts: [0.5, 0.8, 0.9, 1.0], // Alert at 50%, 80%, 90%, and 100%
});

// Get budget status
const budgetStatus = await unicraft.budgets.getStatus(budget.id);
console.log("Budget status:", budgetStatus);

Performance Monitoring

Response Time Tracking

Monitor response times across providers and models:

// Get response time metrics
const responseTimes = await unicraft.analytics.getResponseTimes({
  time_range: "7d",
  group_by: "provider",
  percentiles: [50, 90, 95, 99],
});

responseTimes.forEach((provider) => {
  console.log(`${provider.name}:`);
  console.log(`  P50: ${provider.p50}ms`);
  console.log(`  P90: ${provider.p90}ms`);
  console.log(`  P95: ${provider.p95}ms`);
  console.log(`  P99: ${provider.p99}ms`);
});

Error Rate Monitoring

Track and analyze errors:

// Get error analysis
const errorAnalysis = await unicraft.analytics.getErrorAnalysis({
  time_range: "7d",
  group_by: "error_type",
});

errorAnalysis.forEach((error) => {
  console.log(`${error.type}: ${error.count} occurrences`);
  console.log(`  Rate: ${error.rate}%`);
  console.log(`  Trend: ${error.trend}`);
});

Availability Monitoring

Monitor provider availability:

// Get availability metrics
const availability = await unicraft.analytics.getAvailability({
  time_range: "30d",
  providers: ["openai", "anthropic", "google"],
});

availability.forEach((provider) => {
  console.log(`${provider.name}:`);
  console.log(`  Uptime: ${provider.uptime}%`);
  console.log(`  Downtime: ${provider.downtime} minutes`);
  console.log(`  Incidents: ${provider.incidents}`);
});

Custom Dashboards

Creating Custom Dashboards

Create custom dashboards for specific use cases:

// Create custom dashboard
const dashboard = await unicraft.dashboards.create({
  name: "Production Monitoring",
  description: "Monitor production AI usage",
  widgets: [
    {
      type: "line_chart",
      title: "Request Volume",
      metric: "requests",
      time_range: "24h",
      group_by: "hour",
    },
    {
      type: "gauge",
      title: "Success Rate",
      metric: "success_rate",
      threshold: 0.95,
      time_range: "1h",
    },
    {
      type: "bar_chart",
      title: "Cost by Provider",
      metric: "cost",
      group_by: "provider",
      time_range: "7d",
    },
  ],
});

Dashboard Widgets

Available widget types:

const widgetTypes = {
  line_chart: "Time series data visualization",
  bar_chart: "Categorical data comparison",
  pie_chart: "Proportional data representation",
  gauge: "Single metric with threshold",
  table: "Tabular data display",
  metric: "Single value display",
};

Alerts and Notifications

Setting Up Alerts

Configure alerts for important metrics:

// Create alert
const alert = await unicraft.alerts.create({
  name: "High Error Rate",
  description: "Alert when error rate exceeds 5%",
  condition: "error_rate > 0.05",
  duration: "5m",
  channels: ["email", "slack"],
  recipients: ["admin@company.com", "#ai-alerts"],
});

// Create cost alert
const costAlert = await unicraft.alerts.create({
  name: "Budget Exceeded",
  description: "Alert when monthly budget is exceeded",
  condition: "monthly_cost > budget_amount",
  duration: "1m",
  channels: ["email", "webhook"],
  webhook_url: "https://your-app.com/webhooks/budget-alert",
});

Alert Channels

Configure different notification channels:

// Email notifications
const emailChannel = {
  type: "email",
  recipients: ["admin@company.com", "team@company.com"],
  template: "alert_template",
};

// Slack notifications
const slackChannel = {
  type: "slack",
  webhook_url: "https://hooks.slack.com/services/...",
  channel: "#ai-alerts",
};

// Webhook notifications
const webhookChannel = {
  type: "webhook",
  url: "https://your-app.com/webhooks/unicraft-alerts",
  headers: {
    Authorization: "Bearer your-token",
  },
};

Reporting

Automated Reports

Schedule automated reports:

// Create scheduled report
const report = await unicraft.reports.create({
  name: "Weekly AI Usage Report",
  template: "weekly_usage",
  schedule: "0 9 * * 1", // Every Monday at 9 AM
  recipients: ["team@company.com"],
  format: "pdf",
  include_charts: true,
});

Custom Reports

Generate custom reports:

// Generate custom report
const customReport = await unicraft.reports.generate({
  name: "Q1 AI Analysis",
  start_date: "2024-01-01",
  end_date: "2024-03-31",
  sections: [
    "usage_summary",
    "cost_analysis",
    "performance_metrics",
    "provider_comparison",
  ],
  format: "pdf",
  include_raw_data: true,
});

Export Options

Export data in various formats:

// Export usage data
const usageData = await unicraft.analytics.export({
  start_date: "2024-01-01",
  end_date: "2024-01-31",
  format: "csv",
  include_breakdown: true,
});

// Export cost data
const costData = await unicraft.analytics.export({
  start_date: "2024-01-01",
  end_date: "2024-01-31",
  format: "json",
  group_by: "provider",
});

Integration

Webhook Integration

Set up webhooks for real-time notifications:

// Create webhook
const webhook = await unicraft.webhooks.create({
  name: "Usage Notifications",
  url: "https://your-app.com/webhooks/unicraft",
  events: [
    "request.completed",
    "error.occurred",
    "budget.exceeded",
    "provider.unavailable",
  ],
  secret: "your-webhook-secret",
});

Third-party Integrations

Integrate with popular monitoring tools:

// Grafana integration
const grafanaConfig = {
  type: "grafana",
  url: "https://grafana.company.com",
  api_key: "your-grafana-api-key",
  dashboard_id: "unicraft-dashboard",
};

// DataDog integration
const datadogConfig = {
  type: "datadog",
  api_key: "your-datadog-api-key",
  app_key: "your-datadog-app-key",
  metrics: ["response_time", "cost", "success_rate"],
};

Best Practices

Monitoring Strategy

1. Define Key Metrics: Identify the most important metrics for your use case
2. Set Appropriate Thresholds: Configure alerts with realistic thresholds
3. Regular Reviews: Schedule regular reviews of monitoring data
4. Automate Responses: Set up automated responses for common issues
5. Document Procedures: Document procedures for handling alerts

Performance Optimization

1. Baseline Establishment: Establish performance baselines
2. Trend Analysis: Look for trends in performance data
3. Capacity Planning: Use historical data for capacity planning
4. Continuous Improvement: Regularly optimize based on monitoring insights

Cost Management

1. Budget Planning: Use historical data for budget planning
2. Cost Allocation: Implement cost allocation by project or team
3. Optimization Opportunities: Regularly identify optimization opportunities
4. ROI Analysis: Analyze return on investment for AI usage

Troubleshooting

Common Monitoring Issues

1. Missing Data: Check data collection configuration
2. Inaccurate Metrics: Verify metric calculation logic
3. Alert Fatigue: Adjust alert thresholds and frequency
4. Performance Impact: Monitor monitoring system performance

Debug Mode

Enable debug mode for troubleshooting:

// Enable debug mode
const debugMetrics = await unicraft.analytics.getMetrics({
  time_range: "1h",
  debug: true,
});

console.log("Debug info:", debugMetrics.debug);

Next Steps

After setting up monitoring:

1. Review and optimize alert thresholds
2. Set up automated reports
3. Integrate with existing monitoring tools
4. Train team on monitoring procedures
5. Regularly review and update monitoring configuration