PRODUCTION_DEPLOYMENT_GUIDE
AIOSx Production Deployment Guide
🚀 Production Deployment Checklist
This guide covers deploying AIOSx to real production with:
- Helm chart deployment
- Real customer clusters
- External service integrations (IBKR, Coinbase, Twilio, Zendesk, Salesforce)
- PoV flows with real business data
- CorporateBrain 24/7 operation
- Live ROI collection
Prerequisites
Infrastructure Requirements
- ✅ Kubernetes cluster (1.20+) with 16GB+ RAM
- ✅ Helm 3.x installed
- ✅ PostgreSQL 15+ database
- ✅ Redis (optional, for caching)
- ✅ Domain names configured (DNS)
- ✅ SSL certificates (Let's Encrypt or custom)
- ✅ External secret management (AWS Secrets Manager, HashiCorp Vault, or Kubernetes Secrets)
External Service Accounts
- ✅ Trading: IBKR (Interactive Brokers), Coinbase Pro, Binance
- ✅ Payments: Stripe, PayPal (✅ Implemented - See Billing Integration)
- ✅ Communication: Twilio (SMS/Voice)
- ✅ Support: Zendesk, Jira, ServiceNow
- ✅ CRM: Salesforce, HubSpot
- ✅ Monitoring: Prometheus, Grafana (or Datadog, New Relic)
Production Features Status
- ✅ Enhanced JWT Authentication: Database-backed with PostgreSQL (See Authentication Guide)
- ✅ HTTPS/TLS: Automatic certificate management (See Production Features)
- ✅ Stripe & PayPal Billing: Real payment processing with webhooks
- ✅ Ops Copilot: Natural language chat interface
- ✅ Predictive Self-Healing: AI-powered failure prediction
Step 1: Deploy Helm Chart
1.1 Prepare Helm Values
Create production-values.yaml:
yamlglobal:domain: aiosx.comapiDomain: api.aiosx.comconsoleDomain: console.aiosx.comenvironment: production# Databasepostgres:enabled: trueauth:username: aiosxpassword: ${POSTGRES_PASSWORD} # From secretdatabase: aiosxprimary:persistence:enabled: truesize: 200GistorageClass: gp3# Redis (optional)redis:enabled: truearchitecture: standalonemaster:persistence:enabled: truesize: 50Gi# Backendbackend:enabled: truereplicaCount: 3image:repository: aiosx/backendtag: "1.0.0"env:# DatabaseDATABASE_URL: "postgresql://aiosx:${POSTGRES_PASSWORD}@aiosx-postgres:5432/aiosx"REDIS_URL: "redis://aiosx-redis-master:6379/0"# External ServicesIBKR_HOST: "127.0.0.1"IBKR_PORT: "7497" # Paper: 7497, Live: 7496COINBASE_API_KEY: "${COINBASE_API_KEY}"COINBASE_API_SECRET: "${COINBASE_API_SECRET}"TWILIO_ACCOUNT_SID: "${TWILIO_ACCOUNT_SID}"TWILIO_AUTH_TOKEN: "${TWILIO_AUTH_TOKEN}"ZENDESK_SUBDOMAIN: "${ZENDESK_SUBDOMAIN}"ZENDESK_EMAIL: "${ZENDESK_EMAIL}"ZENDESK_API_TOKEN: "${ZENDESK_API_TOKEN}"SALESFORCE_USERNAME: "${SALESFORCE_USERNAME}"SALESFORCE_PASSWORD: "${SALESFORCE_PASSWORD}"SALESFORCE_SECURITY_TOKEN: "${SALESFORCE_SECURITY_TOKEN}"STRIPE_API_KEY: "${STRIPE_API_KEY}"# EnvironmentAIOSX_ENV: "production"AIOSX_DEBUG: "false"LOG_LEVEL: "INFO"# Public URLsPUBLIC_API_BASE_URL: "https://api.aiosx.com"PUBLIC_FRONTEND_BASE_URL: "https://aiosx.com"resources:requests:cpu: "500m"memory: "1Gi"limits:cpu: "2000m"memory: "4Gi"# Frontendfrontend:enabled: truereplicaCount: 2ingress:enabled: truehosts:- host: aiosx.compaths:- path: /pathType: Prefixtls:- secretName: aiosx-tlshosts:- aiosx.com# Domain Kernelskernels:trading:enabled: truereplicas: 2resources:requests:cpu: "2"memory: "4Gi"limits:cpu: "4"memory: "8Gi"defi_fx:enabled: truereplicas: 2llm:enabled: truereplicas: 3resources:requests:cpu: "4"memory: "8Gi"limits:cpu: "8"memory: "16Gi"voiceos:enabled: truereplicas: 2security:enabled: truereplicas: 2# Agent MeshagentMesh:enabled: truereplicas: 5# Monitoringmonitoring:prometheus:enabled: truegrafana:enabled: trueadminPassword: "${GRAFANA_PASSWORD}"# Backupbackup:enabled: trueschedule: "0 2 * * *" # Daily at 2 AMretentionDays: 30
1.2 Create Kubernetes Secrets
bash# Create namespacekubectl create namespace aiosx# Create secretskubectl create secret generic aiosx-secrets \--from-literal=postgres-password='YOUR_SECURE_PASSWORD' \--from-literal=coinbase-api-key='YOUR_COINBASE_KEY' \--from-literal=coinbase-api-secret='YOUR_COINBASE_SECRET' \--from-literal=twilio-account-sid='YOUR_TWILIO_SID' \--from-literal=twilio-auth-token='YOUR_TWILIO_TOKEN' \--from-literal=zendesk-subdomain='YOUR_ZENDESK_SUBDOMAIN' \--from-literal=zendesk-email='YOUR_ZENDESK_EMAIL' \--from-literal=zendesk-api-token='YOUR_ZENDESK_TOKEN' \--from-literal=salesforce-username='YOUR_SALESFORCE_USERNAME' \--from-literal=salesforce-password='YOUR_SALESFORCE_PASSWORD' \--from-literal=salesforce-security-token='YOUR_SALESFORCE_TOKEN' \--from-literal=stripe-api-key='YOUR_STRIPE_KEY' \--from-literal=grafana-password='YOUR_GRAFANA_PASSWORD' \--namespace=aiosx
1.3 Deploy with Helm
bash# Add Helm repository (if using remote)helm repo add aiosx https://charts.aiosx.comhelm repo update# Or use local chartscd deployment/helm/aiosx-umbrella# Installhelm install aiosx . \--namespace aiosx \--create-namespace \--values production-values.yaml \--set-file secrets=aiosx-secrets# Verify deploymenthelm status aiosx -n aiosxkubectl get pods -n aiosx
Step 2: Configure External Service Integrations
2.1 Trading Platforms (IBKR, Coinbase, Binance)
Interactive Brokers (IBKR)
python# Store IBKR credentials via APIPOST /monetization/credentials/store{"provider": "ibkr","secret_type": "api_key","value": "YOUR_IBKR_API_KEY","metadata": {"host": "127.0.0.1","port": 7497, # Paper trading"client_id": 1},"rotation_schedule_days": 90}
IBKR Connection Setup:
- Install IB Gateway or TWS
- Enable API access in TWS: Configure → API → Settings
- Set
Trusted IPsto your Kubernetes cluster IPs - Configure Trading Kernel to connect:
yamltrading:ibkr:enabled: truehost: "ib-gateway-host"port: 7497client_id: 1
Coinbase Pro
python# Store Coinbase credentialsPOST /monetization/credentials/store{"provider": "coinbase","secret_type": "api_key","value": "YOUR_COINBASE_API_KEY","metadata": {"api_secret": "YOUR_COINBASE_SECRET","passphrase": "YOUR_PASSPHRASE","sandbox": false # Set to true for testing}}
2.2 Communication (Twilio)
python# Configure Twilio for VoiceOS KernelPOST /monetization/credentials/store{"provider": "twilio","secret_type": "api_key","value": "YOUR_TWILIO_ACCOUNT_SID","metadata": {"auth_token": "YOUR_TWILIO_AUTH_TOKEN","phone_number": "+1234567890"}}
VoiceOS Configuration:
yamlvoiceos:twilio:enabled: trueaccount_sid: "${TWILIO_ACCOUNT_SID}"auth_token: "${TWILIO_AUTH_TOKEN}"phone_number: "+1234567890"
2.3 Support (Zendesk)
python# Configure Zendesk integrationPOST /monetization/credentials/store{"provider": "zendesk","secret_type": "api_token","value": "YOUR_ZENDESK_API_TOKEN","metadata": {"subdomain": "your-subdomain","email": "your-email@example.com"}}
Support Engine Configuration:
yamlsupport:zendesk:enabled: truesubdomain: "${ZENDESK_SUBDOMAIN}"email: "${ZENDESK_EMAIL}"api_token: "${ZENDESK_API_TOKEN}"
2.4 CRM (Salesforce)
python# Configure Salesforce integrationPOST /monetization/credentials/store{"provider": "salesforce","secret_type": "oauth_token","value": "YOUR_SALESFORCE_ACCESS_TOKEN","metadata": {"username": "your-username","password": "your-password","security_token": "your-security-token","instance_url": "https://your-instance.salesforce.com"}}
Sales Engine Configuration:
yamlsales:salesforce:enabled: trueusername: "${SALESFORCE_USERNAME}"password: "${SALESFORCE_PASSWORD}"security_token: "${SALESFORCE_SECURITY_TOKEN}"instance_url: "https://your-instance.salesforce.com"
Step 3: Deploy Real Customer Cluster
3.1 Create Customer Cluster
bash# Use customer-cluster Helm valueshelm install customer-cluster-1 \deployment/helm/customer-cluster \--namespace customer-1 \--create-namespace \--set global.region=us-east-1 \--set global.environment=production \--set global.replicaCount=3 \--set global.highAvailability=true \--set ingress.tls[0].hosts[0]=api.customer1.example.com
3.2 Provision Tenant
python# Create tenant via APIPOST /commercial/provisioning/create{"customer_name": "Acme Corp","customer_id": "acme-corp-001","domains": ["trading", "llm", "security"],"plan": "enterprise","regions": ["us-east-1", "us-west-2"]}# Execute provisioningPOST /commercial/provisioning/{plan_id}/execute
3.3 Verify Production Health
bash# Check production healthGET /commercial/production/health/{tenant_id}# Expected response:{"tenant_id": "acme-corp-001","status": "healthy","checks": {"slo_compliance": true,"logging_configured": true,"incident_routing": true,"backups_enabled": true,"dr_configured": true,"security_posture": "compliant"}}
Step 4: Run PoV Flows with Real Business Data
4.1 Create PoV Deployment
python# Generate PoV kitPOST /pov/generate{"prospect_name": "Beta Customer","prospect_id": "beta-001","capabilities": ["trading", "support", "llm", "security"],"sample_data": {"tickets": [{"id": "TICKET-001","subject": "Password reset request","priority": "high","status": "open"}],"sales_leads": [{"id": "LEAD-001","company": "Tech Corp","value": 50000,"stage": "qualified"}],"trading_history": [{"id": "TRADE-001","asset_pair": "BTC/USD","side": "buy","quantity": 1.0,"price": 50000.0}]}}# Deploy PoVPOST /pov/{deployment_id}/deploy
4.2 Execute PoV Playbook
python# Run Trading PoV PlaybookPOST /pov/playbooks/trading/execute{"tenant_id": "beta-001","duration_hours": 72,"goals": ["Demonstrate 10-20% improvement in trade execution speed","Show risk management capabilities"]}# Run Support PoV PlaybookPOST /pov/playbooks/support/execute{"tenant_id": "beta-001","duration_hours": 72,"goals": ["Demonstrate 30-50% reduction in resolution time","Show autonomous Level 1 support"]}
4.3 Get PoV Report
python# Get PoV resultsGET /pov/report/{tenant_id}# Response includes:{"deployment_id": "pov-001","status": "completed","workflows_executed": 150,"value_created": {"trading": {"execution_speed_improvement": "18%","risk_reduction": "25%"},"support": {"resolution_time_reduction": "42%","cost_savings": "$5,000/month"}},"improvements_achieved": ["Automated 80% of Level 1 support tickets","Reduced trade execution latency by 18%","Prevented 3 security incidents"]}
Step 5: Enable CorporateBrain 24/7 Operation
5.1 Initialize CorporateBrain
python# Start CorporateBrainPOST /autonomous/corporate-brain/start{"planning_horizon": "quarterly","strategic_objectives": [{"name": "Increase MRR by 20%","target_kpi": "mrr","target_value": 120000.0,"deadline": "2024-12-31T00:00:00Z","priority": 10},{"name": "Reduce support costs by 30%","target_kpi": "support_cost","target_value": 7000.0,"deadline": "2024-12-31T00:00:00Z","priority": 8}]}
5.2 Configure Long-Horizon Planning
python# Set planning loopPOST /autonomous/corporate-brain/planning-loop{"enabled": true,"interval_hours": 24, # Daily planning cycle"horizons": ["daily", "weekly", "monthly", "quarterly"],"objectives": ["revenue_growth","cost_optimization","customer_satisfaction","market_expansion"]}
5.3 Monitor CorporateBrain
bash# Check CorporateBrain statusGET /autonomous/corporate-brain/status# Get strategic planGET /autonomous/corporate-brain/plan# Get execution progressGET /autonomous/corporate-brain/progress
Step 6: Start Collecting Live ROI
6.1 Enable ROI Tracking
python# Start Trading ROI experimentPOST /customer/roi/experiments/trading/start{"tenant_id": "acme-corp-001","baseline_period_days": 30,"experiment_period_days": 90,"metrics": ["execution_speed","slippage_reduction","win_rate_improvement","risk_reduction"]}# Start Support ROI experimentPOST /customer/roi/experiments/support/start{"tenant_id": "acme-corp-001","baseline_period_days": 30,"experiment_period_days": 90,"metrics": ["resolution_time","cost_per_ticket","escalation_rate","customer_satisfaction"]}
6.2 View Live ROI Metrics
python# Get ROI summaryGET /customer/{tenant_id}/roi?period_start=2024-01-01&period_end=2024-03-31# Response:{"tenant_id": "acme-corp-001","period": {"start": "2024-01-01T00:00:00Z","end": "2024-03-31T23:59:59Z"},"roi_metrics": {"trading": {"execution_speed_improvement": "18%","slippage_reduction": "12%","win_rate_improvement": "8%","estimated_value": "$25,000/month"},"support": {"resolution_time_reduction": "42%","cost_per_ticket_reduction": "35%","escalation_rate_reduction": "28%","estimated_value": "$8,000/month"},"total_estimated_value": "$33,000/month","roi_percentage": "450%"}}
Step 7: Real-World Validation Experiments
7.1 Trading ROI Experiments
Note: Value experiment endpoints (
/value/trading/roi/experiment,/value/support/roi/experiment, etc.) are planned but not yet implemented. Use ROI tracking endpoints instead.
python# Start Trading ROI experiment (Planned - returns planned status)POST /customer/roi/experiments/trading/start{"tenant_id": "acme-corp-001","baseline_period_days": 30,"experiment_period_days": 90,"metrics": ["execution_speed","slippage_reduction","win_rate_improvement","risk_reduction"]}# Get ROI resultsGET /customer/{tenant_id}/roi?period_start=2024-01-01&period_end=2024-03-31
7.2 Support Cost Reduction
python# Start Support ROI experiment (Planned - returns planned status)POST /customer/roi/experiments/support/start{"tenant_id": "acme-corp-001","baseline_period_days": 30,"experiment_period_days": 90,"metrics": ["resolution_time","cost_per_ticket","escalation_rate","customer_satisfaction"]}
7.3 Sales Uplift
Note: Sales ROI experiments are planned but not yet implemented.
7.4 Live A/B Tests
Note: A/B testing endpoints (
/value/ab-test/start) are planned but not yet implemented.
7.5 Shadow Mode Tests
Note: Shadow mode testing endpoints (
/value/shadow-mode/start) are planned but not yet implemented.
Step 8: Prepare for Paying Customers
8.1 Configure Billing
python# Set up billing for tenantPOST /billing/tenant/{tenant_id}/configure{"plan": "enterprise","billing_cycle": "monthly","payment_method": "stripe","stripe_customer_id": "cus_xxxxx","usage_based_billing": true,"overage_rules": {"llm_requests": {"included": 10000,"overage_rate": 0.001},"api_requests": {"included": 100000,"overage_rate": 0.0001}}}
8.2 Set Up SLA Monitoring
python# Define SLA for tenantPOST /sla/define{"tenant_id": "acme-corp-001","sla_tier": "enterprise","slo_definitions": [{"metric": "api_latency_p95","target": 200, # milliseconds"error_budget": 0.01 # 1% error budget},{"metric": "uptime","target": 0.999, # 99.9% uptime"error_budget": 0.001}]}# Monitor SLA complianceGET /sla/report/{tenant_id}
8.3 Enable Marketplace
python# Publish workflow to marketplacePOST /monetization/marketplace/publish{"item_type": "workflow","name": "Elite Trade Flow","description": "Multi-kernel trading workflow","pricing_model": "usage_based","price": 0.10, # Per execution"creator_id": "aiosx"}# Enable marketplace for tenantPOST /monetization/marketplace/enable{"tenant_id": "acme-corp-001","marketplace_access": true}
8.4 Configure Multi-Region
python# Set up multi-region deploymentPOST /global/regions/configure{"regions": [{"region_id": "us-east-1","enabled": true,"primary": true,"kernels": ["trading", "llm", "security"]},{"region_id": "us-west-2","enabled": true,"primary": false,"kernels": ["llm", "security"]},{"region_id": "eu-west-1","enabled": true,"primary": false,"kernels": ["llm", "security"]}],"failover_policy": "automatic"}# Get global statusGET /global/status
Console UI
The AIOSx Console provides a comprehensive web interface accessible after authentication:
- URL:
http://localhost:3000(or your configured frontend URL) - Features: 23 feature tabs covering all platform capabilities
- Real-Time: WebSocket connection for live updates
- Interactive: Forms, search, filtering, charts, detailed views
- Documentation: See Console UI Guide for complete documentation
Quick Access
- Navigate to the login page
- Sign in with your credentials
- Access all features through the tabbed interface
- Use search and filters to find items quickly
- View real-time updates via WebSocket connection
Monitoring & Observability
Health Checks
bash# API healthcurl https://api.aiosx.com/health# Component healthcurl https://api.aiosx.com/health/component/kernel-registry# Mesh overviewcurl https://api.aiosx.com/ops/mesh/overview# SLO statuscurl https://api.aiosx.com/ops/slo
Metrics & Dashboards
- Prometheus:
http://prometheus.aiosx.com:9090 - Grafana:
http://grafana.aiosx.com:3000(admin/${GRAFANA_PASSWORD}) - API Metrics:
https://api.aiosx.com/metrics(Prometheus format)
Logs
bash# View backend logskubectl logs -f deployment/aiosx-backend -n aiosx# View kernel logskubectl logs -f deployment/trading-kernel -n aiosx# View all podskubectl get pods -n aiosx
Production Checklist
- Helm chart deployed successfully
- All pods running and healthy
- Database migrations completed
- External service credentials configured
- IBKR/Coinbase connections tested
- Twilio integration verified
- Zendesk/Salesforce connected
- Customer cluster provisioned
- PoV flows executed with real data
- CorporateBrain running 24/7
- ROI tracking enabled
- A/B tests configured
- Billing configured for tenants
- SLA monitoring active
- Multi-region failover tested
- Backup schedule configured
- Monitoring dashboards set up
- Alerting rules configured
- Security posture validated
- Compliance exports tested
Next Steps
- Monitor First 24 Hours: Watch all metrics, logs, and alerts
- Validate ROI Experiments: Review initial ROI data
- Optimize Performance: Use AKO recommendations
- Scale as Needed: Add replicas based on load
- Onboard First Paying Customer: Use PoV → Production flow
- Generate Investor Reports: Use
/investor/reportendpoint - Iterate and Improve: Use CorporateBrain insights
Support
- Documentation:
/docsdirectory - API Docs:
https://api.aiosx.com/docs - Health Status:
https://api.aiosx.com/health - Ops Console:
https://console.aiosx.com
🎉 You're now running AIOSx in REAL PRODUCTION!
Current Implementation Status
✅ Fully Implemented
- Core Services: Kernel Registry, Health Bus, Orchestrator, Workflow Engine, AKO Controller
- AGI Services: AgentMesh, ReasoningTracer, CognitiveLoops, PolicyEngine, SimCore
- Monetization Services: APIMonetization, PaymentAdapter (✅ Real Stripe/PayPal Integration), RevenueLedger, ProfitabilityEngine, ProductCatalog, Marketplace
- Production Features:
- ✅ Enhanced JWT Authentication (PostgreSQL-backed)
- ✅ HTTPS/TLS with cert-manager
- ✅ Stripe & PayPal Billing (real API integration)
- ✅ Ops Copilot (natural language interface)
- ✅ Predictive Self-Healing (Prometheus-based failure prediction)
- Enterprise Services: ProcessExecutor, EnterpriseMemory, RevenueEngine, Governance, KPIEngine, ResourcePlanner, HumanAIBridge, DigitalTwin
- Commercial Services: TenantProvisioningFlow, ProductionHealthChecklist, CustomerLifecycleManager, SLAEngine, BillingEngine
- PoV Services: PoVKitGenerator
- ROI Services: CustomerROITracker
📚 Documentation
- Production Features Guide - Complete guide to all production features
- Authentication Guide - Database-backed auth setup
- Billing Integration - Stripe/PayPal integration
- Quick Start Production - Fast setup guide
- Autonomous Services: CorporateBrain, OpportunityScanner, SkillConstructor, AgentConstructor, WorkflowEvolutionEngine
✅ Implemented Endpoints
- Commercial:
/commercial/provisioning/create,/commercial/provisioning/{plan_id}/execute,/commercial/production/health/{tenant_id} - PoV:
/pov/generate,/pov/{deployment_id}/deploy,/pov/{deployment_id},/pov/report/{tenant_id} - ROI:
/customer/{tenant_id}/roi,/customer/{tenant_id}/success_dashboard - CorporateBrain:
/autonomous/corporate-brain/start,/autonomous/corporate-brain/planning-loop,/autonomous/corporate-brain/status,/autonomous/corporate-brain/plan,/autonomous/corporate-brain/progress - Enterprise:
/enterprise/processes,/enterprise/memory/objects,/enterprise/kpi/summary,/enterprise/revenue/strategies,/enterprise/governance/execute - Monetization:
/monetization/api-keys,/monetization/payments,/monetization/revenue/summary,/monetization/marketplace/items - AGI:
/agi/agents,/agi/insight_chains,/agi/cognitive_loop/status,/agi/cognitive/kpis
⚠️ Planned (Endpoints Exist but Return "Planned" Status)
- ROI Experiments:
/customer/roi/experiments/trading/start,/customer/roi/experiments/support/start- These endpoints exist but return a "planned" status message
- Full implementation requires additional backend logic
📋 Not Yet Implemented
- Value Experiment Endpoints:
/value/trading/roi/experiment,/value/support/roi/experiment,/value/sales/roi/experiment - A/B Testing Endpoints:
/value/ab-test/start,/value/ab-test/{test_id}/results - Shadow Mode Endpoints:
/value/shadow-mode/start - PoV Playbook Execution:
/pov/playbooks/trading/execute,/pov/playbooks/support/execute- PoV generation and deployment work, but playbook execution requires additional implementation
🔧 Service Initialization
All services listed in "Fully Implemented" are initialized in aiosx/api/initialization.py and available at startup. Services are registered with their respective API endpoints and ready for use.
📝 API Documentation
All implemented endpoints are documented in the Swagger UI at:
- Local:
http://localhost:8000/docs - Production:
https://api.aiosx.com/docs
🚀 Next Steps for Full Production Readiness
- Implement Value Experiment Endpoints: Add full ROI experiment tracking and analysis
- Implement A/B Testing: Add A/B test framework for model and workflow comparison
- Implement Shadow Mode: Add shadow mode testing for safe rollout
- Implement PoV Playbooks: Add playbook execution engine for automated PoV flows
- Complete Commercial Service Registration: Add setter functions for all commercial services
- Complete Autonomous Service Registration: Add setter functions for all autonomous services (currently only CorporateBrain is registered)