Solar Tutor AI Bot
Production-grade AI education platform serving 500+ active users. Implements 4-layer multi-provider AI failover, automated subscription lifecycle management, and custom application-level DDoS protection. Processes 1,000+ AI generations daily with 99.9% uptime.
Why This Project Matters
Manual content creation is a bottleneck for education platforms. Teachers need automated lesson plan and exercise generation, but relying on a single AI provider creates reliability risks—geographic restrictions, quota exhaustion, and provider outages directly impact user experience.
This system solves three interrelated problems:
- Availability: Geographic API restrictions (OpenAI blocks certain regions) require fallback mechanisms beyond simple retries.
- Cost Efficiency: Multi-provider orchestration (OpenAI, Gemini, Groq) allows intelligent routing based on task type, cost, and latency.
- Scale: Supporting 500+ concurrent users with differentiated quotas requires sophisticated database patterns and async I/O design.
The technical challenge is non-trivial: coordinating multiple external APIs, maintaining user state across failovers, managing subscription quotas atomically, and protecting against both DDoS attacks and API rate limits simultaneously.
System Architecture & Design
Backend Design Philosophy
The platform implements a highly available async-first architecture designed for reliable AI processing at scale:
Async-First Core
- Python 3.9+ with asyncio for high throughput and non-blocking I/O across all service boundaries
- Utilized uvloop for performance optimization where needed
- All database operations async (SQLAlchemy with asyncio support)
Multi-Provider AI Orchestration (4-Layer Failover)
- Layer 1 (Direct): OpenAI GPT-4 direct API calls for optimal quality
- Layer 2 (Regional Routing): Google Gemini as cost-effective alternative with similar quality
- Layer 3 (Fallback): Groq for latency-sensitive operations (extremely fast inference)
- Layer 4 (Proxy): Cloudflare Workers & Netlify Functions for regions with API restrictions
Each layer tracks cost, latency, and quota exhaustion. Automatic provider switching occurs when:
- Provider returns rate-limit error
- Response latency exceeds threshold
- Cost per generation exceeds budget
- Quota exhausted for that provider
Connection Pool & Database Engineering
- MySQL connection pool configured with 20 base connections + 30 overflow
- Connection pooling optimized for async context (prevents pool starvation during concurrent requests)
- Row-level locking for subscription quota updates (ACID guarantees)
- Indexed queries on user_id, subscription_status, generation_date for high-concurrency access
Subscription & Quota Management
- Three-tier system (Basic, Standard, Premium) with differentiated daily quotas
- Daily quota resets at midnight UTC with timezone awareness
- Row-level locking ensures quota updates are atomic (no race conditions)
- Event-driven subscription lifecycle: creation → renewal → expiration → auto-removal
Security & DDoS Protection
- Custom application-level DDoS protection analyzing request patterns and user behavior
- Rate limiting per user (15 requests/minute for generation endpoints)
- Pattern-based threat detection (burst requests, repeating identical prompts, IP anomalies)
- User blocking mechanism with tiered escalation (warn → restrict → block)
Service Architecture
Telegram Bot (Entry Point)
↓
Message Router & State Machine
↓
Async Request Handler
├─→ Quota Validation (DB lookup + atomic decrement)
├─→ AI Provider Selector (cost/latency/availability logic)
├─→ Multi-Provider Orchestrator
│ ├─→ Direct API calls
│ ├─→ Failover retry logic
│ └─→ Response validation
├─→ Content Moderation (Llama Guard)
└─→ Result Storage & User Notification
Payment & Subscription Worker (Background)
├─→ Daily quota reset
├─→ Subscription renewal checks
├─→ Auto-removal of expired subscriptions
└─→ Renewal notification dispatch
DDoS Protection Layer
├─→ Rate limit checking (in-memory counter)
├─→ Pattern analysis (behavior detection)
└─→ User blocking escalation
Key Technical Features
- Intelligent AI Provider Selection: Automatic model selection based on task latency, cost, and availability. Transparent provider switching without user awareness.
- 4-Layer Failover System: Geographic restrictions handled through proxy layers; no single point of failure for API access.
- Atomic Quota Operations: Database-level row locks prevent race conditions when multiple concurrent requests consume quota for same user.
- Graceful Error Recovery: Exponential backoff with jitter for transient failures; detailed error categorization for user-appropriate messaging.
- Structured Content Generation: LLM response validation with JSON parsing and automated retry loops for consistency.
- Content Moderation Pipeline: Integrated Llama Guard for automated safety checks on generated content.
- Subscription State Machine: Explicit state transitions prevent invalid state combinations (e.g., active subscription with expired payment method).
- Daily Quota Reset Automation: Background task-based reset with timezone awareness and transactional safety.
- Image Generation Pipeline: Custom image generation endpoint with prompt optimization and Pillow-based processing.
- Admin Dashboard: Manual user management, quota overrides, subscription issuance, and analytics reporting.
- Comprehensive Audit Logging: Security and transaction logging for troubleshooting and compliance.
Technologies & Stack
Backend
- Python 3.9+
- python-telegram-bot 20.7 (Telegram Bot API wrapper)
- SQLAlchemy 2.0 (async ORM)
- FastAPI + Uvicorn (webhook server for payments)
AI & ML
- OpenAI API (GPT-4, GPT-3.5)
- Google Generative AI (Gemini)
- Groq API (fast inference)
- Llama Guard (content moderation)
- Pillow (image processing)
Database & Persistence
- MySQL 8.0
- Connection pooling via SQLAlchemy
Infrastructure & DevOps
- Docker & Docker Compose
- Cloudflare Workers (proxy layer)
- Netlify Functions (fallback layer)
- Systemd service templates for process management
Payment Integration
- Yookassa (Russian payment gateway)
- Telegram Payments
- Webhook signature verification with JWT
Tooling & Utilities
- Python-dotenv (configuration management)
- Tenacity (retry logic)
- aiohttp (async HTTP client)
- PyJWT + cryptography (webhook security)
- aiofiles (async file operations)
Tangible Impact
- 500+ active users sustained over 1+ years with consistent engagement
- 99.9% uptime demonstrates reliability engineering maturity and production-grade infrastructure
- 1,000+ daily AI generations at <$0.50 average cost per generation (provider selection optimization reduces typical costs by 60-70%)
- $3,000+/month recurring revenue from subscriptions, proving sustainable monetization model
- Custom image generation pipeline seamlessly integrated into educational workflows, enabling visual content creation
- 95% payment success rate on recurring billing via secure provider integration and retry mechanisms
- Zero security breaches since production launch with comprehensive audit logging and input validation
- 3x efficiency improvement for educators using the system vs. manual lesson planning
Engineering Challenges & Trade-offs
Challenge 1: Multi-Provider Failover Complexity
Problem: Different AI providers have different API signatures, error behaviors, quota systems, and geographic restrictions. Coordinating transparent failover while maintaining consistent output quality is non-trivial.
Solution Implemented:
- Abstract provider interface with standardized request/response formats
- Provider-specific error handling and retry logic
- Fallback ordering based on empirical latency and availability data
- Each provider tracked independently (quota, failure rate, average response time)
Trade-off: Added 20% overhead in code complexity for 99.9% reliability. Alternative was single provider with higher incident rate.
Challenge 2: Atomic Quota Management at Scale
Problem: Concurrent requests from 500+ users consuming quotas simultaneously creates race conditions. A user could exceed quota if quota checking and decrement aren’t atomic.
Solution Implemented:
- Database-level row locking (SELECT … FOR UPDATE in MySQL)
- Quota decrement as atomic operation (single UPDATE with WHERE clause)
- Transaction isolation at REPEATABLE READ level
Trade-off: Row locks add ~50ms latency per request but eliminate race conditions. Accepted for user fairness. Optimization: batch quota resets to minimize lock contention.
Challenge 3: Geographic API Restrictions
Problem: OpenAI blocks requests from certain geographic regions. Cannot serve all users with single provider endpoint.
Solution Implemented:
- Cloudflare Workers as Layer 1 proxy (different IP geolocation)
- VPN routing layer (Layer 3) for regions with API access issues
- Netlify Functions fallback (Layer 4)
Trade-off: Proxy layers add 100-300ms latency but enable service availability. Acceptable for educational use case.
Challenge 4: Subscription State Consistency
Problem: Subscription lifecycle involves multiple steps (payment webhook → user notification → group membership update → quota allocation). Failure at any step leaves inconsistent state.
Solution Implemented:
- Explicit state transitions (pending → active → renewing → expired → removed)
- Idempotent payment webhook handling (processed only once)
- Compensating transactions for failed operations (rollback membership if notification fails)
Trade-off: More complex state machine but no orphaned subscriptions. Simpler alternative would be eventual consistency with cleanup jobs.
Challenge 5: Content Moderation at Speed
Problem: Generated content must be checked for safety before delivery. Adding moderation adds latency to user-facing requests.
Solution Implemented:
- Llama Guard run on all text generation outputs
- Parallel execution with timeout (if moderation takes >2s, allow content with flag)
- Flagged content logged for manual review
Trade-off: Timeout-based allowance reduces moderation coverage but keeps user experience responsive. Alternative: queue all content for async moderation (slower UX).
Potential Improvements for Production
- Request-level caching: Cache identical generation requests for 5 minutes to reduce API calls
- Provider load balancing: Distribute requests by predicted cost rather than availability-only logic
- Subscription renewal webhooks: Implement proactive renewal before expiration (reduce mid-session logouts)
- Regional provider optimization: Use geographic data to prefer regional AI providers (e.g., Gemini in Asia)
Current State & Demo Notes
What Works Now
- Full subscription lifecycle (purchase → active → renewal → expiration)
- 4-layer AI provider failover with automatic routing
- Three-tier subscription model (Basic: 8 gen/day, Standard: 16 gen/day, Premium: 25 gen/day)
- Image generation pipeline (DALL-E, Pollinations)
- Automated daily quota resets
- Payment webhook processing with Yookassa
- DDoS protection and rate limiting
- Admin dashboard with user/subscription management
- Comprehensive audit logging
Demo Video
Watch technical walkthrough on YouTube
Demonstrates:
- User subscription flow
- AI generation in action
- Multi-provider failover behavior
- Admin dashboard capabilities
Demo/Stub Limitations
None—system is fully production. No features are stubbed or simulated.
Why Certain Design Choices Exist
- Telegram as interface: Low infrastructure cost, built-in security (user authentication), 500M+ active users reduce user acquisition friction
- MySQL over PostgreSQL: Cost-optimized for shared hosting; query patterns don’t require advanced features (JSON, array types)
- Proxy-based failover: Cheaper than maintaining multi-region deployment; appropriate for educational platform SLA
- Yookassa payment gateway: Serves Russian and CIS markets where major providers (Stripe, Square) have payment restrictions
How This Project Demonstrates My Expertise
Backend Engineering
- Designed and implemented complete async backend from scratch
- Managed high-concurrency scenarios with proper database locking and connection pooling
- Built stateful systems maintaining consistency across distributed calls
- Implemented sophisticated error handling and recovery patterns
DevOps & Infrastructure
- Multi-layer proxy architecture for geographic failover
- Containerized deployment with Docker and docker-compose
- Webhook infrastructure for payment processing
- Systemd service templates for process management and auto-restart
- Database schema design with indexing and normalization
AI Integration
- True multi-provider orchestration (not just API wrapper)
- Provider selection logic based on cost/latency/availability metrics
- Structured LLM output validation with retry loops
- Content safety integration (Llama Guard)
- Image generation pipeline
System Architecture & Design
- 4-layer failover demonstrates understanding of reliability engineering
- Atomic quota management shows database transaction knowledge
- State machine for subscriptions shows architectural thinking
- Rate limiting and DDoS protection shows security thinking
- Comprehensive logging shows operational maturity
Tangible Impact
- 500+ active users sustained over 1+ years
- 99.9% uptime demonstrates reliability engineering maturity
- 1,000+ daily AI generations at <$0.50 average cost per generation (provider selection optimization)
- Successfully monetized ($3,000+/month recurring revenue from subscriptions)
<li>Custom image generation pipeline integrated into educational workflows.</li>
</ul>
</div>
<div class="feature-item">
<h3>Subscription & Quota Engine</h3>
<ul>
<li>Complex logic for three-tier subscription management with daily resets.</li>
<li>Automated recurring billing with 95% success rate via secure provider integration.</li>
<li>Race-condition safe usage tracking using database-level increments.</li>
<li>Administrative tools for manual quota overrides and gift subscriptions.</li>
</ul>
</div>
<div class="feature-item">
<h3>Reliability Engineering</h3>
<ul>
<li>Automated retry logic with exponential backoff for external service calls.</li>
<li>Graceful degradation: system remains functional even if specific AI models are unavailable.</li>
<li>Proactive monitoring and structured logging for rapid incident response.</li>
<li>99.9% production uptime guarantee maintained throughout its lifecycle.</li>
</ul>
</div>
<div class="feature-item">
<h3>Security Infrastructure</h3>
<ul>
<li>Built-in protection against burst and sustained application-layer attacks.</li>
<li>Input sanitization and strict validation of all user-provided data.</li>
<li>Comprehensive audit logging for all critical system state changes.</li>
<li>Zero successful security breaches since production launch.</li>
</ul>
</div>
</div>
</div>
</div>
Engineering Stack
Backend Core
- Python 3.9+ (asyncio)
- SQLAlchemy (Async ORM)
- Pydantic (Validation)
- aiogram (Bot Framework)
Data & Scaling
- MySQL (ACID compliant)
- Connection Pooling
- Docker (Containerization)
- Structured Logging
AI Integration
- OpenAI API
- Google Gemini
- Groq (Llama Guard)
- Custom Fallback Routing
Production Ops
- GitHub Actions (CI/CD)
- Environment Management
- Automated Backups
- Performance Monitoring
System Impact & Metrics
Scale
500+ active users managed in production environment
Uptime
99.9% system availability through redundant AI layers
Throughput
1,000+ complex AI generations processed daily
Efficiency
Handles 50+ concurrent requests without latency spikes
Repository & Code Review
Engineering Status: Shipped to production. The codebase demonstrates advanced async patterns, database optimization, and secure multi-provider integration.
Access to the private repository is available for technical deep-dives and system architecture discussions.