Wingman AI Python & Svelte, 2023

Wingman AI is probably the reason you know us. It is a powerful framework that allows you to use your voice to interact with various AI providers, process conversations, and trigger actions on your computer. Think of it as a bridge between your voice, AI language models, and computer control - all wrapped in customizable characters called “Wingmen” that serve as your interface to this world.

The beauty of Wingman AI is that you don’t need to be a developer to create and customize these characters. AI can be complex and intimidating, but Wingman AI aims to make it accessible to everyone.

Motivation

We built Wingman AI to solve several problems:

1. Accessibility: Making AI interactions more natural through voice
2. Immersion: Enhancing gaming experiences with context-aware AI assistants
3. Productivity: Controlling applications hands-free with natural language
4. Customization: Allowing non-technical users to build personalized AI assistants
5. Extensibility: Providing a framework for developers to create custom skills and integrations

What sets Wingman AI apart is that it doesn’t rely on exact phrases to trigger actions. The underlying AI models understand the context of your conversation and can interpret your intent naturally. Say “It’s raining! I can’t see a thing!” while playing a racing game, and your Wingman could trigger a command to activate the wipers - without you having to remember the exact command phrase.

Features

Wingman AI Core is Open Source and acts as a backend API with an impressive array of features:

• Flexible Voice Input: Push-to-talk or voice activation to capture your speech
• Multiple AI Providers: Support for OpenAI, Google Gemini, Azure, Groq, OpenRouter, and more
• Various Speech-to-Text Options: From cloud-based services to local models
• High-Quality Text-to-Speech: Multiple providers including OpenAI TTS, Azure TTS, Edge TTS, Elevenlabs, and local options
• Sound Effects: Enhancing responses with appropriate audio cues
• Multilingual Support: Works with any language supported by your chosen AI provider
• Intelligent Command Execution: AI-powered execution of keyboard and mouse commands
• Custom Skills Framework: Create capabilities that can do almost anything
• Directory-Based Configuration: No database needed, just simple file-based setup
• API and WebSocket Support: For integration with other applications

We also offer a client with an intuitive GUI that makes configuring and managing your Wingmen even easier.

Use Cases

The possibilities with Wingman AI are virtually endless:

• Gaming Enhancement: Role-play with AI while gaming for increased immersion. Have air traffic control in Star Citizen or Flight Simulator, or talk to game characters with their own voices.
• Information Access: Get live data like trade information, build guides, or wiki content read to you by a character voice you control.
• Automation: Execute complex keystrokes and macros in applications through natural conversation.
• Accessibility: Control your computer hands-free with voice commands interpreted by AI.
• Custom Assistants: Create specialized assistants for specific tasks or domains.

Technology

Wingman AI Core is built with modern technologies:

• Python: The entire backend is written in Python
• FastAPI & Pydantic: For the API layer and data validation
• WebSockets: For real-time communication between core and clients
• Multiple AI & Speech Services: Integration with various providers
• Directory/file-based Configuration: Using YAML for simple setup
• Skills Framework: Allows extending functionality with custom Python modules

The architecture follows a modular approach with clear separation between different components:

1. Audio Capture: Handles recording your voice
2. Speech-to-Text: Transcribes your speech to text
3. AI Processing: Sends text to AI models and processes responses
4. Command Execution: Triggers actions based on AI decisions
5. Text-to-Speech: Converts AI responses to spoken audio
6. Client Communication: Keeps connected clients updated about progress

For Developers

Developers can extend Wingman AI by:

• Creating custom Wingmen with specialized capabilities
• Building Skills that integrate with external services
• Implementing new providers for AI, STT, or TTS
• Extending the core framework with new features

The codebase is designed to be open and hackable, with well-documented base classes and plenty of extension points.

For Everyone Else

If you’re not a developer, you can still enjoy the full power of Wingman AI through our easy-to-use client. It allows you to:

• Configure every aspect of your Wingmen
• Record commands with keyboard and mouse actions
• Switch between different configurations for various use cases
• Manage provider settings and API keys
• Monitor system performance and usage

As usual, we opted for a Svelte/SvelteKit solution for the client, which is fast and responsive. The client communicates with the backend via WebSockets, providing real-time updates and feedback. It’s wrapped in Tauri so that users can install it on Windows as a native application.

Wingman Pro

Wingman Pro takes away the pain of setting up API keys and paying “per use” for AI services. It provides a subscription-based model that allows you to use the service without worrying about costs. It also made us a SaaS company, which is a new experience for us. We offer a unlimited/uncapped access to our models for a fixed price and you can use our 7-day trial to test it out.

Wingman Pro also unlocks the full potential of our client, allowing you to use all the features without any limitations and without ever touching a config file again.

Future Plans

We’re constantly improving Wingman AI with new features and capabilities. The community is also creating amazing extensions, Skills, and custom Wingmen that push the boundaries of what’s possible.

If you’re interested in Wingman AI, check out our website or join our Discord community to learn more and get involved!

Acknowledgements

Wingman AI stands on the shoulders of many open source projects and wouldn’t be possible without the contributions of our vibrant community. We’re particularly grateful to our moderators, contributors, and other supporters who help make this project better every day.