JARVIS (also published as HuggingGPT) is a research project from Microsoft Research that treats a large language model as a controller for coordinating a wide ecosystem of specialized AI models. The system was introduced in a 2023 paper titled 'HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace' (arXiv:2303.17580) and open-sourced on GitHub under the MIT license at github.com/microsoft/JARVIS. The system operates through a four-stage pipeline: task planning (the LLM decomposes a user request into structured subtasks using few-shot prompting), model selection (it queries Hugging Face Hub model descriptions to pick the best-fit specialist per subtask), task execution (subtasks are dispatched to the chosen models, either locally via Hugging Face Transformers, through Hugging Face Inference Endpoints, or via cloud APIs), and response generation (results from all subtasks are aggregated and returned as a coherent answer). This architecture allows a single natural-language prompt to trigger coordinated multi-modal workflows involving image generation, object detection, text translation, audio processing, and more. The project was significant in the field as an early demonstration that LLMs could act as an orchestration layer over the broader ML ecosystem, predating the widespread adoption of LangChain agents and AutoGPT-style loops. It accumulated a large number of GitHub stars and influenced subsequent agent frameworks including Microsoft's own Semantic Kernel. By mid-2023 the repository entered a declared 'rebuilding phase' and follow-on work shifted toward related tools such as EasyTool (released January 2024), which focuses on concise tool instruction for LLM-based agents. The Hugging Face hosted demo space (microsoft/HuggingGPT) was later shut down. As of 2025-2026 the core microsoft/JARVIS repository remains publicly available on GitHub but shows limited active maintenance, with most recent development activity concentrated in the adjacent EasyTool and TaskBench repositories rather than the core JARVIS codebase. Because JARVIS is open-source and self-hosted, there is no subscription or usage fee for the framework itself. Running it does require an OpenAI API key (for the LLM controller) and optionally a Hugging Face Pro subscription for GPU-backed inference endpoints, making the practical cost dependent on third-party API usage. Key features: - Four-stage pipeline: task planning, model selection, task execution, and response aggregation - Orchestrates 40+ specialized models from Hugging Face Hub via natural language - Three execution modes: local (Transformers), hybrid (HF Inference Endpoints), and lightweight (cloud APIs only) - Multi-modal task support: image, audio, video, text, and structured data tasks in a single prompt - LLM acts as a controller/planner, not a direct task executor - Open-source under MIT license on GitHub (github.com/microsoft/JARVIS) - Follow-on tooling: EasyTool for concise tool instruction (January 2024)
Free and open-source (MIT license). Requires OpenAI API key for the LLM controller (pay-per-use) and optionally Hugging Face Inference Endpoints (free tier available; GPU-backed tiers cost ~$3.15/hour for A10G). No separate JARVIS subscription fee.