Current LLMs can now reason, code, and generate with incredible fluency in isolated scenarios However, they don’t know your customer, your inventory, your workflows and your data. To realize the full potential of AI agents, they need access to relevant contextual data from diverse sources like personal files, knowledge repositories, and external tools. Furthermore, these agents must be capable of acting on that context, such as modifying documents or sending emails.

Previously, integrating AI models with these varied external resources was a complex and inefficient process. Developers often relied on bespoke code or specialized plugins tailored to each specific data source or API, resulting in fragile and unscalable integrations.

Anthropic introduced the Model Context Protocol (MCP), an open standard intended to connect AI assistants to a broader ecosystem of data and tools. Announced in November 2024, MCP has now gained significant traction and we are seeing a lot of softwares publishing their MCP servers.

What is MCP and how does it work?

MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools. - From Anthropic docs

MCP Servers are programs that expose data and capabilities to LLMs via the MCP protocol.

For example, a Slack MCP server will expose the following capabilities: Send a message to a channel, Search for messages in a channel, Get the list of channels, Get the list of users and more.

A Github MCP server will expose the following capabilities: Get the list of repositories, Get the list of issues, Get the list of pull requests, Create a pull request on a repository and more.

If an LLM is provided access to a Slack and Github MCP server, we can create agents very easily by prompting to the LLM:

Get open pull requests on my repository test-repo and send me a slack message with the list of pull requests.

If we provide Github and MCP servers to the MCP client, it will automatically be able to call the model, get the tool calls, execute the tool calls to finally acheive the goal of sending me the slack message with the list of pull requests.

You can also create your own MCP servers to expose your data and internal APIs as tools to LLMs.

Democratizing MCP Access and AI agentic workflows

The above concept of MCP servers presents a really powerful system wherein it becomes really easy for everyone inside an organization to build agents really fast and explore a wide range of automations. However, to enable a system like this, we need to solve the following problems:

Introducing the Truefoundry MCP Registry in AI Gateway

Truefoundry AI gateway comes with a MCP registry, centralized authentication and a MCP client built into the gateway that can orchestrate the agentic loop between the LLM and the MCP servers.

The key functionalities available are:

  1. Centralized MCP Registry: You can add public as well as your self hosted MCP servers which are registered in the Truefoundry Control Plane. The Control plane maintains the centralized registry of all the MCP servers and their authentication mechanisms. It handles user-specific OAuth2 flows, securely storing and refreshing access tokens and ensuring users can only access resources they are authorized for.

  2. Access Control: While registering a MCP server, you can specify the list of users/teams that have access to the MCP server.

  3. Unified Key to access all MCP servers: Any user can generate a single Personal Access Token (PAT) using which they can access all the models and MCP servers that they have access to. You can also generate a Virtual Account Token(VAT) to provide access to a specific set of MCP servers to an application.

  4. Agent Playground: Truefoundry AI gateway provides a playground where in users can play with prompts and different tools of MCP servers to build agents. Truefoundry comes with commonly used tools like Websearch, WebScraping, document extraction, code execution. The gateway comprises of a MCP client that orchestrates executing the tools decided by the LLM providers.

The Gateway also streames the progress of the request back to the UI so that the user can see the LLM responses, tool calls and the tool responses.

  1. Use MCP Servers in Code: The Gateway also shows the code snippets using which you can start using the MCP servers in your code.