Carbon AI: Connect external data to LLMs

Imagine you’re building AI agents with access to user data and have started looking for ways to improve responses beyond passing the user’s context into the prompt à la prompt engineer. You begin to experiment with RAG (Retrieval Augmented Generation) and start to chunk and vectorize some PDFs, then load them into your vector database. You’re excited about the results as the LLM responses start to look more contextually accurate.

But you run into an issue. Most user data you want to access is stuck in various file formats and within 3rd-party integrations like Notion and Google Drive. Having users download and convert all their content into PDFs introduces too much friction. You realize the need now to set up an entire ingestion pipeline, managing everything from OAuth flows to custom parsers and scheduled syncs. This isn’t what you signed up for – you wanted to build a GenAI app, not an ETL pipeline.

This is where Carbon comes in. Carbon handles the heavy lifting on the ingestion side and makes building RAG into your GenAI app simple. They built a high availability, high throughput, and low latency retrieval pipeline for you to leverage out of the box.

Below is a sample flow that shows how their customers, like AskAI, TypingMind, and Jenni.ai, integrate Carbon with the rest of their GenAI stack. Carbon works with whatever vector store you already use, or you can rely on their in-house highly-performant vector database (<15 ms latency).

Implement the JavaScript SDK

First, you need a frontend for users to connect their content. Carbon handles this flow entirely. The SDK enables you to build specific flows that fit your custom use case.

They have pre-built components for users to upload local files (text, audio, and image), submit URLs for web scraping, and sync from 3rd-party apps. For their connectors with 3rd-party apps, Carbon manages the flow for users to authenticate their accounts and manage their documents.

Set up webhooks

You can subscribe via webhooks to be notified when a file is initially synced and after each scheduled re-sync. If you use their pre-built React component, they also have callback functions available for more immediate feedback.

Retrieve content

There are many ways to retrieve user content from Carbon. If you already have a vector database like Pinecone set up, you can directly retrieve and store the chunks and vectors for a particular file once Carbon notifies you that the file is processed. They have built-in chunking that is fully customizable, but you also have direct access to the plain text and raw files as well.

Carbon provides built-in hybrid search functionality to run both keyword and semantic searches and return the most relevant results. Hybrid search is available to any Carbon customer using their in-house vector database hosted on Qdrant. From our testing, it seems Carbon can match the performance of Pinecone’s p1 pods but at a fraction of the cost.

Next Steps

If Carbon sounds interested and you’d like a free 14-day trial courtesy of aitools.fyi, please book a time here to get onboarded!

Their team is always constantly adding new 3rd party connectors and embedding models to their ecosystem. If there’s a certain connector or model you’re looking for, let their founder know at derek@carbon.ai.