Together AI — Distributed AI Inference Guide

In this tutorial, you'll learn about Together AI. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Together AI provides a cloud platform for running open-source LLMs with distributed inference, offering low-latency access to models like Llama 3, Mixtral, DeepSeek, and Qwen through a single API.

What You'll Learn

• Setting up Together AI API keys and client libraries
• Running chat and text generation models with distributed inference
• Using Code Generation and specialized models
• Building with embeddings and function calling
• Fine-tuning open-source models on custom datasets
• Optimizing cost and latency for production workloads

Why Together AI Matters

Together AI differentiates itself through distributed inference — splitting model execution across multiple GPUs to reduce latency and cost compared to single-GPU inference. This architecture enables running 70B+ parameter models at speeds comparable to smaller models. DodaTech's Doda Browser uses Together AI for fast inline code suggestions, and DodaZIP leverages their embedding endpoints for document search indexing. This guide covers the complete Together AI platform for building production AI applications.

flowchart LR
    A["Together AI API Key\n& SDK Setup"] --> B["Chat Completion\nLlama 3, Mixtral"]
    A --> C["Text Generation\nCode & Analysis"]
    A --> D["Embeddings\nTogether Embeddings"]
    B --> E["Streaming &\nFunction Calling"]
    A --> F["Fine-Tuning\nCustom Models"]
    F --> G["Deployed\nCustom Endpoint"]
    B --> H["Low-Latency\nDistributed Inference"]
    style B fill:#dbeafe,stroke:#2563eb

Getting Started with Together AI

Together AI uses API keys for authentication. Sign up at together.ai, navigate to API Keys in your dashboard, and create a new key. The platform offers a free trial with $5 in credits.

from together import Together
import os

client = Together(api_key=os.environ["TOGETHER_API_KEY"])

Expected output: No output — the client initializes silently. If TOGETHER_API_KEY is missing, you'll see together.error.AuthenticationError: API key not found.

Together AI categorizes models by capability. Here's the landscape:

Chat Completions

The chat completions endpoint is the primary interface for LLM interaction. Together AI returns responses with model metadata and usage statistics.

messages = [
    {"role": "system", "content": "You are a cybersecurity expert."},
    {"role": "user", "content": "Explain the difference between XSS and CSRF attacks."}
]

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    messages=messages,
    max_tokens=500,
    temperature=0.7
)

print(response.choices[0].message.content)

Expected output:

**XSS (Cross-Site Scripting)** and **CSRF (Cross-Site Request Forgery)** are both web security vulnerabilities, but they target different aspects:

**XSS (Cross-Site Scripting):**
- Attacker injects malicious scripts into a trusted website
- Targets the user's browser directly
- Steals cookies, session tokens, or redirects users
- Example: A comment field that executes `<script>alert('xss')</script>`

**CSRF (Cross-Site Request Forgery):**
- Attacker tricks user into executing unwanted actions on a website where they're authenticated
- Targets web application state-changing requests
- Forces users to submit forms or click malicious links
- Example: A hidden `<img>` tag that triggers a fund transfer

**Key difference:** XSS exploits the user's trust in a website; CSRF exploits the website's trust in the user's browser. Durga Antivirus Pro uses both XSS and CSRF detection patterns in its web security scanning module.

The response includes model, usage, and timing metadata. Access response.usage.prompt_tokens and response.usage.completion_tokens for cost tracking.

Streaming Chat for Real-Time Applications

For chatbots and live code suggestions, streaming drastically improves user experience by showing tokens as they're generated.

stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Write a Python function to validate email addresses."}],
    stream=True,
    max_tokens=300
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Expected output:

import re

def validate_email(email):
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    return bool(re.match(pattern, email))

# Example usage
print(validate_email("user@example.com"))  # True
print(validate_email("invalid-email"))     # False

Streaming yields content deltas incrementally. This is the pattern Doda Browser uses for inline code suggestions that appear as you type, providing near-instant feedback without waiting for the full response.

Function Calling

Together AI supports tool use, allowing models to request structured function calls.

tools = [
    {
        "type": "function",
        "function": {
            "name": "scan_file_hash",
            "description": "Check a file hash against known malware database",
            "parameters": {
                "type": "object",
                "properties": {
                    "hash": {"type": "string", "description": "SHA-256 hash of the file"},
                    "threshold": {"type": "string", "enum": ["low", "medium", "high"]}
                },
                "required": ["hash"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Scan the file with hash a1b2c3d4 at high threshold."}],
    tools=tools,
    tool_choice="auto"
)

tool_call = response.choices[0].message.tool_calls[0]
print(f"Function: {tool_call.function.name}")
print(f"Arguments: {tool_call.function.arguments}")

Expected output:

Function: scan_file_hash
Arguments: {"hash": "a1b2c3d4", "threshold": "high"}

Durga Antivirus Pro uses function calling to let models query internal threat intelligence databases, scan results, and remediation steps — all through natural language conversation.

Code Generation Models

Together hosts specialized code models like DeepSeek Coder and CodeLlama. These models are optimized for programming tasks and often outperform general-purpose models on code benchmarks.

response = client.completions.create(
    model="deepseek-ai/deepseek-coder-33b-instruct",
    prompt="Create a Python class that handles AES-256 encryption and decryption "
           "with key derivation from a password. Include type hints and docstrings.",
    max_tokens=800,
    temperature=0.2
)

print(response.choices[0].text)

Expected output:

import os
import base64
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC

class AES256Encryptor:
    """Handles AES-256 encryption/decryption with password-based key derivation."""

    def __init__(self, password: str, salt: bytes | None = None):
        self.salt = salt or os.urandom(16)
        kdf = PBKDF2HMAC(
            algorithm=hashes.SHA256(),
            length=32,
            salt=self.salt,
            iterations=100000,
        )
        key = base64.urlsafe_b64encode(kdf.derive(password.encode()))
        self.cipher = Fernet(key)

    def encrypt(self, data: str) -> bytes:
        return self.cipher.encrypt(data.encode())

    def decrypt(self, token: bytes) -> str:
        return self.cipher.decrypt(token).decode()

The temperature=0.2 setting makes Code Generation more deterministic and reliable — perfect for production code where consistency matters.

Embeddings for Semantic Search

Together AI's embedding endpoint provides vector representations optimized for RAG and semantic search workflows.

response = client.embeddings.create(
    model="togethercomputer/m2-bert-80M-8k-retrieval",
    input=[
        "Doda Browser blocks third-party trackers automatically",
        "DodaZIP uses AES-256 encryption for archive protection",
        "Durga Antivirus Pro scans files using heuristic analysis]
    ]
)

for i, embedding in enumerate(response.data):
    print(f"Embedding {i}: dimension={len(embedding.embedding)}")
    print(f"First 3 values: {embedding.embedding[:3]}")

Expected output:

Embedding 0: dimension=768
First 3 values: [0.0456, -0.0234, 0.0789]
Embedding 1: dimension=768
First 3 values: [-0.0123, 0.0567, -0.0345]
Embedding 2: dimension=768
First 3 values: [0.0678, -0.0456, 0.0123]

These embeddings can be stored in vector databases for similarity search. DodaZIP uses Together embeddings to index archive contents for fast semantic file lookup across thousands of compressed documents.

Fine-Tuning on Together AI

Together AI supports fine-tuning of popular open-source models. You provide a dataset in JSONL format and Together handles the training infrastructure.

from together import Together

client = Together()
fine_tune = client.fine_tuning.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    training_file="https://storage.mysite.com/training_data.jsonl",
    n_epochs=3,
    learning_rate=1e-5,
    lr_scheduler="cosine",
    wandb_api_key=os.environ.get("WANDB_API_KEY")
)

print(f"Fine-tune job ID: {fine_tune.id}")
print(f"Status: {fine_tune.status}")

Expected output:

Fine-tune job ID: ft-abc123def456
Status: pending

Fine-tuning typically takes 1-4 hours depending on dataset size. The training file must be JSONL format with each line containing:

{"messages": [{"role": "user", "content": "What is DodaZIP?"}, {"role": "assistant", "content": "DodaZIP is a compression tool..."}]}

Durga Antivirus Pro uses fine-tuned models to recognize industry-specific security threats that general-purpose models would misclassify.

Performance Comparison: Together vs Other Providers

Together AI's distributed architecture splits 70B models across multiple GPUs, reducing per-token latency by up to 3x compared to single-GPU inference.

Common Errors

1. AuthenticationError: API key not found

The API key is missing, invalid, or expired. Verify TOGETHER_API_KEY is correctly set in your environment and hasn't been regenerated.

2. InvalidRequestError: Model not available

The requested model name is incorrect or the model isn't available in your region. Check available models using client.models.list().

3. RateLimitError: Too many requests

Together AI enforces rate limits (tier 1: 10 RPM, tier 2: 100 RPM). Upgrade your tier or implement request queuing with exponential backoff.

4. BadRequestError: Context length exceeded

Your prompt exceeds the model's maximum context window. Llama 3 70B supports up to 128K tokens. Trim your input or use a model with larger context.

5. InternalServerError: Service error

Temporary infrastructure issue on Together AI's side. Implement retry logic with tenacity and jitter for production robustness.

6. InsufficientQuotaError: Out of credits

Your account has exhausted its credits. Check usage at together.ai/account/billing and add funds.

7. InvalidModelError: Model doesn't support function calling

Not all models support tools/function calling. Llama 3 70B and Mixtral 8x22B support it; older models may not. Verify model capabilities before implementing tool use.

Practice Questions

1. What makes Together AI's distributed inference different from standard API inference?
2. How does streaming improve the user experience in chat applications?
3. What JSONL format is required for fine-tuning datasets?
4. Which Together models support function calling?
5. How can you check available models programmatically?

Answers:

1. Distributed inference splits model execution across multiple GPUs, reducing latency by up to 3x for large models like Llama 3 70B compared to single-GPU inference.
2. Streaming returns content tokens incrementally, enabling real-time display without waiting for full generation — critical for chatbots and code assistants.
3. Each line is a JSON object with a messages array: [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}].
4. Llama 3 70B, Mixtral 8x22B, and newer models. Older models like early Llama 2 variants do not support function calling.
5. Call client.models.list() to get all available models with their capabilities, pricing, and context Windows.

Challenge: Doda Browser needs a smart tab organizer that analyzes open tabs by content, suggests grouping by topic, and generates a summary of each group. Build a solution using Together AI's chat completion and function calling to automatically organize 10+ browser tabs.

Mini Project: AI-Powered Code Assistant

Build a CLI code assistant that writes, reviews, and explains code:

from together import Together
import os

client = Together(api_key=os.environ["TOGETHER_API_KEY"])
messages = [{"role": "system", "content": "You are an expert Python developer."}]

def code_assistant(user_input: str) -> str:
    messages.append({"role": "user", "content": user_input})
    response = client.chat.completions.create(
        model="deepseek-ai/deepseek-coder-33b-instruct",
        messages=messages,
        temperature=0.3,
        max_tokens=1000
    )
    reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

print("=== Code Assistant (type 'exit' to quit) ===")
while True:
    prompt = input("\n>> ")
    if prompt.lower() == "exit":
        break
    result = code_assistant(prompt)
    print(f"\n{result}")

Expected output:

=== Code Assistant (type 'exit' to quit) ===

>> Write a function that checks if a string is a palindrome ignoring case and spaces

def is_palindrome(s: str) -> bool:
    cleaned = ''.join(c.lower() for c in s if c.isalnum())
    return cleaned == cleaned[::-1]

# Tests
print(is_palindrome("A man, a plan, a canal: Panama"))  # True
print(is_palindrome("race a car"))  # False

Try it: Ask the assistant to review your existing code, explain complex algorithms, or generate unit tests. The session maintains context across turns. Extend it with file I/O to analyze entire projects by reading source files into the conversation context.

FAQ

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.