LLM Setup Guide

Choose your LLM backend: Local models (complete control, privacy) or AWS Bedrock (enterprise Claude with Zero Data Retention).

Option 1: Local LLM Backends

All local backends provide complete data privacy - your code never leaves your infrastructure. Choose based on your preferences:

• LM Studio - Easy GUI, great for getting started
• Ollama - Simple CLI, Docker-friendly
• llama.cpp - Lightweight, maximum control
• vLLM - High-performance serving, best for production

LM Studio Setup

Prerequisites

Setup Steps

1. Install LM Studio

Download from lmstudio.ai and install for your platform (macOS, Windows, Linux).

2. Download Model

1. Open LM Studio
2. Navigate to "Discover" tab
3. Search for "qwen3-30b-a3b"
4. Download Q4_K_M quantization (recommended for performance/quality balance)

3. Start Local Server

1. Click "Local Server" tab
2. Select your downloaded model
3. Configure server settings:
   • Port: 1234 (default)
   • Context Length: 20000 tokens
   • Max Tokens: 8000
4. Click "Start Server"

You should see "Server running on http://localhost:1234" when successful.

4. Configure drep

Update your config.yaml:

llm:
  enabled: true
  endpoint: http://localhost:1234/v1
  model: qwen3-30b-a3b
  temperature: 0.2
  max_tokens: 8000

  # Rate limiting
  max_concurrent_global: 5
  requests_per_minute: 60
  max_tokens_per_minute: 100000

  # Cache configuration
  cache:
    enabled: true
    directory: ~/.cache/drep/llm
    ttl_days: 30
    max_size_gb: 10.0

  # Circuit breaker (optional)
  circuit_breaker_threshold: 5
  circuit_breaker_timeout: 60

5. Verify Setup

Test your configuration:

drep scan owner/repo --show-metrics

Expected output:

• Code quality findings
• Missing docstring suggestions
• Metrics showing token usage
• Cache hit rate > 0% on second scan

Remote LM Studio

For remote LM Studio instances (e.g., a dedicated server):

llm:
  endpoint: https://lmstudio.example.com/v1
  api_key: ${LM_STUDIO_KEY}  # If authentication enabled

Set the API key as an environment variable:

export LM_STUDIO_KEY=your-api-key-here

Model Recommendations

Choose a model based on your available RAM and performance requirements:

Recommended: Qwen3-30B-A3B provides the best balance of quality and performance for code review tasks.

Ollama Setup

Ollama provides simple CLI-based model management with Docker-friendly deployment:

1. Install Ollama from ollama.ai
2. Pull a model: ollama pull qwen2.5-coder:32b
3. Update config.yaml:

llm:
  endpoint: http://localhost:11434/v1  # Ollama OpenAI-compatible API
  model: qwen2.5-coder:32b

llama.cpp Setup

llama.cpp provides lightweight, low-level control with minimal dependencies:

1. Build llama.cpp with server support
2. Start server: ./server -m model.gguf --port 8080
3. Update config.yaml:

llm:
  endpoint: http://localhost:8080/v1
  model: your-model-name

Option 2: AWS Bedrock (Enterprise Claude with ZDR)

AWS Bedrock provides enterprise-grade Claude 4.5 models with Zero Data Retention (ZDR). Your data stays in your AWS infrastructure, is never used for model training, and meets strict compliance requirements.

Prerequisites

• AWS account with Bedrock access
• AWS CLI configured or credentials in ~/.aws/credentials
• Model access enabled in AWS Bedrock console

Setup Steps

1. Enable Model Access

1. Go to AWS Bedrock Console (region: us-east-1 or your preferred region)
2. Navigate to Model Access in the left sidebar
3. Click Modify model access
4. Select Anthropic Claude models:
   • Claude Sonnet 4.5
   • Claude Haiku 4.5
5. Click Save changes
6. Wait for access to be granted (usually instant)

2. Configure AWS Credentials

Bedrock uses the standard AWS credentials chain. Choose one method:

Method A: AWS CLI Configuration

aws configure
# Enter your AWS Access Key ID
# Enter your AWS Secret Access Key
# Default region: us-east-1
# Default output format: json

Method B: Environment Variables

export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key
export AWS_DEFAULT_REGION=us-east-1

Method C: Credentials File

Create ~/.aws/credentials:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
region = us-east-1

3. Configure drep

Update config.yaml to use Bedrock:

llm:
  enabled: true
  provider: bedrock  # Required for Bedrock

  bedrock:
    region: us-east-1
    model: anthropic.claude-sonnet-4-5-20250929-v1:0

  # General LLM settings
  temperature: 0.2
  max_tokens: 4000

  # Rate limiting (lower for Bedrock)
  max_concurrent_global: 3
  requests_per_minute: 30

  # Cache configuration
  cache:
    enabled: true
    ttl_days: 30

4. Verify Setup

Test your Bedrock configuration:

drep scan owner/repo --show-metrics

Expected output:

• Successful LLM initialization: "LLM backend: AWS Bedrock"
• Code quality findings
• Metrics showing token usage
• No authentication errors

Supported Bedrock Models

Bedrock Regions

Bedrock Troubleshooting

AccessDeniedException

• Verify model access is enabled in AWS Bedrock console
• Check IAM permissions include bedrock:InvokeModel
• Ensure you're in the correct AWS region

ThrottlingException

• Reduce max_concurrent_global (try 2-3)
• Lower requests_per_minute (try 20-30)
• Consider using Haiku 4.5 for faster throughput

Invalid model ID

• Verify the model ID matches exactly (case-sensitive)
• Check model availability in your region
• Use correct format: anthropic.claude-sonnet-4-5-20250929-v1:0

Credentials not found

• Run aws configure to set up credentials
• Check ~/.aws/credentials file exists
• Verify AWS_ACCESS_KEY_ID environment variable is set

Troubleshooting (Local Backends)

Connection Refused

If drep cannot connect to the LLM server:

• Verify LM Studio is running
• Check endpoint URL matches (default: http://localhost:1234)
• Test connection: curl http://localhost:1234/v1/models

Circuit Breaker is OPEN

If you see "Circuit breaker is OPEN" errors:

• Wait for recovery timeout (default 60 seconds)
• Check LM Studio logs for errors
• Verify the model is loaded and ready
• Increase circuit_breaker_timeout in config.yaml

Cache Not Working

If cache hit rate stays at 0%:

• Verify cache.enabled: true in config.yaml
• Check cache directory exists and is writable
• Ensure commit SHA is stable (don't scan uncommitted changes)
• Run drep metrics --detailed to see cache statistics

Slow Performance

To improve performance:

• Enable caching (should give 80%+ cache hit rate on re-scans)
• Increase max_concurrent_global (but watch RAM usage)
• Use a smaller model (Mistral-7B) for faster responses
• Reduce context_length in LM Studio

Next Steps

• Return to Quick Start to run your first scan
• Explore advanced configuration options
• View examples of CI/CD integration and webhooks