Can you process large texts with local LLMs?

I recently spend time benchmarking multiple local models on the premise of how good they are at finding data in large context. Imagine you have confidential documents and you’d like to get information out of them. Local AI is a great solution to this. Instead of sharing the documents with a cloud AI you simple feed it to your local AI and give instructions.

But which one to choose from? There’re multiple factors playing into this:

• Size of the model. Does it fit your ram?
• Moe or dense Model? Moe models are much faster but less smart
• Do you need additional world knowledge?
• How well does it adhere to your prompt?
• How much does it hallucinate?
• How good is it long context understanding?

The task

I invented a task to measure models on. I gave it a very large Swift file (50k tokens) and asked it a question that requires it to understand what the code does, find specific snippets, and adhere to a guardrail (don’t propose changes, just give me snippets from the code):

WHICH CODE SNIPPETS FROM THIS CODE WOULD I NEED TO UNDERSTAND HOW TO ADD A PREFIX IMAGE BEFORE TITLE. DON’T CHANGE ANY CODE, JUST GIVE ME THE SNIPPETS I NEED.

I ran this against multiple models locally.

Hardware

M4 Macbook Pro Max with 128 GB Memory running Sequoia 15.7.3 and LM Studio 0.4.5

Included models

I tried to include models from smaller to larger sizes, including very large models which only run on 128GB of ram because they’re heavily quantized or dumbed down via REAP.

Performance

Evaluation