Blog

Technical writing on LLM safety, AI agents, context engineering, transformers, deep learning, tool calling, and applied machine learning by Masih Moafi.

  • Codex and Elpis Under the Hood

    One prompt traced through Codex CLI's Rust source, from the submission queue to the sandbox to compaction, with the exact places where Elpis hooks in. The code is quoted, not just cited.

    • Loop: steering, streaming, parallel tools.
    • Safety: policy, approvals, the Linux sandbox.
    • Elpis: Smart Prune, the Context Ledger, work graphs.
  • Aider Under the Hood

    How Aider works and how it differs from a tool-calling agent like Codex: a model with no tools, edits parsed from text, and a repo map ranked by PageRank. The code is quoted, not just cited.

    • Loop: bounded reflections, not tool calls.
    • Edits: SEARCH/REPLACE parsed from the reply.
    • Context: a repo map ranked by PageRank.
  • Neural Language Models Before Transformers

    From bigram counts and Bengio's MLP to recurrent state, gated memory, and causal CNNs.

    • Baselines: bigram and MLP.
    • Memory: RNN, LSTM, and GRU.
    • Progression: causal CNNs.
  • Transformers From Scratch in Code

    From tensor shapes and self-attention to a tiny GPT, RoPE, KV caching, and FlashAttention.

    • Shapes: tensors and causal attention.
    • Implementation: a tiny GPT.
    • Speedups: KV cache, FlashAttention, and RoPE.
  • Did fine-tuning help a small language model classify banking requests?

    A measured Qwen3.5-0.8B LoRA experiment on Banking77, with a separate official test and an explicit limit on what unseen means.

    • Task: 77 banking intent labels.
    • Validation: 85.3% after fine-tuning.
    • Official test: 84.1% on 3,076 messages.
  • RoPE

    Mechanics-first explanation of Rotary Position Embeddings, 2D rotations, relative position, and long-context attention in Transformers.

    • Geometry: two-dimensional rotations.
    • Dot products: relative positions.
    • Implementation: vectorized in PyTorch.
  • Using-Pydantic-For-Structured-Outputs

    Pydantic models for validated structured outputs, schema-constrained LLM responses, and typed tool calling in Python.

    • Schemas: typed responses.
    • Validation: errors and handling.
    • Evaluation: a tool-calling suite.
  • AMR-Parsing-Summarization: A Graph-to-Text Framework

    A PyTorch prototype that turns sentence meaning graphs into abstractive summaries.

    • Encoding: AMR graphs.
    • Generation: graph to text.
    • Workflow: training and inference.
  • Cross-Modal Knowledge-Guided Summarization

    BERT summarization guided by multimodal knowledge graphs in a paper-reproduction prototype.

    • Graphs: multimodal knowledge.
    • Summarization: BERT-guided.
    • Evaluation: ROUGE and ablations.
  • Refusal Direction — Paper Critique

    The one-dimensional refusal result is real. I just think the interpretation can be narrower than the title sounds.

    • Features: refusal sits on a linear direction.
    • Interventions: edit at the representation level.
    • Limits: scope and causality claims.
  • Role Confusion — Paper Critique

    Prompt injection may be a source-authentication problem inside the model, not just bad instruction following.

    • Framing: injection as source authentication.
    • Probes: chain-of-thought forgery.
    • Gaps: prompt-injection evaluation.
  • Alignment Faking — Paper Critique

    A real existence proof for training-aware strategic behaviour, but not evidence that current models generally hide secret goals.

    • Setup: training-aware compliance.
    • Evidence: behaviour versus scratchpad.
    • Design: follow-up evaluations.
  • AutoDAN — Paper Critique

    Readable automated jailbreaks are the real contribution; the stealth and evaluation claims need narrower wording.

    • Attacks: hierarchical genetic search.
    • Prompts: readable jailbreaks.
    • Limits: transfer and baselines.