Overview
This project implements the Cross-modal Knowledge guided Model (CKGM) for abstractive text summarization. Traditional sequence-to-sequence neural summarization models suffer from a fundamental failure mode: generating factually inconsistent, hallucinated, or fabricated information when condensing long source texts.
CKGM directly tackles factual drift by constructing structured multimodal knowledge graphs (MKG) that ground entity relationships across both source articles and associated visual imagery before language generation begins.
Problem Formulation & Motivation
When neural summarization models (such as vanilla BART, T5, or BERT-based encoders) summarize complex articles, they frequently encounter entity confusion—attributing actions to incorrect subjects or dropping essential constraints. In multimodal news articles, accompanying images and captions contain rich factual anchors that can verify entity relationships.
CKGM addresses this by:
Entity Extraction: Parsing named entities and relational triplets from the text.
Visual Feature Grounding: Aligning visual bounding boxes and scene features with identified entities.
Graph-Guided Encoding: Infusing structured relational embeddings into the transformer self-attention layers so generated tokens adhere to verified knowledge graph edges.
Key Innovations
Multimodal Knowledge Graph Construction: Dynamically fuses factual triplets extracted from source text with region-of-interest (RoI) features from accompanying images.
Cross-Modal Fact Verification: Links textual entity nodes with visual representations to validate entity interactions and prevent hallucinations during generation.
Knowledge-Guided BERT Fusion: Injects structured graph embeddings directly into pre-trained BERT attention mechanisms, ensuring output fluency while enforcing factual correctness.
Entity Memory Layer: Employs an efficient external entity memory algorithm for scalable graph traversal, faster training throughput, and robust multi-entity fusion.
Architecture Components
Component Architecture & Role Function TextEncoder BERT Transformer Backbone Tokenizes, processes, and encodes contextual text representations ImageProcessor Faster R-CNN / ResNet Feature Extractor Extracts visual region embeddings from paired article images MKGConstructor Dynamic Graph Neural Layer Builds unified multimodal knowledge graph connecting text entities and visual objects CKGMBert Knowledge-Injected Decoder Decodes abstractive summaries conditioned on graph entity memory and attention states
Experimental Evaluation & ROUGE Performance
Extensive experiments on standard multimodal benchmarks—including the CNN/DailyMail news corpus and the MSCOCO captioning/summarization dataset—demonstrate that CKGM significantly improves both factual consistency and linguistic fluency.
Key empirical findings:
Higher ROUGE-1 / ROUGE-2 / ROUGE-L: Achieving consistent gains over text-only summarization baselines through structured entity grounding.
Substantial Hallucination Reduction: FactCC and human evaluation metrics showed a marked drop in fabricated entity relationships compared to ungrounded BERT and Seq2Seq baselines.
Technical Stack & Dependencies
Deep Learning Framework: PyTorch & PyTorch Geometric (torchgeometric)
Language Models: Hugging Face Transformers (bert-base-uncased)
Computer Vision: Torchvision ResNet & Object Detection feature backbones
Datasets: CNN/DailyMail, MSCOCO
Links & Source Code
GitHub Repository • Documentation • Original Research Paper (Springer)