EFH Innovation Sprint: Multimodal RAG System
EFH Innovation Sprint - Team Project
Developed in a 5-person team at the Ecosystems, Finance & Health (EFH) Innovation Sprint hackathon (March 5-7, 2025, Nairobi). The pipeline uses multimodal RAG (text, tables, and images) to make complex scientific PDFs on climate-biodiversity-health linkages queryable and understandable. Built with LangChain, LangGraph concepts, unstructured.io for PDF parsing, vision-language models for image summarisation, and Together AI LLMs/embeddings. Won "Best Overall Code" award and placed 6th overall due to late submission.
PythonLangChainTogether AIunstructuredLlama-3-VisionLlama-3-70BpandasTorch
Impact & Results
- Built a functional multimodal RAG pipeline in <3 days
- Won "Best Overall Code" in competitive Kenya AI hackathon
- Gained hands-on experience with modern RAG techniques, vision LLMs, and agentic patterns
- Explored multiple LLM providers and embedding models
Architecture
- PDF -> unstructured.io (hi-res) -> text/table/image extraction
- Image summarisation with Llama-3.2-Vision
- Text/table summarisation with Llama-3-70B
- Multi-vector retriever with parent/child documents and summaries
- In-memory vector store + byte store for fast prototyping
Challenges
- Processing large scientific PDFs with mixed content (tables, figures)
- Handling vision summarisation at scale during hackathon constraints
- Balancing retrieval accuracy vs. response quality under time pressure
- Late submission impacting final ranking despite strong code quality
Solutions
- Used unstructured[all-docs] hi-res strategy for robust extraction
- Parallel batch summarisation with concurrency limits
- Multi-vector retriever pattern for better precision
- Learned importance of early submission for future hackathons
Key Takeaways
Advanced RAG patterns: multi-vector, parent/child docs, image summarisation
Practical trade-offs when using vision-language models
Team collaboration under tight hackathon deadlines
Value of clean, well-structured code in competitive settings
Project Gallery
Browse through project illustrations
