
title: "From PDF to Knowledge Graph"
description: "Discover how 33MED synthesizes unstructured medical literature into an actionable, queryable database using a lean Node.js, MongoDB, and React swarm engine."
date: 2023-10-24
author: "33MED Engineering"
category: "Panoptix Swarm & AI Orchestration"
From PDF to Knowledge Graph: Synthesizing Unstructured Medical Publications into Actionable Databases
The vanguard of computational immunology and surgical robotics isn't just limited by hardwareβit is bottlenecked by data. Every day, thousands of critical biomedical insights are locked away in unstructured PDFs, academic journals, and preprint servers. For 33MED to build genuinely autonomous biotech infrastructure, we need machines that can read, synthesize, and act on this data at scale.
Enter the next evolution of our ImmuNav prototype, powered by the Panoptix Swarm.
Our engineering mandate for Phase 1 was clear: strip out academic infrastructure bloat. No convoluted Python frameworks, no heavy RabbitMQ message brokers, and no multi-layered, fragile database ORMs. Instead, we have architected a highly performant, deterministic, and lean Node.js, MongoDB, and React ecosystem capable of transforming raw biomedical literature into a computable Knowledge Graph.
Here is the technical blueprint of how 33MED achieves real-time data ingestion, deterministic AI orchestration, and autonomous biological simulation.
ποΈ The Native Node/Mongo Swarm Engine
In distributed systems, complexity is the enemy of reliability. We eliminated dedicated message brokers in favor of utilizing MongoDB as a durable, atomic task queue. By designing a lightweight schema, we track agent states and pipeline jobs natively.
The Atomic Queue Schema
Our architecture relies on two foundational collections: agents and tasks. This design guarantees that every simulation parameterβwhether itβs a global directive to halt syndesmophyte growth or a specific molecular targetβis fully auditable.
// Collection: agents
{
_id: ObjectId("66654a..."),
role: "BioinformaticsOperator",
status: "active",
globalObjective: "Identify molecular targets to halt syndesmophyte growth and spine fusion.",
humanGuidanceOverride: "Focus explicitly on small-molecule inhibitors targeting IL-17 pathways.",
lastPulse: ISODate("2026-06-08T22:00:00Z")
}
// Collection: tasks (The Lightweight Queue)
{
_id: ObjectId("66654b..."),
agentId: ObjectId("66654a..."),
tool: "gromacs",
status: "queued",
payload: {
inputFiles: ["target_sequence.fasta"],
md_params: { nsteps: 50000, dt: 0.002, integrator: "md" } // Validated by Zod
},
humanSignOff: { isApproved: true, actionedAt: ISODate(...) },
logs: []
}
The Background Processing Loop
Within our Node.js Simulation Worker (Port 4000), a recursive polling loop queries MongoDB for actionable, human-approved jobs. It natively spawns child processes for heavy computational CLI tools, appending stdout/stderr logs directly back to the database document in real-time.
async function processQueue() {
// 1. Global Interrupter Check
const globalStatus = await db.collection('settings').findOne({ key: 'swarm_status' });
if (globalStatus?.isPaused) {
setTimeout(processQueue, 5000);
return;
}
// 2. Atomic fetch of the next approved task
const task = await db.collection('tasks').findOneAndUpdate(
{ status: 'queued', 'humanSignOff.isApproved': true },
{ $set: { status: 'processing', startedAt: new Date() } },
{ sort: { createdAt: 1 }, returnDocument: 'after' }
);
if (!task) return setTimeout(processQueue, 2000);
try {
// 3. Native binary execution via Child Process
const binaryProcess = child_process.spawn(getToolBinary(task.tool), task.payload.cliArgs);
binaryProcess.stderr.on('data', (data) => {
db.collection('tasks').updateOne({ _id: task._id }, { $push: { logs: data.toString() } });
});
binaryProcess.on('close', async (code) => {
const finalStatus = code === 0 ? 'completed' : 'failed';
await db.collection('tasks').updateOne({ _id: task._id }, { $set: { status: finalStatus } });
processQueue();
});
} catch (error) {
await db.collection('tasks').updateOne({ _id: task._id }, { $set: { status: 'failed', error: error.message } });
processQueue();
}
}
π‘ The Ingestion Engine: From PubMed to Gemini
Unstructured text is useless to a robotics or computational immunology pipeline. Our Data Ingestion Agents operate as automated background tasks, harvesting raw scientific papers and passing them through Googleβs Gemini SDK to extract deterministic JSON payloads.
- PubMed Sourcing: Utilizing the NCBI Entrez E-utilities with a registered API key, we fetch literature at 10 requests/second. Target queries are hyper-specific (e.g.,
Ankylosing Spondylitis AND syndesmophyte). - arXiv Sourcing: We query the native arXiv Atom/XML feed (
cat:q-bio.*) for immediate access to cutting-edge preprints without rate-limit constraints. - LLM Extraction Pipeline: Raw XML is injected into Gemini with strict structural instructions:
const systemPrompt = `You are an expert computational immunologist. Analyze the provided scientific text.
Extract explicit molecular targets, binding affinities, genetic expressions, or pathways related to Ankylosing Spondylitis.
Return the output strictly in the following JSON format:
{
"targetProtein": "string",
"associatedCytokines": ["string"],
"observedEffect": "string",
"confidenceScore": 0.00
}`;
const result = await geminiModel.generateContent([systemPrompt, rawPaperText]);
const structuredData = JSON.parse(result.response.text());
await db.collection('knowledge_base').insertOne(structuredData);
The result? A deeply interconnected, queryable Knowledge Graph of biological pathways ready for simulation.
𧬠Orchestrating the Core Tool Matrix
Building a commercial SaaS platform requires strict adherence to licensing. We bypass restrictive tools (like AlphaFold 3's non-commercial wrapper) and directly orchestrate open-source alternatives using Node's child_process.
- Chai-1 / Boltz-1 (Apache 2.0 / MIT): For multimodal molecular and structure prediction. We execute standalone CLI binaries, passing dynamically generated JSON/SMILES inputs.
- GROMACS (GNU LGPL): For molecular dynamics and binding thermodynamics. We monitor real-time thread progress via spawned
gmxcommands. - Bioconductor (Open Source): For genomic pathway mapping. We execute standalone R scripts via the
RscriptCLI. - Chaste (BSD License): For tissue-scale cellular calcification modeling. We fire compiled C++ binaries fed by agent-generated XML payloads.
- OpenSim CLI (Apache 2.0): For musculoskeletal joint and muscle load simulation, validating physical mechanical stresses programmatically.
ποΈ Human-In-The-Loop (HITL) Control Center
Autonomous systems must be governed by absolute executive control. The ImmuNav React Frontend (Port 5173) serves as a Swarm Management Dashboard, allowing bioengineers to oversee and intercede without touching configuration files.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β IMMUNAV AUTOMATION HUB [ SYSTEM STATUS: RUNNING ] β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β [ π GLOBAL SWARM PAUSE ] [ π RESUME SYSTEM ] [ βοΈ EDIT GLOBAL OBJECTIVE ] β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β ACTIVE AGENTS β
β π€ Agent_Ingestion_1 β Status: [ WORKING ] β Task: Scraping arXiv q-bio β
β π€ Agent_Operator_3 β Status: [ BLOCKED ] β Task: Awaiting Config Approvalβ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β π¨ AGENT CONFIGURATION APPROVAL GATEWAY β
β Task ID: #98322 β Tool Target: GROMACS Molecular Dynamics β
β Target Interaction: Small-molecule inhibitor binding to IL-17 Receptor β
β β
β [ View Agent-Suggested md_params Payload ] β
β { β
β "integrator": "md", β
β "nsteps": 50000, β
β "dt": 0.002 β
β } β
β β
β βοΈ [ Manually Edit JSON Parameters Inline ] β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β [ β REJECT CONFIGURATION ] [ β οΈ APPROVE & RUN JOB ] β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- The Global Interrupter: Toggling the Global Pause sends an HTTP request updating
settings.swarm_statusin MongoDB. The backend worker catches this and gracefully halts without corrupting active memory calculations. - Guidance Injector: Bioengineers can type high-level pivots (e.g., "Shift focus to IL-17A") directly into the UI. This is appended to the agentβs core prompt, dynamically steering the LLM's next deductive leap.
- Mandatory Approval Gates: When an AI agent designs a thermodynamics simulation, the task is marked
pending_review. The engineer verifies theZod-validated parameters, tweaks the JSON inline, and hits Approveβseamlessly passing the payload to the local hardware.
π§ͺ Micro-Dataset Testing Protocol: Iterating Safely
Computational physics is resource-heavy. To iterate fast on the swarm engine without exhausting APIs or melting local GPUs, 33MED enforces a strict Micro-Dataset Testing Protocol:
- Sourcing Sandboxes: We hardcode boundaries during development (
&retmax=3for PubMed;&max_results=3for arXiv). This ensures our regex sanitizers and Gemini extraction work flawlessly before opening the data floodgates. - Molecular Sandboxes: We do not test Chai-1 or GROMACS on complex human spinal proteins initially. We use Hen Egg White Lysozyme (PDB ID:
1AKI), a standard benchmarking structure. Energy minimization on Lysozyme takes less than two minutes, providing rapid validation that our Node runtime processes exits codes and stdout strings cleanly. - Mechanical Sandboxes: Instead of loading full biomechanical skeletal rigs, we utilize simple single-joint pendulums or 2D cellular grid templates found in the OpenSim and Chaste repositories to test our React dashboard's rendering of thermodynamic charts and XML generation.
By transforming PDFs into a queryable Knowledge Graph, and linking that graph directly to deterministic, open-source computational execution, 33MED is not just cataloging biomedical dataβwe are building the operating system for autonomous biotechnology.
Welcome to Phase 1.
