300k/s Zero-Copy Streaming, 10-Node Oracle Exadata RAC Deep Tuning & AI-Assisted / Agentic Data Development Factory
Operating across Landline, Broadband, and Mobile (GSM) domains serving 50M+ subscribers on
an asymmetric 10-node Oracle Exadata RAC cluster (8x X9M-2 + 2x X10M mega-nodes, 17 storage
cells, 896 physical cores / 1,792 hardware threads, ~11 TB RAM, 94.48% Smart Scan filtering across 403.8 PB,
25B-26B rows/day across 2,120+ concurrent sessions), the enterprise faced 14.5-hour EL load phases, nightly
batch SLA overruns, 512 caged hardware threads on X10M expansion nodes, 11,635 minutes of
sort extent pool latch serialization, ~18.6 TB daily broadcast temp spills, 169.3 hours of
enq: HV lockouts, and 36.5 billion daily blind updates.
Principal Architecture & Engineering Delivered
- PyArrow C++ Zero-Copy Streaming Ingestion: Engineered a real-time CDR ingestion
engine using PyArrow C++ native zero-copy decompression, Celery 5.6+, Redis 8.x, Linux
inotify, and a live FastAPI SSE dashboard, processing 200,000 compressed archives daily (~20 TB compressed, >20.5B records) at up to ~300,000 records/sec under an 80 MB RAM ceiling (cutting EL windows from 14.5h to 3.5h at 1/10th CPU/RAM cost). - 10-Node Exadata RAC Uncaging & DBRM v5.0 Architecture: Uncaged +512
trapped hardware threads (+40% cluster CPU capacity) on X10M nodes, calibrated proportional
PGA memory (350 GB / 650 GB preventing >4 PB disk spills) and
parallel_force_local = TRUEBI logon triggers, and deployed the 2-window DBRM v5.0 resource plan (DWH_ETL_NIGHT_PLAN&DWH_DSS_DAY_PLANanchored to UTC+3) - collapsingresmgr:cpu quantumwaits by 99.9% (down to 1.12m), stopping 14,567-slave parallel query leaks, slashing nightly batch turnaround by 5.0 hours/day, and recovering 975.9 CPU hours/month (-22.3% compute savings). - Non-Invasive CBO SQL Patching, Storage & HyperLogLog Stats: Deployed
non-invasive
DBMS_SQLDIAGSQL Patches & Profiles eliminating ~18.6 TB/day ofPX SEND BROADCASTtemp spills without touching legacy ETL code; refactored 2.85B-row outerDISTINCTqueries from 130.5 hours DB time (798 GB temp spill) to <2 minutes; eliminated 36.5B daily blind updates; eradicated 100% ofenq: SScontention (105.9h to 0.00m) and 98.4% of sort latch waits via 32 MB / 64 MB uniform temp extents; rebuilt 14 high-concurrency staging tables with 16-bucket Hash Partitioning to eradicate 99.6% ofenq: HVlockouts (169.3h to 0h) and reclaim 89.3% NVMe storage; engineered a HyperLogLog incremental bulk statistics engine across 68,000+ partitions; and tuned Linux HugePages (132 GB), TCP buffers (134 MB), and JDBC fetch/batch arrays (140x fewer roundtrips), accelerating core ETL pipelines by 89.6% to 98.0%. - AI-Assisted / Agentic Data Development Factory: We developed specialized AI
Skills and MCP solutions for Data Engineering, Data Vault 2.1 / TM Forum SID Architecture,
Governance development/audit (Apache Ranger & Atlas), and DB Performance Tuning, paired with a
programmatic
dbt-to-ODI 12c SDK bridge and our "Loop Engineering" Agentic Developer OS.
enq: SS & 99.6% enq: HV
eradicated