Data architectures that hold in production
Lukas Rozado · Data Engineer · Azure/AWS · PostgreSQL · Python
I design and implement scalable data architectures and resilient pipelines. I work on high-performance ingestion, cloud integration and databases tuned for critical environments. One of my pipelines runs at >99.9% uptime in production.
Projects
Showing all projects
Enterprise Crypto Data Lake: Multi-Account Ingestion: Architecture for continuous financial auditing across dozens of institutional accounts on a crypto exchange, with full historical backfill from each account's inception. At its core, a rate limiter shared across processes reads the official consumed-quota header on every HTTP response, never an estimate from call counts, which eliminates the IP-ban pattern that existed before this architecture. A mandatory Bronze layer keeps the raw payload in JSONB and feeds a full Medallion model; loading runs through native PostgreSQL COPY, a temp table and UPSERT, sustaining tens of millions of records a year with credentials isolated in Azure Key Vault.
Unified Banking Ledger Platform: A platform that merged two separate financial pipelines, banking/PIX and ledger reconciliation, into a single system running every minute, persistent and orchestrated. Every data domain, accounts, PIX keys, PIX transactions, partners, ledger movements, OTC desk, gets its own producer-consumer queue, its own isolated cursor and watermark, and flushes in batches of 5,000 records straight to the database. Endpoints with no native delta filter use early-exit: they read from the top and stop on their own once N consecutive pages have already crossed the saved watermark, with no need to re-scan the entire history on every run. Partner balances run on two strategies, a 15-minute delta for whoever moved recently and a daily full refresh for everyone else. A separate lifecycle auditor fixes PIX transaction status that changes after capture, without re-scanning anything beyond what's needed.
High-Concurrency Financial Reconciliation Pipeline: A hybrid ingestion pipeline for high-transaction financial data: a near real-time API lane driven by a high-water mark, and a daily lane that downloads a CSV and anti-joins it against the database to recover gaps invisible to the API (pagination offset drift). Extraction runs on a producer-consumer architecture with a bounded queue, and checkpoint state lives per page, not per day, resuming from the exact point of interruption after any failure. Historical backfill runs in a strictly sequential procedural process, isolated from the production ingestion.
OTC Desk Reconciliation Pipeline: Daily ETL that ingests a human-maintained operational spreadsheet (no stable primary key, with rows reordered, deleted and recreated at the source) into a relational data warehouse, resolving inconsistently typed counterparty names via fuzzy matching with a 92% similarity threshold. Instead of incremental upsert, the load runs a Full Replace (truncate + full reload) on every execution. Deduplication runs on a SHA-256 hash over the transaction's signature plus an occurrence counter (nonce), deliberately preserving identical legitimate splits instead of collapsing them. A schema-shielding layer extracts strictly the expected financial columns and ignores any new column the business side adds without notice.
Fiscal Document Consolidation Pipeline: A monthly incremental pipeline that downloads XML+PDF pairs of fiscal documents via a fiscal document provider's API, resolves the owning entity for each document by substring-matching the tax identifier inside the raw decoded content, never trusting an explicit reference tag, and consolidates output into one combined ZIP plus one per-entity ZIP, organized by document type and date.
Unified Multi-Chain Ledger Integrator: A resilient pipeline that consolidates transactions across dozens of blockchains and contracts (EVM and Solana), reading straight from the node via raw RPC (eth_getLogs filtered by event topic) instead of relying on a third-party indexer. Block ranges too dense for the API split recursively until they fit. The core differentiator is the self-healing reconciliation engine: it compares on-chain event counts against the database range by range, heals any mismatch on its own by deleting and re-extracting the batch, and closes with a final accounting check comparing total on-chain supply against the net supply computed in PostgreSQL.
Institutional Digital Asset Custody Integrator: An ingestion orchestrator for institutional crypto asset custody, with each regional workspace isolated, its own credential loaded from a vault, and two parallel lanes: transactions and per-vault balance snapshots. A state machine switches on its own between full backfill and incremental mode, with a 5-minute overlap at the handoff so no record is lost at the window's edge. A separate, lighter auditor re-scans only the transactions still in a non-terminal status, starting from the oldest pending record, and upserts whatever changed, with no need to re-scan the entire history to catch a transaction that took a while to settle.
BaaS & Financial Settlement Data Sink: A programmatic integration pipeline that centralizes multi-bank financial settlement data (Banking-as-a-Service), designed for maximum security between isolated systems, with corporate authentication via Azure Active Directory and tokens. Before any native PostgreSQL COPY, every load passes through a dedicated sanitization layer; a flattening engine turns nested logs into audit columns queryable directly, with no join; and the historical backfill script runs isolated from the production checkpoint, with no risk of reprocessing or corrupting the incremental in progress.
Predictive Pipeline in Competitive and ML Engine: End-to-end algorithmic engine orchestrated for quantitative trading. The architecture includes massive daily web scraping, rigorous normalization, and a Feature Engine optimized to feed the 'Model Factory' with 42 structured variables. The system autonomously manages retraining (backtesting) and live inference, sending high-precision signals for hedge strategies and dashboards.
LCK Spring 2024 Players Statistics: Public, fully verifiable dataset covering the complete LCK (League of Legends Champions Korea) Spring 2024 split: players, champions, and matches organized into three relational tables. Collected via web scraping with BeautifulSoup and Selenium from Gol.gg, with rigorous metric normalization, ready for EDA, visualization, and predictive modeling. Published on Kaggle with a maximum usability score.