Senior Data Engineer

Railroad19, Inc is seeking a Senior Data Engineer with deep, hands‑on experience in building modern lakehouse architectures on GCP. This role focuses on designing, developing in Python & Spark, and delivering reusable data‑sharing adapters that connect BigQuery‑backed data products to Snowflake and Databricks using Iceberg UniForm and Delta Sharing.

About Railroad19:

Core Responsibilities:

• Design and implement the UniForm write layer (Delta + Iceberg dual metadata).
• Build GCS → BigQuery ingestion pipelines for structured operational datasets.
• Develop and implement in Python and Spark.
• Implement Kafka-based CDC patterns for real-time and near-real-time ingestion.
• Develop data lineage, dependency tracking, and modular adapter code.
• Configure Snowflake Horizon external tables for zero-copy reads.
• All data hub tables are to be written once using Delta Lake with Iceberg UniForm enabled… readable by all target consumers without conversion.
• Implement and certify Delta Sharing endpoints for Databricks consumers.
• Build governed access layer components: RBAC, connector registry entries, tenant-scoped authorization.
• Align semantic layer models with LookML and KPI catalog definitions.
• Collaborate with cross‑functional teams to deliver end‑to‑end features.
• Troubleshoot issues across the full stack and contribute to code quality.

Skills/Experience:

• 6+ years of proven enterprise-level experience in Python & Spark
• Advanced experience in GCP BigQuery
• Strong working knowledge of Apache Iceberg, Delta Lake, Iceberg UniForm
• Experience with Delta Sharing; Kafka / CDC pipelines
• Specific work experience in Snowflake Horizon Catalog; Databricks Unity Catalog
• Solid experience with Data lake architecture & ingestion pipeline design
• Excellent Communication skills and the ability to work cohesively with multiple teams.

Preferred Experience – Nice to Have

• Prior delivery in enterprise SaaS, media, or advertising technology.
• Active daily use of AI-assisted development tools (Claude Code preferred).