Senior Data Engineer (Pyspark, AWS)

About Blue Ridge

Blue Ridge is a market-leading SaaS company specializing in AI-driven supply chain planning. We help global retailers and distributors predict demand, optimize inventory, and reduce waste through powerful machine learning and automation. Our platform delivers accurate forecasting, smart replenishment, and seamless ERP integration—backed by a customer-first model that has earned us “Best Relationship” recognition on G2.

Learn more about the company and what we do: http://blueridgeglobal.tilda.ws

Why Join Us?

We’re in the middle of a major tech transformation- rebuilding our platform for scale, speed, and real-time decision-making. This role is a key part of that future.

We’re modernizing from a legacy .NET/C#/SQL stack to a cloud-native big data architecture. The result? Up to 20x performance gains, real-time analytics, and expanded market potential in dynamic sectors like grocery and FMCG.

About The Role

You’ll work with our Data Platform Architect, engineers in India and Uzbekistan, and report directly to the CTO (PST) and VP of Engineering (EST). Two more engineers will join later this year.

You’ll build and scale data pipelines using PySpark and AWS - powering core analytics and machine learning services across our platform.

Your mission: optimize big data pipelines and build scalable, production-ready ETL/ELT workflows using PySpark on AWS/Databricks.

Responsibilities

1. Migrate legacy data workflows to a distributed Spark-based stack
2. Build high-performance pipelines using PySpark and Delta Lake
3. Optimize Spark jobs for cost and compute efficiency (partitioning, caching, tuning)
4. Support both real-time streaming and batch processing
5. Collaborate with ML engineers, backend devs, and data scientists
6. Monitor performance, enforce best practices, and mentor junior engineers

Tech & Infrastructure

1. Spark, PySpark, Databricks
2. AWS: EMR, S3, Lambda, IAM, VPC
3. Orchestration: Airflow (MWAA)
4. Data Lake: Delta/Parquet (Bronze/Silver/Gold architecture)
5. Testing: PyTest, Robot Framework
6. Future plans: Kafka, CDC, multi-cloud (GCP, Azure)

What We’re Looking For

1. 5+ years in data engineering; strong big data background
2. 7+ years of hands-on AWS experience is a must, including deep familiarity with EMR, IAM, VPC, EKS, ALB, and Lambda.
3. Expert-level PySpark and Python skills
4. Experience with Delta Lake, Airflow, streaming tech (Kafka a plus)
5. Strong SQL and data modeling skills
6. Good communicator, fluent in English (B2+)
7. Experience with distributed teams across time zones
8. Bonus if you’re familiar with .NET systems or have cloud/data certifications (AWS, Databricks, Spark, etc.)

Recruitment Process

# 1 Initial Interview: Up to 1 hour with HR or/and including a self-assessment form (Click to fill out the form). If you prefer, you can skip the call and discuss all questions and details in writing instead. Just let us know!

# 2 Managerial Interview (Optional): 30-60 minutes (You will meet with the CTO to learn more about the company, the position, and future plans directly from the source.)

# 3 Test Assignment: up to 113 minutes on iMocha platform (Data Structures - Graph data structure, Array and String manipulation - All in Python, with a few MCQ questions on Spark)

# 4 Technical Interview: Platform/Application Architect: up to 1h. The technical interview at Blue Ridge is a structured, 60–90 minute discussion, designed to evaluate both theoretical knowledge and hands-on experience in data engineering.

# 5 Offer & Paperwork: Up to 30 minutes with the CTO to finalize conditions and complete necessary paperwork.

Got Questions or Interested? Let’s Connect!