Big Data Engineer

We are looking for an experienced Big Data Engineer for our Big data team to develop and maintain the corporate Data Lake. We are seeking a professional capable of effectively supporting the building of machine learning models and data science across various business domains.  In addition to your Big Data duties, we would be happy to provide support and education to get Tableau Certified!

What you'll actually do

• Develop Data Platform services for Data-Driven Decision-Making, such as:

  • Implement and operate Data Lake (S3 MinIO + Spark).

  • Maintain Spark on k8s.

  • Maintain and enrich an environment for the creation of data assets such as (ETL / ELT pipelines, DDL, etc.) via the development of data transformation tools
    

• Active participation in the development of the data platform environment:

  • Creating and monitoring compliance with data security policies, code policies, and naming conventions.

  • Participate in Data Platform architecture development.

  • Recommend ways to improve data reliability, efficiency, and quality.

Who we’re looking for

• Strong knowledge of Spark architecture; deployment and maintaining Spark on Hadoop, k8s or standalone experience.

• Experience with relational databases like Vertica, PostgreSQL, ClickHouse and similar.

• Experience with Big Data Processing & Data Lake building.

• Experience building ETLs / ELTs to take data from various data sources.

• Python/Scala skills associated with data engineering activities.

• Expert structured query language (SQL) knowledge.

• Understanding of base principles CI/CD in the data processing area.

It will also be a plus:

• Practice knowledge of Apache Spark on batch and real-time data processing.

• Awareness of building stream-processing systems using solutions such as Flink/spark-streaming.