Big Data Engineer

Position: Big Data Engineer

Location: Preference for all candidates to work from our Columbia, MD office location. Hybrid approach: 2-3 days in the office; 2-3 days WFH;

Duration: Long Term

Responsibilities

• Extract and integrate data from multiple data systems and organize data in a format that can be easily read by human or machine.
• Transition of legacy ETLs with Java and Hive queries to Spark ETLs.
• Design and develop data processing solutions and custom ETL pipelines for varied data formats like parquet and Avro.
• Design, develop, test and release ETL mappings, mapplets, workflows using Streamsets, Java MapReduce, Spark and SQL.
• Performance tuning of end-to-end ETL integration processes.
• Analyze and recommend optimal approach for obtaining data from diverse source systems.
• Work closely with the data architects and interface with business stakeholders to understand requirements and offer solutions.

Preferred Qualifications

• BS in Information Technology, Computer Science, Software Engineering or related field.
• Understanding of distributed computing principles and hands-on experience in Big Data Analytics and development.
• Good knowledge of Hadoop and Spark ecosystems including HDFS, Hive, Spark, Yarn, MapReduce and Sqoop.
• 4+ years of Big Data development experience.
• 3+ years of working experience in ETL development and functional programming knowledge preferably with Scala, Spark, Java, Python, R.
• 3+ years of experience tuning Spark/Java coding, SQL and No SQL.
• AWS development using big data technologies preferred.
• AWS cloud-certified, Databricks, and Snowflake experience a plus.

All your information will be kept confidential according to EEO guidelines.