019790TS Big Data Engineer in Durham, NC (W2 ONLY, local preferred)
Durham, North Carolina, United States
Job type: fulltime
Job industry: I.T. & Communications
- You will be helping a team build data pipelines to feed our next generation cloud data & analytics platform
- You will leverage your technical knowledge and experience to help design data ingestion and data preparation patterns which are reusable within and across teams
- Daily you will be working a variety of tasks (analysis, development, DevOps) with a variety of technologies (Python, Airflow, AWS and many more)
- You will be collaborating with squad resources in an agile framework to ensure work is planned and completed in the sprint cycle
- Bachelor s degree or similar in a technology related field (e.g. Engineering, Computer Science, etc.) required.
- 1+ years of hands-on experience developing and solutions in AWS, and hands-on knowledge of EC2, EBS, EKS, S3, IAM, CloudFormation, KMS, Route 53, RDS and other related AWS services.
- 2+ years of advanced hands-on experience in implementing batch and near real-time Big Data integration frameworks and/or applications using various technologies (Hadoop, Spark, Impala, Sqoop, etc.)
- Advanced experience with Python and SQL is required.
- Experience building DAGs with Apache Airflow is a plus.
- 3+ years of hands-on experience developing large-volume applications with various types of database systems: relational (Oracle, Postgres, MySQL, MSSQL) required, and knowledge of Snowflake is a plus.
- Experience with Linux environments, ability to interface with the OS using system tools, scripting languages, integration frameworks, etc.
- Experience with DevOps, Continuous Integration and Continuous Delivery (Maven, Jenkins, Stash, Docker).
- Experience and comfort executing projects in Agile environments.
- Excellent verbal and written communication skills.
- provided by Dice