Big Data Engineer - Full Time - Richmond, VA
Kaynes Technology Inc
University Of Richmond, Virginia, United States
Job type: fulltime
Job industry: I.T. & Communications
POSITION : Big Data Engineer with AWS LOCATION : Philadelphia, PA JOB TYPE : Full time/Permanent EXPERIENCE LEVEL : 4-8 Years
Primary skills are Hadoop, Python but experience with AWS is preferred
Building data pipelines, writing ETL logic in PySpark, or Scala.
Ingestion of data into AWS from trafficking systems, on-premise databases (Teradata, MS SQL
Server, HP Vertica etc.), on-premise Hadoop cluster etc.
Spinning up appropriate EMR, EC2 instances for the job
Scheduling the jobs and automating the jobs
Creating the data sets in the appropriate format, i.e. Parquet
Ingesting the data into S3/Redshift or any data ware house for external team to consume into their