Big Data Engineer/Python Developer

Person needs to be a python developer and be familiar with Big Data and map/Reduce technologies. PySpark background would be a plus.
This resource will be taking data from 3rd parties and ingesting and aggregating data to levels in which the business requires.
5+ years in python Worked with a data lake 5+ years with Bash/ksh/sh Scripting experience with Spark (pyspark)
Explored potentially using Luigi (a powerful open library used to build pipelines) Experience with a python based automated runbook to aid with testing the data pipeline
Experience with using python with bash scripting to munge/wrangle log data and generate excel formatted reports

Leave a Reply