Data Engineer
Posted 2 days ago
Position: Data Engineer
Location: NYC
Exp: 6+ Years
Employment Type: W2 & 1099
If you are interested, kindly share your updated resume at
Responsibilities:
- Design, build, and optimize robust, scalable pipelines in Databricks (PySpark, SQL, Delta Lake) for structured, semi-structured, and unstructured data.
- Ingest and process data from diverse sources: relational databases, APIs, PDFs, Excel, flat files, and web scraping.
- Implement data quality frameworks, validation checks, and automated tests to ensure reliability across the pipeline lifecycle.
- Conduct UAT/UAD processes to align outputs with business requirements and ensure trust in data.
- Own pipelines end-to-end—from development through deployment, monitoring, scaling, and continuous improvement.
- Apply best practices for dataset versioning, reproducibility, and lineage (Delta Lake, MLflow, or equivalent)
- Optimize pipelines for large-scale performance and cost-efficiency in Databricks and AWS environments.
- Collaborate with data science and analytics teams to deliver model-ready datasets and ensure schema stability.
- Produce clear technical documentation (design docs, lineage diagrams, data dictionaries) and communicate effectively with technical and business stakeholders
Required Qualifications
- 6+ years of professional experience in Data Engineering or related roles.
- Hands-on expertise in Databricks (PySpark, SQL, Delta Lake).
- Proven track record of building and deploying end-to-end production pipelines at scale.
- Experience working with structured and unstructured data, including extracting information from PDFs, Excel, web scraping, and APIs.
- Strong foundation in data quality, testing, and observability (e.g., Great Expectations, Deequ, dbt tests, or similar).
- Ability to troubleshoot, debug, and optimize Spark/Databricks workloads for scale and performance.
About eGrove:
eGrove Systems is a highly profiled IT solution provider to a vast nation wide clientele. Our solutions include e-commerce customization, mobile app development, digital marketing, and website optimization. We are always looking for the right talents to join our team. You can visit our website (https://www.egrovesys.com) for more details about our company.
An Equal Opportunity/Affirmative Action Employer/Minority/Female/ADA/Veterans Employer.
Flexible work from home options available.
About this listing
No resume yet? Fix that first
Build a free, recruiter-ready resume in minutes, then tailor it to this job in one tap.
Create a free resume