Candidate
Jobs / Data in New York / eGrove Systems Corporation

Data Engineer

Posted 2 days ago

eGrove Systems Corporation · New York
Apply on eGrove Systems Corporation's site ↗You'll apply directly on the company's site. We keep this job in your tracker so you never lose the thread.

Position: Data Engineer

Location: NYC

Exp: 6+ Years

Employment Type: W2 & 1099

If you are interested, kindly share your updated resume at

Responsibilities:

  • Design, build, and optimize robust, scalable pipelines in Databricks (PySpark, SQL, Delta Lake) for structured, semi-structured, and unstructured data.
  • Ingest and process data from diverse sources: relational databases, APIs, PDFs, Excel, flat files, and web scraping.
  • Implement data quality frameworks, validation checks, and automated tests to ensure reliability across the pipeline lifecycle.
  • Conduct UAT/UAD processes to align outputs with business requirements and ensure trust in data.
  • Own pipelines end-to-end—from development through deployment, monitoring, scaling, and continuous improvement.
  • Apply best practices for dataset versioning, reproducibility, and lineage (Delta Lake, MLflow, or equivalent)
  • Optimize pipelines for large-scale performance and cost-efficiency in Databricks and AWS environments.
  • Collaborate with data science and analytics teams to deliver model-ready datasets and ensure schema stability.
  • Produce clear technical documentation (design docs, lineage diagrams, data dictionaries) and communicate effectively with technical and business stakeholders

Required Qualifications

  • 6+ years of professional experience in Data Engineering or related roles.
  • Hands-on expertise in Databricks (PySpark, SQL, Delta Lake).
  • Proven track record of building and deploying end-to-end production pipelines at scale.
  • Experience working with structured and unstructured data, including extracting information from PDFs, Excel, web scraping, and APIs.
  • Strong foundation in data quality, testing, and observability (e.g., Great Expectations, Deequ, dbt tests, or similar).
  • Ability to troubleshoot, debug, and optimize Spark/Databricks workloads for scale and performance.

About eGrove:

eGrove Systems is a highly profiled IT solution provider to a vast nation wide clientele. Our solutions include e-commerce customization, mobile app development, digital marketing, and website optimization. We are always looking for the right talents to join our team. You can visit our website (https://www.egrovesys.com) for more details about our company.

An Equal Opportunity/Affirmative Action Employer/Minority/Female/ADA/Veterans Employer.

Flexible work from home options available.