Job Description

Hello There,

My name is Amit Kumar , and I serve as the Sr. Technical Recruiter at Kanak-IT INC. I am reaching out to share an excellent career opportunity for the role of " Kafka and Data Lake Engineer " with our esteemed client. If you are interested then please share your updated resume at Amitkumar@kanakits.com .

Title: Kafka and Data Lake Engineer

Location: 100% Remote

Position Type: Contract

Only on W2

Note: "Ideally this person would have a ServiceNow background"

Kafka and Data Lake Engineer

A Kafka and Data Lake Engineer is a data engineer who designs, builds, and manages data infrastructure using Apache Kafka for real-time data streaming and a data lake for storing large volumes of data. This role is vital for organizations that need to process and analyze both real-time streaming data and historical data to gain insights. For VA, this includes ServiceNow to the Data Lake via Kafka Bus.

Responsibilities

• Design data pipelines: Build robust, scalable, and secure data pipelines to ingest, process, and move data from various sources into the data lake using Kafka.
• Administer Kafka clusters: Deploy, configure, and maintain Kafka clusters and related ecosystem tools, such as Kafka Connect and Schema Registry, ensuring high availability and performance.
• Manage the data lake: Oversee the architecture and governance of the data lake, including managing data storage (e.g., in AWS S3 or ADLS), security, and metadata.
• Develop data processing applications: Create producers and consumers to interact with Kafka topics using programming languages like Python, Java, or Scala.
• Perform stream processing: Use tools like Kafka Streams, Apache Flink, or ksqlDB to perform real-time data transformations and analytics.
• Ensure data quality and security: Implement data quality checks, manage data lineage, and enforce security controls such as encryption, access controls (ACLs), and compliance (e.g., GDPR).
• Monitor and troubleshoot: Set up monitoring and alerting for Kafka and data lake infrastructure and respond to incidents to ensure operational reliability.
• Collaborate with teams: Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver reliable data solutions.
• Essential skills and qualifications
• Experience: Proven experience designing and managing data platforms with Apache Kafka and big data technologies.
• Programming: Strong proficiency in languages like Python, Java, or Scala.
• Big data technologies: Expertise in big data processing frameworks, such as Apache Spark and Apache Flink.
• Cloud platforms: Hands-on experience with cloud environments (AWS, Azure, or GCP) and relevant services like S3, Glue, or Azure Data Lake Storage.
• Data lake architecture: A solid understanding of data lake design principles, including storage formats (e.g., Delta Lake, Apache Iceberg), data modeling, and governance.
• Databases: Experience with various database systems, including both SQL and NoSQL.
• Infrastructure management: Familiarity with infrastructure-as-code tools like Terraform or Ansible and containerization with Docker and Kubernetes.
• Professionals in this field can advance from entry-level data engineering positions to senior roles, and then to a Big Data Architect or Solutions Architect, where they oversee large-scale data infrastructure.

Education : Bachelor's Degree - Degree in engineering, or a related scientific or technical discipline is required.

Substitution for Education - 10 years or 8 years of additional relevant experience may be substituted for education

Relevant certifications

• Pursuing certifications can validate your expertise and boost your career.
• For Kafka:
• Confluent Certified Administrator for Apache Kafka (CCAAK)
• Confluent Certified Developer for Apache Kafka (CCDAK)
• For Data Lake and Cloud:
• Databricks Certified Data Engineer
• AWS Certified Data Engineer
• Microsoft Certified: Azure Data Engineer Associate

Amit Kumar

Sr. Technical Recruiter

Email : Amitkumar@kanakits.com

LinkedIn :

Kanak IT is an equal opportunity employer. We consider all applicants for employment without regard to citizenship, immigration status, race, gender, disability, or any other protected category.

We respect your Online Privacy. This is not an unsolicited mail, If you are not interested in receiving our e-mails then please reply with a "REMOVE" in the subject to support@kanakelite.com and mention all the e-mail addresses to be removed with any e-mail addresses, which might be diverting the e-mail to you.

Job Tags

Similar Jobs