AES - Delivery - Digital Test Automation Engineer

You will own quality for data pipelines, microservices, APIs, GUIs, and LLM/ML integrations running on Databricks and Microsoft Azure. The role focuses on test strategy, test case design and execution, automation, ticket management, and agentic QA for LLM workflows. Minimum experience: 5 years

Key ResponsibilitiesTest Strategy and Planning Define and maintain QA strategy for data, API, GUI, microservices, and LLM/ML features; prioritize test coverage based on risk and business impact.Test Case Design and Execution Create clear, reusable test cases for functional, integration, regression, performance, and pricing scenarios; execute manual and automated tests and maintain test artifacts.Data and Database Testing Validate ETL/ELT pipelines on Databricks and Delta Lake; write and run T‑SQL validation queries; verify data lineage, completeness, and correctness.API Testing Design and run contract, integration, and negative tests for RESTful APIs; use Postman, OpenAPI/Swagger, and automated API test suites.GUI Testing Create and execute UI test plans for web applications; perform exploratory testing and maintain automated UI tests using Selenium, Playwright, or equivalent.Microservices Validation Test service contracts, message flows, resilience, and backward compatibility across containerized microservices; validate observability and error handling.LLM and ML QA Design evaluation metrics and test harnesses for ML models and LLM integrations; validate inference correctness, hallucination rates, latency, and cost tradeoffs.Prompt Engineering and Agentic QA Create minimalistic, cost‑aware prompts and prompt templates; implement agentic QA processes for autonomous agents including safety checks, guardrails, and evaluation pipelines.Pricing and Business Logic Testing Validate pricing engines, discount rules, billing flows, and edge cases; create scenario matrices and automated checks for pricing correctness.Automation and CI/CD Integrate tests into CI/CD pipelines; author automated test suites, test runners, and validation scripts; collaborate with SRE/DevOps on test environments.Ticketing and Incident Management Raise, triage, and manage tickets in Azure DevOps or Jira; document reproduction steps, root cause analysis, and remediation.Documentation and Collaboration Produce test plans, runbooks, release checklists, and postmortems; mentor junior QA engineers and participate in cross‑functional reviews.Required Skills and ExperienceExperience: Minimum 5 years in QA, software testing, or related roles.Databricks and Data Testing: Hands‑on experience validating Databricks pipelines, Delta Lake, and Spark outputs.Microsoft SQL Server: Medium proficiency in T‑SQL for validation queries and database testing.Azure: Practical experience with Azure Data Factory, ADLS/Blob Storage, Azure DevOps, and Azure Functions.API Testing: Strong experience with REST APIs, Postman, OpenAPI/Swagger, and automated API testing.GUI Testing: Experience with Selenium, Playwright, or equivalent UI automation tools and exploratory testing.Microservices: Familiarity testing containerized services, message queues, and service orchestration.LLM and ML Fundamentals: Understanding of ML lifecycle, model evaluation, prompt engineering basics, and LLM failure modes.Prompt Engineering: Ability to craft minimalistic prompts and maintain prompt libraries for reproducible tests.Agentic QA Awareness: Knowledge of agentic QA concepts, safety checks, and evaluation metrics for autonomous agents.Testing Tools and Automation: Experience with test frameworks, CI/CD integration, and test data management.Ticketing Tools: Experience using Azure DevOps, Jira, or similar.Soft

Skills

Strong analytical thinking, clear written communication, and cross‑team collaboration.

Nice to HaveCertifications: ISTQB, Azure Data Engineer, or Databricks certifications.Programming: Python for test automation and data validation; familiarity with Scala or Java.Observability: Experience with Application Insights, Datadog, Prometheus, or Grafana.Performance Testing: Experience with JMeter, k6, or similar tools.LLM Ops: Experience with prompt versioning, cost monitoring, and model evaluation frameworks.Infrastructure as Code: Familiarity with Terraform or ARM templates for test environment provisioning.Interview and Evaluation GuideTechnical screen: SQL validation problem and API testing scenarios.Practical exercise: Create test cases and automated checks for a sample Databricks ETL and an API that returns pricing.LLM prompt exercise: Provide minimal prompts for a given LLM task and design evaluation metrics for correctness and safety.System design: Design an agentic QA pipeline for autonomous agents including test harness, safety gates, and monitoring.Behavioral: Examples of ticket triage, incident postmortems, and cross‑team collaboration.Success Criteria at 6 MonthsTest Coverage: Comprehensive, automated test suites for data pipelines, APIs, and core GUI flows.LLM Readiness: Reproducible prompt templates, evaluation metrics, and agentic QA checks integrated into CI.Reliability: Reduced production incidents and faster mean time to resolution through clear runbooks and test automation.Pricing Confidence: End‑to‑end validated pricing flows with scenario tests and regression guards.Collaboration: Trusted QA partner to engineering and product teams with documented processes and mentoring outcomes.

Key ResponsibilitiesTest Strategy and Planning Define and maintain QA strategy for data, API, GUI, microservices, and LLM/ML features; prioritize test coverage based on risk and business impact.Test Case Design and Execution Create clear, reusable test cases for functional, integration, regression, performance, and pricing scenarios; execute manual and automated tests and maintain test artifacts.Data and Database Testing Validate ETL/ELT pipelines on Databricks and Delta Lake; write and run T‑SQL validation queries; verify data lineage, completeness, and correctness.API Testing Design and run contract, integration, and negative tests for RESTful APIs; use Postman, OpenAPI/Swagger, and automated API test suites.GUI Testing Create and execute UI test plans for web applications; perform exploratory testing and maintain automated UI tests using Selenium, Playwright, or equivalent.Microservices Validation Test service contracts, message flows, resilience, and backward compatibility across containerized microservices; validate observability and error handling.LLM and ML QA Design evaluation metrics and test harnesses for ML models and LLM integrations; validate inference correctness, hallucination rates, latency, and cost tradeoffs.Prompt Engineering and Agentic QA Create minimalistic, cost‑aware prompts and prompt templates; implement agentic QA processes for autonomous agents including safety checks, guardrails, and evaluation pipelines.Pricing and Business Logic Testing Validate pricing engines, discount rules, billing flows, and edge cases; create scenario matrices and automated checks for pricing correctness.Automation and CI/CD Integrate tests into CI/CD pipelines; author automated test suites, test runners, and validation scripts; collaborate with SRE/DevOps on test environments.Ticketing and Incident Management Raise, triage, and manage tickets in Azure DevOps or Jira; document reproduction steps, root cause analysis, and remediation.Documentation and Collaboration Produce test plans, runbooks, release checklists, and postmortems; mentor junior QA engineers and participate in cross‑functional reviews.Required Skills and ExperienceExperience: Minimum 5 years in QA, software testing, or related roles.Databricks and Data Testing: Hands‑on experience validating Databricks pipelines, Delta Lake, and Spark outputs.Microsoft SQL Server: Medium proficiency in T‑SQL for validation queries and database testing.Azure: Practical experience with Azure Data Factory, ADLS/Blob Storage, Azure DevOps, and Azure Functions.API Testing: Strong experience with REST APIs, Postman, OpenAPI/Swagger, and automated API testing.GUI Testing: Experience with Selenium, Playwright, or equivalent UI automation tools and exploratory testing.Microservices: Familiarity testing containerized services, message queues, and service orchestration.LLM and ML Fundamentals: Understanding of ML lifecycle, model evaluation, prompt engineering basics, and LLM failure modes.Prompt Engineering: Ability to craft minimalistic prompts and maintain prompt libraries for reproducible tests.Agentic QA Awareness: Knowledge of agentic QA concepts, safety checks, and evaluation metrics for autonomous agents.Testing Tools and Automation: Experience with test frameworks, CI/CD integration, and test data management.Ticketing Tools: Experience using Azure DevOps, Jira, or similar.Soft

Skills

Strong analytical thinking, clear written communication, and cross‑team collaboration.

Nice to HaveCertifications: ISTQB, Azure Data Engineer, or Databricks certifications.Programming: Python for test automation and data validation; familiarity with Scala or Java.Observability: Experience with Application Insights, Datadog, Prometheus, or Grafana.Performance Testing: Experience with JMeter, k6, or similar tools.LLM Ops: Experience with prompt versioning, cost monitoring, and model evaluation frameworks.Infrastructure as Code: Familiarity with Terraform or ARM templates for test environment provisioning.Interview and Evaluation GuideTechnical screen: SQL validation problem and API testing scenarios.Practical exercise: Create test cases and automated checks for a sample Databricks ETL and an API that returns pricing.LLM prompt exercise: Provide minimal prompts for a given LLM task and design evaluation metrics for correctness and safety.System design: Design an agentic QA pipeline for autonomous agents including test harness, safety gates, and monitoring.Behavioral: Examples of ticket triage, incident postmortems, and cross‑team collaboration.Success Criteria at 6 MonthsTest Coverage: Comprehensive, automated test suites for data pipelines, APIs, and core GUI flows.LLM Readiness: Reproducible prompt templates, evaluation metrics, and agentic QA checks integrated into CI.Reliability: Reduced production incidents and faster mean time to resolution through clear runbooks and test automation.Pricing Confidence: End‑to‑end validated pricing flows with scenario tests and regression guards.Collaboration: Trusted QA partner to engineering and product teams with documented processes and mentoring outcomes.