Job Summary:
We are seeking a hands-on Databricks Administrator with experience in AWS-based deployments to support the setup, configuration, and day-to-day administration of Databricks E2 workspaces. The ideal candidate will work closely with platform engineers, data teams, and security personnel to ensure smooth operations, performance optimization, and compliance with organizational policies.
- Set up and configure Databricks E2 workspaces across multiple AWS accounts.
- Manage user onboarding, role-based access control (RBAC), Unity Catalog, and workspace entitlements.
- Administer clusters (interactive, job, and shared clusters), including configuration, scaling policies, and runtime upgrades.
- Create and manage secret scopes, instance profiles, and external locations for secure access to AWS resources (S3, Redshift, RDS, etc.).
- Support day-to-day platform operations, including job monitoring, failure resolution, cluster optimization, and cost tracking.
- Work with GitHub Actions, Terraform, and CI/CD pipelines to automate workspace deployment and governance.
- Monitor workspace performance, logs, and usage metrics to identify inefficiencies and proactively resolve issues.
- Collaborate with security teams to ensure encryption, audit logging, and policy enforcement are in place.
- Provide support to data engineers and analysts on best practices, troubleshooting, and resource configuration.
Required Qualifications:
- 3+ years of experience with Databricks in a production setting.
- Strong understanding of AWS services (S3, IAM, EC2, RDS, Redshift, KMS).
- Hands-on experience with Unity Catalog, Cluster Policies, and workspace-level security features.
- Experience writing Terraform or managing Databricks via APIs and CLI.
- Proficient in debugging job failures and performance bottlenecks in Spark/Delta workloads.
- Familiarity with Git-based CI/CD (e.g., GitHub Actions) for infrastructure and notebook deployments.
- Strong understanding of access control, entitlement models, and secure data handling.
- Ability to create and maintain documentation, SOPs, and runbooks for ongoing support.
- Should hold existing IRS MBI clearance (Need to provide MBI letter).
Preferred Qualifications:
- Databricks Certified Associate/Professional Admin or Data Engineer.
- AWS Certifications
Job Types: Full-time, Contract
Pay: $105,000.00 - $145,000.00 per year
Benefits:
- 401(k)
- Dental insurance
- Health insurance
- Paid time off
- Vision insurance
Application Question(s):
- Must Have IRS MBI Clearance
Work Location: Remote