Job Description
Summary
The Data Engineering team is part of the broader DevOps department, and sits at the core of the business, bridging the fast-paced and never-ending data feeds from the BitMEX trading platform to the entirety of the firm. Whether it is about business development, financial products, risk management, compliance & market surveillance, marketing consistent and reliable data is key, making Data Engineering the cornerstone to the firms success.
Responsibilities
- Design and maintain enhancements to our data warehouse, data lake, and data pipelines to increase their reliability and consistency
- Improve queriability of large historical datasets through industry-standard tools, careful data representation and aggregation for both technical and business units
- Ensure data governance and security/retention policies can be implemented and enforced
- Ensure that operational system integrations driven from the data stack are running, monitored and available
- Continually review the data platform to ensure that it is fit for purpose and meeting the needs of the business
- Support and maintain downstream integrations from our data lake, for example business intelligence and visualization tools and third party systems
Qualifications
- Experience in the data engineering field with demonstrated design and technical implementation of data warehouses
- Experience with OLAP databases, how they differ from OLTP databases, and data structuring/modeling with understanding of key data points in a business sense for trade-offs between storage/performance and usability
- Experience with building, deploying, and troubleshooting reliable, idempotent and consistent data pipelines working with disparate in-house and external data-sources, e.g. using Airflow DAGs
- Experience with AWS Redshift, Glue Data Catalog, S3, PostgreSQL, Parquet, Iceberg, Trino, and how they are managed using Terraform & Kubernetes
- Experience with data loading, extraction, manipulation and preparing data for ingestion and integration with visualization platforms such as Tableau.
Skills
- AWS
- Data Structures
- Database Management
- SQL