- Setting up BPCE-SA Datalake – HORTONWORKS
- Data ingestion framework development from scratch (Java 8, Spring, Scala, Spark2, Hive, Hbase, HDFS, git, jenkins, XlDeploy, KMS, Ranger…) :
- Software architecture Design
- Development of the different bricks of the framework
- Establishment of the Devops chain to ensure continuous integration and deployment
- Ensure data security throughout the data manipulation chain (encryption before and after ingestion) with access management to exposed data
- Detecting Check’s Frauds (Azure, Databricks, Python) :
- Encryption of checks on premise beforse sending them to Azure Storage
- Ensuring the good transmission of millions of checks from on-premise to Azure blob Stroage
- Ensuring data(checks) decryption on Databricks and the connection to Key Vault
- Optimimizing the Loading of millions of checks to Spark dataframes in order to decrypt them in memory.
- Implementing (on-premise) data anonymization for sensitive data
- Administration of the services of Hortonworks platform :
- HDFS storage management
- Management of the YARN resource manager (queues …)
- Spark Tuning
- Integration of Ambari with LDAP
- Configuration of high availability and cluster kerberization
- Configuration of Ranger policies (HDFS, Hive) and KMS encryption
- Ingestion of different business data (risks and finance) within BPCE SA in the datalake following a threelayer design on HIVE (RAW (raw data), ENHANCED (prepared data), EXPOSURE (data to be exposed)) (Spark , Hive, Scala, PySpark(python) )
- Development of a tool that implements non-regression tests for large positional data files ( for risks and finances) between different closing dates (Spark, scala)
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 555$/jour |
Expérience | 4 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 445$/jour |
Expérience | 4 ans |
Localisation | Roumanie |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 900$/jour |
Expérience | 3 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 365$/jour |
Expérience | 3 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 190$/jour |
Expérience | 3 ans |
Localisation | Allemagne |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 660$/jour |
Expérience | 4 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 280$/jour |
Expérience | 3 ans |
Localisation | Roumanie |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 900$/jour |
Expérience | 4 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 660$/jour |
Expérience | 4 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 555$/jour |
Expérience | 3 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 460$/jour |
Expérience | 3 ans |
Localisation | France |
Data Engineer
Type de contrat | Freelancers |
---|---|
Taux journalier | 660$/jour |
Expérience | 3 ans |
Localisation | France |