Big Data Integration
As a data engineer on a project for a private bank, I worked on moving data from an existing warehouse into a centralized Cloudera–Hadoop repository to support analytics.
Connecting source systems to the data lake
The project required end-to-end data integration and governance. Talend managed the integration process, with Sqoop transferring data from source systems into the Cloudera cluster.
Once the data was stored in HDFS, Talend processed it and sent the integrated data back to Cloudera.
The integration process
- Trigger process
- Sqoop process
- Format conversion
- Data integration