AlphaZetta’s data engineers design and deploy your means for data acquisition and integration. They will create data access protocols; data quality and governance policies; data transformation pipelines; and target load routines.
In many projects, 80% of the time is spent on data preparation and aggregation. Only 20% is reserved for real analytical work. With AlphaZetta’s Data Acquisition and Integration service, we ensure that clients can focus on the 20%.
Data acquisition and integration tasks can be grouped into two distinct categories. ETL/ELT handles functional aspects on how to extract data from the source system to load them into target platforms including all steps to transform the data into a target format. Our data governance process ensures that all processing of data meets corporate policies and enterprise standards.
Stage 1: Reference pipelining
In the first step, an AlphaZetta solution architect is integrated into the engineering team of the client. The team decides on a reference data stream and perform the following steps:
- Revise internal services user roles to facilitate access to incoming data and to store data in the corresponding target systems
- Design of a workflow that then loads the data from the source to the target system, adding required transformations
- Decide on technologies for message queues, streaming, or workflow orchestration, if needed
- Integrate encryption in transit and at rest where it is necessary
- Define an access policy based on the least privilege model
- Aggregate, transform, and prepare data for analytics
- Create a data catalog to gain more insights into the client’s data
- Reference analysis and report of the integrated data.
Stage 2: Process refinement
Subsequent data acquisition and integration for futher data streams is based on the insights gained in the kick-off and performed as a process. Processes can be adjusted depending on the parameters of the data (e.g. structure, volume, delivery (streaming or batch)).
Associated programs: Data and Analytics Cloud Migration; Analytics Experience (AX)
Commonly associated services: Cloud Native Design for Analytics, Agile Deployment; BI, Data Visualisation and Dashboard Development .