Network Automation Framework
MAT at a Glance
MAT is an automation framework developed by Iquall Networks for large telecommunications networks. MAT’s main goal is to generate a work environment oriented to transform the operation of the network towards a dynamic environment such as SDN and NFV using tools and concepts from the Devops movement.
Simple, Fast and Modular
With its Hyper Converged infrastructure architecture based on containers and Kubernetes, MAT can provide real HA capabilities, rapid deployment and almost no network outage on upgrade operations.
Continuous Integration (CI/CD)
The Continuous Integration Manager is in charge of finding and managing the changes in elements of every automation and use case (automation files in Python, Ansible playbooks, Jinja2 templates, Parsers, etc) and coordinate the versioning of such files to maintain an historical control of modifications. The implementation of this module is based on GIT, the most widely used module in the market for source code management in software development, where it can also be used to track changes in any set of files. As a distributed revision control system, it is geared towards being fast, maintaining data integrity, and has support for distributed workflows. Also, this module provides the developer with an IDE (Integrated Development Environment), code editor, playbook testing options and syntax highlighting.
Jobs
A Job in MAT represents any automation execution of any kind of available application. As such, every time that a Network Workflow, a Network Task or any automation application is executed, a new Job is created to reflect on the system that execution in a unique, centralized and isolated way from the application that has triggered it.
Through the Job Manager, MAT can control the distributed execution of tasks (task execution scheduler), regardless of which application is requiring the execution of that automation. This module allows the execution of automation flows in an immediately or scheduled fashion, belonging to different MAT applications such as Network Workflows, Network Tasks, Backups, Compliance, Snapshots, etc. in an immediately or scheduled fashion and providing a suitable execution environment and features for the end user such as visualization of the execution tasks path, task logs, reports of results, errors encountered, initial parameters management and much more information related to that specific execution.
Schedules
MATs Job Manager also offers the possibility of viewing all the schedules created and available for future executions, regardless of the application that requires them. Scheduling allows jobs to be executed in delayed time when they are defined, and offers different configuration options. A user can schedule a job from within an application instance. Each application in MAT allows the user to establish when the executions should occur with different scheduling configurations to choose from.
Available execution schedules are:
- Single Job: Allows the user to set a deferred execution of a single job at a specific time.
- Periodic Job: Allows the user to set a time when executions should start.
- Continuous Job: Allows the user to set an interval of time between executions of the same job. That being the case, a new job will be launched when the ending time of the last job is greater than the specified schedule time.
Network Workflows
Network Workflow application allows you to define and control the execution of automation flows in operations designed to be workflows with multiple and complex task sequences, that may have a significant impact on the network. This module allows the specification of different execution flows according to the goal pursued, where every task that comprises it can be specified for each flow. In addition, it offers excellent traceability, monitoring and control of the status and output of each specific task that is part of the execution flow.
Network Workflow application also allows defining the initial parameters suitable of a workflow, the execution conditions (whether it is available to run in the development environment, the concurrency limit, among others) and more operational issues such as the possibility of adding specific documentation for the automation or assign a category using tags to make your search easier.
Network Tasks
Network Task application allows the user to run simple and straightforward automation processes, with much faster execution time than a regular Network Workflows and without the complexity of the flow control. As a network automation alternative to Network Workflows, Network Tasks are simpler and intended to be used for quick actions rather than a Network Workflow execution design.
This module allows the specification of simple and straightforward tasks written in Python, with a single objective purpose. As well as Network Workflows, it offers excellent traceability, monitoring and control of the status and output the task.
Data Modeling
Data modeling in MAT aims at modeling, managing and storing the entities and information structures that are necessary for the realization of each use case within MAT. Its function is to provide a robust but friendly interface for the definition of these data structures. Its predefined and commonly used data fields allow the user to control common validations regarding the type of value, format, etc. In addition, it allows us to associate different structures to each other, providing the flexibility to create a model of related information entities without the need for a low-level database definition and field structure itself, such as defining a primary key, the data type of each field, its length, etc.
Machine Learning Module
The Machine Learning module incorporated in MAT allows, in a parameterizable way, to dynamically detect traffic patterns (or any type of KPI evaluated) and predict the future values that the curve will take, in a future time delta.
The module essentially uses measurements from the MAT Time-Series DB, to analyze the existing values and to be able to calculate and feed back the DB with new trending and forecasting information. It also uses unsupervised learning techniques, which in essence do not require human intervention or pre-labeled data, and therefore can produce results faster and fully automatically.
Performance Module
The Performance Measurement Module is a module mainly used both for reports of specific use cases, as well as for feedback and analysis on network elements (NE) KPIs. This module allows the analysis of various KPIs to be monitored based on a historical measurement . It is implemented using Time-series DBs, which are databases optimized for time-series data, allowing queries of historical data, full of time intervals, accumulations, and arbitrary time zone conversions to be handled much more effectively.
Its main function is to make dashboards available for users, and to control automation processes triggering , so that it is possible to execute a certain specific automation task or workflow based on defined thresholds, on slopes in the curves or various pending status conditions of the measurements of the chosen KPIs. In addition, together with the Event Handling Module, it allows executing a certain specific workflow based on the occurrence of a certain event.
As examples, we can mention the execution a network reconfiguration (Example: changes in BGP distribution policies or priority of a specific CDN) based on a detection of a performance problem, or to start workflows based on monthly averages, SLAs measurements and situations detected automatically (Machine learning).
Regarding alert management, MAT allows a concrete and specific definition of the KPIs measured and existing in each use case, in which it is possible to define the type of measurement to be used, the alert trigger (based on thresholds, relative values or in binary state) and other significant parameters, such as the assignment of filter or selection tags. These alerts can be used as the initiator (trigger) of an automation flow, either a Network Workflow, Network Tasks, or an administrative BPM workflow.
Advanced Inventory Module
MAT’s Advanced Inventory Module is an extremely versatile module that allows you to combine various sources of information to solve complex queries that require real-time association of data from each of these sources. The main goal of this module is to be able to offer a flexible and centralized query method with real time data association and graphic result generation.
The module allows to define an integrated logic of association of data in real time, with information from different data sources, by means of a programmable algorithm that allows determining how the data is interrelated in order to solve complex queries with multiple parameters and filters.
It admits different types of sources of information, some being based on information queried at the moment (online query), others processed on a batch fashion (imported or synchronized from external repositories) and even interrelated with information obtained from domain network managers (NMS) or network elements directly.
Events Handling Module
Events Handling Module is a module integrated in MAT mainly used to perform analysis of the events that occur in the network through the output of automated processes, reception of events from devices through syslog, SNMP Traps, netflow, CDRs, or reception of events from network managers (NMS). It is implemented in MAT through the Elastic Stack component suite.
Regarding the collection of events in MAT, it is based on a very light and fast data processing server-side pipeline, which allows data to be collected from a very large variety of sources, transformed on the fly and sent to the search engine and elasticsearch-based analysis. The collector has multiple open-source plugins pre-built to easily index and normalize received data and events.
