Types of Federated Learning in Machine Learning

Federated Learning is a powerful technique that allow a single machine to learn from many different source and converting the data into small pieces sending them to different Federated Learning (FL) is a decentralized of the machine learning paradigm that can enables to model training across various devices while preserving your data the data privacy.

In this article, we are going to learn about federated learning and discuss it's types.

What is Federated Learning?

Federated learning is a machine learning setting where the goal is to train a model across multiple decentralized edge devices or servers holding local data samples, without exchanging them. This approach contrasts with traditional centralized machine learning techniques where all data is uploaded to one server. Instead, in federated learning, the model is trained iteratively at multiple points of data origin, which send model updates rather than raw data to a central server.

Key Components of Federated Learning

1. Local Clients/Devices: These are the nodes or endpoints where data is generated and stored. Each node participates in the training process by computing updates to the model based on its local data.
2. Central Server: The server coordinates the learning process, aggregating the updates received from clients to improve the global model, then distributes the updated model back to the clients.
3. Communication Protocol: This governs the transmission of information (model parameters and updates) between clients and the central server, ensuring efficiency and security

Types of Federated Learning

1. Centralized vs. Decentralized Federated Learning
2. Horizontal vs. Vertical Federated Learning
3. Cross-silo vs. Cross-device Federated Learning

Centralized vs. Decentralized Federated Learning

Centralized Federated Learning

Centralized Federated Learning involves a central server that orchestrates the training process. Participants (clients) train local models on their data and send the updated model parameters to the central server. The server aggregates these parameters to update the global model, which is then distributed back to the clients.

Key Features:

• Coordination: The central server coordinates the training, aggregation, and distribution processes.
• Aggregation: Model updates are aggregated at a single point, simplifying the update process.
• Control: The central server has control over the training process, making it easier to manage.

Example Use Case: A healthcare consortium where hospitals train models locally on patient data and a central server aggregates these models to improve disease prediction models.

Decentralized Federated Learning

Decentralized Federated Learning eliminates the need for a central server. Instead, participants communicate directly with each other to share and aggregate model updates. This peer-to-peer communication ensures that there is no single point of failure and enhances privacy by distributing the aggregation process.

Key Features:

• No Central Server: Participants communicate directly, reducing the reliance on a central authority.
• Robustness: The absence of a central server reduces the risk of a single point of failure.
• Privacy: Enhanced privacy as aggregation is distributed across participants.

Example Use Case: A network of mobile devices collaboratively training a model for predicting app usage patterns without relying on a central server.

Horizontal vs. Vertical Federated Learning

Horizontal Federated Learning (HFL)

Horizontal Federated Learning (also known as Sample-based FL) occurs when datasets across different participants share the same feature space but differ in samples. Essentially, the datasets have the same structure (features) but contain data about different entities.

Key Features:

• Same Features: Datasets have the same features but different samples.
• Collaboration: Participants can collaboratively train a model without sharing their actual data.
• Applicability: Suitable for scenarios where different entities have similar data structures.

Example Use Case: Multiple banks training a fraud detection model on their transaction data without sharing the actual transaction records.

Vertical Federated Learning (VFL)

Vertical Federated Learning (also known as Feature-based FL) occurs when participants have datasets that share the same sample space but differ in features. This means that each participant holds different attributes about the same set of entities.

Key Features:

• Same Samples: Datasets have the same entities but different features.
• Feature Combination: Enables the combination of features from different participants to train a more comprehensive model.
• Privacy: Data privacy is maintained as participants only share intermediate computations.

Example Use Case: A partnership between a retail company and a bank where the retail company has purchasing data and the bank has financial data about the same customers.

Cross-Silo vs. Cross-Device Federated Learning

Cross-Silo Federated Learning

Cross-Silo Federated Learning involves a small number of reliable and stable participants (silos) such as organizations or institutions. These silos typically have significant computational resources and stable network connections.

Key Features:

• Few Participants: Involves a limited number of trusted entities.
• Stable Environment: Participants have stable and reliable computational and network resources.
• Large Datasets: Each participant usually possesses large amounts of data.

Example Use Case: Universities collaborating on a research project where each university has its dataset but wants to build a common predictive model.

Cross-Device Federated Learning

Cross-Device Federated Learning involves a large number of devices such as smartphones, IoT devices, or edge devices. These devices are typically less reliable and have varying computational capabilities and intermittent network connectivity.

Key Features:

• Many Participants: Involves a large number of devices.
• Unstable Environment: Devices may have limited computational resources and unstable network connections.
• Small Datasets: Each device typically has a small amount of data.

Example Use Case: Training a predictive text input model on mobile devices where each device contributes to the model without sending user data to a central server.

Conclusion

Federated Learning offers a versatile framework for collaborative machine learning while preserving data privacy. By understanding the different types of Federated Learning—Centralized vs. Decentralized, Horizontal vs. Vertical, and Cross-Silo vs. Cross-Device—organizations can choose the approach that best fits their needs and constraints. Each type has its unique advantages and challenges, but all share the common goal of enabling robust, privacy-preserving machine learning across distributed data sources.