Resources for building ML pipelines

Breaking Ground: The steps involved in building ML pipelines, resources to learn them:

Data Collection

“Garbage in ,Garbage out “, this is perhaps the golden rule of data science or any Machine Learning model , no matter however advanced the algorithm is , if the data isn’t good the outputs wont be good.

Research have time and again found that realatively simplistic algorithms with quality and class balanced data have the potential to outperform sophesticated algorithms with bad data in terms of metrics and resources consumed [ref.], which can help us realise why access to data and preprocessing it is a crucial step for any ML project pipeline, given below are some sources to refer to for open source datasets,repositories containing datasets:

UCI Machine Learning Repo:

Benefits

• Huge Collection: There are more than 665 datasets in the repository, spanning a variety of topics including computer science, social sciences, artificial intelligence, and communication science.
• Simple Access: Users can search and download datasets straight from the repository's website, making the datasets conveniently accessible.
• Variety of Datasets: There are many different types of datasets in the repository, ranging from traditional datasets like Iris and Dry Bean to more recent ones like RT-IoT2022 and PhiUSIIL Phishing URL.
• Impact of Citations: The repository has received over a thousand citations, demonstrating its considerable influence on the machine learning community.

Drawbacks:

• Restricted Filtering: Although the repository has a search feature, it lacks sophisticated filtering choices for particular kinds of datasets, like datasets that are picture or video-based.
• Requires Preprocessing of the Data: Before being used for machine learning applications, some datasets might need to undergo additional preprocessing.
• Limited Support for Certain Tasks: Certain tasks, like as image recognition and natural language processing, may require additional datasets or preprocessing, and are not specifically supported by the repository

Kaggle datasets:

Kaggle is a popular machine learning and data science platform which provides a large catalogue of open-source datasets from various industries and verticals. Here are some salient features of Kaggle as a source of data:

Benefits:

• Huge Collection of datasets: Thousands of open-source datasets from a variety of industries, including finance, sports, government, food, and more, are available on Kaggle.
• Simple Search: Using the platform's search feature, users may quickly locate particular datasets by keyword, topic, or category.
• Community Support: Data science and machine learning can be done in a collaborative atmosphere with Kaggle, a community-driven platform that allows users to share and work together on datasets, projects, and competitions.
• Competition and Incentives: Kaggle organizes contests with actual awards for participants, promoting machine learning innovation and advancement.
• Flexible Data intake: Kaggle makes it simple to connect datasets into a variety of tools and workflows by providing flexible data intake.

Google Dataset Search:

Google Dataset Search Benefits:

• Huge Collection: One of the biggest dataset collections available is Google Dataset Search, which has over 25 million datasets.
• Simple Search: Finding particular datasets is made simple by the search engine's ability to find datasets using just one term.
• Data-Sharing Ecosystem: By facilitating the development of a data-sharing ecosystem for datasets needed to train AI and machine learning algorithms, Google Dataset Search encourages cooperation and creativity.
• Flexibility: Datasets from a variety of repositories, including those hosted on AWS, Azure, and Kaggle, can be found using the search engine. 

Drawbacks:

• Preparing the data is necessary. Prior to being utilized for machine learning applications, certain datasets might need to undergo extra preprocessing.
• Limited Filtering Options: Although the search engine has a search function, it does not have the ability to filter datasets of a certain type, like image or video datasets, in an advanced manner.

4. Zenodo (open data repositories maintained by CERN):

Benefits:

• Multidisciplinary: Researchers can post research papers, data sets, research software, reports, and any other materials connected to their work in Zenodo, an open repository for broad purposes.
• Persistent DOIs: Every submission is given a persistent DOI, which facilitates easy citation of the stored materials.Large Capacity: Zenodo is appropriate for large datasets and software releases as it permits uploads of files up to 50 GB.
• Open Access: To encourage openness and cooperation in the scientific process, Zenodo makes study findings, data, and analysis code freely and publicly accessible.
• Integration with GitHub: Zenodo has an integration with GitHub that facilitates software development workflows and enables the automatic archiving of software releases

Drawbacks: 

• Limited Support for Certain activities: Since Zenodo is a general-purpose repository, it might not be able to handle certain activities like natural language processing or picture recognition, which call for further preprocessing or datasets. Requires Preprocessing of the Data: Before being used for machine learning applications, some datasets might need to undergo additional preprocessing. 
• Restricted Filtering: Although Zenodo has a search engine, it doesn't provide sophisticated filtering options for particular kinds of information, including datasets that contain images or videos. Technical Requirements: Because Zenodo is based on open source code, some researchers may find it difficult to upload and manage datasets without technical knowledge.

5.Papers with Code : Datasets repositories

Benefits:

• Collaboration: By enabling scholars to work together on datasets and code, Papers with Code helps to both create new research and enhance previously completed work.
• Version Control: Researchers can keep track of changes and keep a record of their work thanks to version control offered by the repository.
• Portability: Researchers can work from different machines using Papers with Code without having to worry about updates being overwritten or lost.
• Forking: The repository allows forking, which encourages creativity and experimentation by allowing researchers to produce different versions of datasets and code.
• Papers with Code provide metadata for datasets, which facilitates the retrieval and comprehension of the data.

Drawbacks:

• Security risks: Because sensitive data and code are shared in public repositories like Papers with Code, security risks may arise.
• Data Quality: Papers with Code may contain datasets and code of varying quality, some of which may be erroneous or incomplete.
• Overwhelming Amount of Data: Researchers may find it difficult to locate the pertinent data and code for their particular needs due to the abundance of datasets and code that are available.
• Papers with Code may have licensing problems since different licenses may apply to some datasets and code, leading to ambiguities and potential legal problems.
• Updating and correcting information and code in the repository is necessary for maintenance, which can be a substantial task.

Data Preprocessing: Separating the Signal from the Noise.

Data obtained by from real world sample collection is rarely ever standardized and in order for the machines to comprehend the data’ features they need to be selected,cleaned , their distribution should be understood and uniform structure should be created for upcoming inputs in the pipeline ,data prepocessing includes tasks such as handling missing values, data normalization, feature scaling, and data transformation.

Here are 5 Unique repositories which can help you learn about data preprocessing:

1. Daily Dose of Data Science Github repository by Avi Chawla:

‍Avi's repository is a curated collection of data preprocessing resources, including tutorials, snippets,references ,datasets and example excersises and projects.

Advantages:

• Comprehensive Coverage: The repository covers a wide range of data preprocessing topics, ensuring you'll find something that suits your needs including libraries like Numpy,Pandas.
• Easy to Follow: The tutorials and guides are written in a clear and concise manner, making it easy to follow along and learn.
• Practical Applications: The projects and examples are based on real-world scenarios, helping you apply data preprocessing concepts to your own projects.
• Constantly Updated: Avi regularly updates the repository with new content, ensuring you'll always have access to the latest and greatest in data preprocessing.

Drawbacks:

• Information Overload: With so many resources available, it can be overwhelming to navigate and find the specific information you need.
• Lack of Depth in Certain Areas: While the repository covers a wide range of topics, some areas may not be explored in as much depth as others.
• Dependence on Avi's Updates: Since the repository is maintained by Avi, the frequency and quality of updates may vary, which could impact the usefulness of the resource.

2. Unstructured-IO/unstructured: The Swiss Army Knife of Preprocessing

Advantages:
This repository offers a treasure trove of open-source libraries and APIs for building custom preprocessing pipelines. With support for various data formats like PDF, images, and JSON, you'll be able to tackle even the most complex data sets. Plus, its wide range of preprocessing tools for NLP, information retrieval, and deep learning will make you a master of data manipulation.

Drawbacks:
However, be prepared for a steeper learning curve due to the complexity of the libraries and APIs. You may also need to invest some time in setting up and configuring the tools for your specific use case.

3. Pytorch/torcharrow: Lightning-Fast Preprocessing with PyTorch

Advantages:
If you're already familiar with PyTorch, you'll love torcharrow. This high-performance model preprocessing library is built on top of PyTorch and provides blazing-fast data processing and caching mechanisms. With support for CSV, JSON, and Avro data formats, you'll be able to work with a variety of data sources.

Drawbancks:
Just keep in mind that torcharrow has limited documentation and community support compared to other PyTorch libraries. You may also need prior knowledge of PyTorch and its ecosystem to get the most out of this repository.

4. Vanderschaarlab/hyperimpute: The Imputation Powerhouse

Advantages:
Hyperimpute is a comprehensive framework for prototyping and benchmarking imputation methods. With its flexible and modular architecture, you'll be able to create custom imputation pipelines that suit your specific needs. Plus, it supports various imputation algorithms and techniques, making it a go-to resource for anyone working with missing data.

Drawbacks:
However, hyperimpute is limited to imputation methods and may not cover other aspects of data preprocessing. You may also need prior knowledge of imputation techniques and algorithms to fully leverage this repository.

5. CleanLab: Your Go-To Toolkit for Data Preprocessing and Cleaning

Real World data is often messy and unbalanced and while these can be easy to detect in low dimensional or visual datasets, doing the same in higher dimensional datasets and fixing them can be time consuming , CleanLab aims to automate a large part of this process in 1 stop tool that offers all abilities of processing, so that researchers and practitioners can devout their time to valuable tasks like ML modelling.

Advantages:

• One-Stop-Shop for Data Cleaning: CleanLab has got you covered with its comprehensive toolkit for data preprocessing and cleaning. No more jumping between different tools and libraries!
• Flexible and Adaptable: CleanLab supports various data formats, including CSV, Excel, and JSON, making it easy to work with different data sources.
• Automation Made Easy: With CleanLab, you can automate tedious data preprocessing tasks, freeing up more time for the fun stuff - like building amazing machine learning models!
• Data-Centric AI and ML: CleanLab is designed with data-centric AI and machine learning in mind, providing tools for finding mislabeled data, uncertainty quantification, and more.

Disadvantages:

• Learning Curve Ahead: CleanLab is a powerful toolkit, but it may take some time to get familiar with its features and tools. Be prepared to invest some time in learning the ropes!
• Data Preprocessing 101: You'll need a good understanding of data preprocessing concepts and techniques to get the most out of CleanLab. If you're new to data preprocessing, you may want to start with some online courses or tutorials.
• Not for Small-Scale Tasks: CleanLab is designed for large datasets and complex data workflows. If you're working with small-scale data, you might find it overwhelming.
• Community Support Could Be Better: While CleanLab is an amazing toolkit, its community support and documentation could be improved. You might need to dig around for answers to your questions.

Data Visualization: Visualize the Beauty of insights and the Beast of Inconsistency:

We need data visualization because it helps us make sense of complex information, spot patterns, and tell stories with our data. The elements of data visualization include the type of plot, color, size, shape, and interactivity - all working together to create a clear and concise visual representation of our data.

Now, when it comes to choosing the right type of plot, it really depends on the type of data we're working with. Here are some popular plot types and the data they're often used for:

• Scatter plotsts: Great for showing relationships between two continuous variables, like the correlation between temperature and ice cream sales,( Disclaimer:- Correlation does not always equate to causation), correlation can be usefull to eleminate reduntant features.
  

• Bar charts: Ideal for comparing categorical data or showing the distribution of a single variable. For example, you can use a bar chart to compare sales figures for different products or visualize the frequency of different categories.
• Line graphs: Ideal for showing trends over time, like stock prices or website traffic.
• Histograms: Useful for displaying the distribution of a single continuous variable, like the ages of a population.
• Heatmaps: Awesome for visualizing relationships between two categorical variables, like the correlation between different product features and customer satisfaction.
• Box plots: Handy for comparing the distribution of a single continuous variable across different groups, like the salaries of different departments.
  

1.Seaborn

Advantages:

• Easy to Learn: Seaborn is a Python-based data visualization library that is relatively easy to learn, even for those without prior experience.
• Highly Customizable: Seaborn provides a wide range of tools and features to customize your visualizations.
• Beautiful Visualizations: Seaborn is known for its beautiful and informative visualizations, making it a great choice for data exploration and presentation.

‍

Disadvantages:

• Not as Interactive: Seaborn is primarily designed for static visualizations, which can limit its interactivity compared to other libraries.
• Not as Web-Friendly: Seaborn is not as well-suited for web-based visualizations as other libraries, such as Plotly.

2. Matplotlib:

Advantages:

• Easy to Learn: Matplotlib is a Python-based data visualization library that is relatively easy to learn, even for those without prior experience.
• Highly Customizable: Matplotlib provides a wide range of tools and features to customize your visualizations.
• Large Community: Matplotlib has a large and active community, ensuring there are many resources available to help you learn.

‍

Disadvantages:

• Not as Interactive: Matplotlib is primarily designed for static visualizations, which can limit its interactivity compared to other libraries.
• Not as Web-Friendly: Matplotlib is not as well-suited for web-based visualizations as other libraries, such as Plotly.

3. Plotly

Advantages:

• Interactive Visualizations: Plotly is designed for interactive visualizations, making it a great choice for web-based applications.
• Easy to Learn: Plotly has a relatively low barrier to entry, even for those without prior experience with data visualization.
• Highly Customizable: Plotly provides a wide range of tools and features to customize your visualizations.

Disadvantages:

• Not as Flexible: Plotly is designed for specific types of visualizations, which can limit its flexibility compared to other libraries.
• Not as Lightweight: Plotly can be a relatively heavy library, which can impact performance in certain applications.

4. Prophet:

Advantages:

• Time Series Expertise: Prophet is a Python-based library specifically designed for time series forecasting and visualization.
• Easy to Learn: Prophet has a relatively low barrier to entry, even for those without prior experience with time series analysis.
• Highly Customizable: Prophet provides a wide range of tools and features to customize your visualizations and models.

‍

Disadvantages:

• Limited to Time Series: Prophet is primarily designed for time series data, which can limit its applicability to other types of data.
• Not as Web-Friendly: Prophet is not as well-suited for web-based visualizations as other libraries, such as Plotly.
  

  Fig-- BoxPlots and Heatmaps for Sensor Data

5.Plotnine

Advantages:

• Biological Data Expertise: Plotnine is a Python-based library specifically designed for biological data visualization, such as genomics and proteomics.
• Easy to Learn: Plotnine has a relatively low barrier to entry, even for those without prior experience with biological data visualization.
• Highly Customizable: Plotnine provides a wide range of tools and features to customize your visualizations.

Disadvantages:

• Limited to Biological Data: Plotnine is primarily designed for biological data, which can limit its applicability to other types of data.
• Not as Web-Friendly: Plotnine is not as well-suited for web-based visualizations as other libraries, such as Plotly.
  
    

Feature Engineering

Feature engineering is the process of transforming raw data into suitable features for modeling.
• It's crucial in the machine learning pipeline, impacting model performance.
• Feature engineering improves model accuracy and performance.
• It reduces data noise, leading to more robust models.
• It increases interpretability by gaining a deeper understanding of data and variable relationships.
• Techniques for Feature Engineering include feature selection, feature transformation, feature extraction, dimensionality reduction, and handling missing values.
• Mastering feature engineering unlocks the full potential of data, enabling more accurate, robust, and interpretable machine learning models.

Here are the top 5 repositories to learn feature engineering, along with their advantages and disadvantages, :

1. H2O/h2o-automl

Advantages:

• Automated Feature Engineering Magic: H2O's AutoML is like having your own personal feature engineering assistant - it automates the process, so you can focus on more important things!
• Scalability Superstar: H2O's AutoML is built to handle massive datasets, making it perfect for big data applications.
• Seamless Integration: H2O's AutoML integrates beautifully with H2O.ai's ecosystem of machine learning tools and platforms.

Disadvantages:

• Limited Customization Options: While H2O's AutoML is amazing, it might not offer the same level of customization as other repositories, like Featuretools.
• H2O.ai Ecosystem Dependence: H2O's AutoML is designed to work within H2O.ai's ecosystem, which might limit its applicability to other platforms and tools.

2. Scikit-learn-contrib category_encoders

Advantages:

• Category Encoding Mastery: Category encoders is like a category encoding ninja - it's got all the techniques you need to encode your categorical data like a pro!
• Easy to Learn, Easy to Use: Category encoders has a super simple API, making it easy to learn and use, even if you're new to category encoding.
• Customization Heaven: With category encoders, you can customize your category encoding pipeline to your heart's content, making it perfect for a wide range of applications.

Disadvantages:

• Category Encoding Only: Category encoders is primarily designed for category encoding, so if you need other types of feature engineering, you might need to look elsewhere.
• Not as Comprehensive as Other Repositories: While category encoders is amazing, it might not offer the same level of comprehensiveness as other repositories, like Featuretools.

3. PyOD/pyod

Advantages:

• Anomaly Detection Mastery: PyOD is like an anomaly detection superhero - it's got all the techniques you need to detect outliers and anomalies like a pro!
• Easy to Learn, Easy to Use: PyOD has a super simple API, making it easy to learn and use, even if you're new to anomaly detection.
• Customization Galore: With PyOD, you can customize your anomaly detection pipeline to your heart's content, making it perfect for a wide range of applications.

Disadvantages:

• Anomaly Detection Only: PyOD is primarily designed for anomaly detection, so if you need other types of feature engineering, you might need to look elsewhere.
• Not as Comprehensive as Other Repositories: While PyOD is amazing, it might not offer the same level of comprehensiveness as other repositories, like Featuretools.

4. tsfresh/tsfresh

Advantages:

• Time Series Feature Engineering Mastery: Tsfresh is like a time series feature engineering wizard - it's got all the techniques you need to extract meaningful features from your time series data!
• Easy to Learn, Easy to Use: Tsfresh has a super simple API, making it easy to learn and use, even if you're new to time series feature engineering.
• Customization Heaven: With tsfresh, you can customize your time series feature engineering pipeline to your heart's content, making it perfect for a wide range of applications.

Disadvantages:

• Time Series Data Only: Tsfresh is primarily designed for time series data, so if you're working with other types of data, you might need to look elsewhere.
• Not as Comprehensive as Other Repositories: While tsfresh is amazing, it might not offer the same level of comprehensiveness as other repositories, like Featuretools.

Here are the top 5 repositories for feature engineering in image, vision, and speech-based data, along with examples for each data category, and their advantages and disadvantages in a human-like tone:

5. Mmcv

Advantages:

• Image and Vision: Mmcv is a lightweight and efficient computer vision library that provides a wide range of feature engineering techniques for image and video data, its based on OpenCV and is optimized for Pytorch and Tensorflow

Example: Extracting object detection features from an image using mmcv's object detection algorithms.

• Mmcv is highly optimized for performance and has a simple and intuitive API. It's also highly customizable.

Disdvantages: 

• Mmcv is a relatively new library, and its community is still growing, not the most optimum option  in Tensorflow as compared to Pytorch.

6. Librosa

Advantages:

• Speech and Audio: Librosa is a Python library for audio signal processing that provides a wide range of feature engineering techniques for speech and audio data.
• Example: Extracting mel-frequency cepstral coefficients (MFCCs) from an audio file using Librosa.
• Librosa is easy to use and provides a high-level API for audio feature extraction. It's also highly optimized for performance.

Disadvantages: 

• Librosa is limited to audio signal processing and may not be suitable for other types of data.

Building  the ML Models: From Distributions to Deployments

Selecting,Initializing  the Model, and training it for our purpose:

1: Scikit-learn

Scikit-learn is a popular machine-learning library in Python, known for its simplicity and versatility. It provides a wide range of algorithms and tools for various ML tasks, including classification, regression, clustering, and more. The repository contains comprehensive documentation, examples, and tutorials to help users get started with sci-kit-learn.

Limitations of Scikit-learn:

• Limited support for deep learning models
• Lack of scalability for large datasets.

2. Mathematics for Machine Learning repository

Despite being primarily a theoretical book for learning maths behind ml, it's repositories have impelementations of many linear algebra based Feature Extraction,Dimesionality Reduction and Optimization techniques which help in gaining insight behind the under the hood processes of many black-box models.

Benefits:
Thorough explanation of the mathematical ideas required for machine learning
covers subjects including calculus, probability theory, linear algebra, and optimization techniques.
Ideal for both novice learners and seasoned practictioners to revise fundamentals.
Gives a strong basis for comprehending models and methods used in machine learning
updated frequently with fresh materials and content

Drawbacks:
For students who are unfamiliar with machine learning or mathematics, it could be too intense as in ,  certain subjects could be too complex for novices.
Demands a substantial investment of time and energy to finish.

3.Microsoft CNTK

Microsoft Cognitive Toolkit (CNTK) is a powerful deep learning framework developed by Microsoft. It supports both high-level and low-level APIs and provides excellent performance on various tasks. The repository offers documentation, tutorials, and examples for utilizing CNTK effectively.

Limitations of CNTK:

• Steeper learning curve compared to some other deep learning frameworks
• Smaller community compared to frameworks like TensorFlow or PyTorch, resulting in potentially limited community 

4.: PyTorch

PyTorch is a widely used deep learning library known for its dynamic computation graph and ease of use. It allows users to define and modify neural networks on the fly, making it suitable for research and prototyping. The repository includes documentation, tutorials, and a vast collection of pre-trained models for PyTorch users.

Limitations of PyTorch:

• Not as optimized as TensorFlow for large-scale production deployment
• Limited support for some advanced GPU features

5. Unity ML-Agents

Advantages: 

Unity ML-Agents is a popular open-source library for reinforcement learning and game development. It provides a wide range of tools and features for building intelligent agents.

Disadvantages: 

Unity ML-Agents may require prior knowledge of Unity and game development.

6. Ray

Advantages:
Ray is a high-performance distributed computing framework for reinforcement learning and other AI applications. It provides a wide range of tools and features for building scalable and efficient AI systems.

Disadvantages: 

Ray may require prior knowledge of distributed computing and AI applications.

‍

7:Stable Baselines-3:

Built on PyTorch, Stable Baselines3 (SB3) is a popular Reinforcement Learning (RL) library.

Benefits: 

• Sturdy Implementations: TD3, A2C, PPO, DDPG, and SAC are examples of dependable algorithms.
• Easy-to-use API: Makes RL agent evaluation and training simpler. Lots of tutorials and comprehensive documentation.
• Modular and Flexible: Simple to extend and customize. A vibrant community with strong support on GitHub and in forums.
  

Drawbacks:

• Restricted Algorithm Variety: Doesn't include certain uncommon RL algorithms.
• Resource-intensive: needing high-performance resources due to its computational demands. Learning Curve: Needs familiarity with PyTorch and RL concepts.
• Dependency on PyTorch: Not as practical for TensorFlow users.

All things considered, SB3 is a useful tool for both practical and resea

8.Hugging Face Transformers:

Advantages:

• State-of-the-art models: Hugging Face Transformers provides a wide range of state-of-the-art models for natural language processing (NLP) tasks, including BERT, RoBERTa, and XLNet.
• Easy to use: The library is designed to be easy to use, with a simple and intuitive API that makes it easy to integrate transformers into your NLP pipeline.
• Fast and efficient: Hugging Face Transformers is highly optimized for performance, making it fast and efficient for large-scale NLP tasks.
• Community support: The library has a large and active community of developers and researchers, which means there are many resources available to help you get started and stay up-to-date with the latest developments.
• Extensive documentation: The library has extensive documentation, including tutorials, guides, and API references, which makes it easy to get started and learn how to use the library.

Drawbacks:

• Steep learning curve: Although the library is meant to be user-friendly, it still necessitates a solid grasp of deep learning and natural language processing ideas, which may be a hurdle for novices
• Heavy on resources: Hugging Face Transformers need a lot of processing power.
• Requires PyTorch: Since the library is based on PyTorch, using Hugging Face Transformers requires having PyTorch installed and set up on your computer.
  poor compatibility with alternative frameworks Hugging Face Transformers has minimal support for TF and JAX, but it is primarily meant to be used with PyTorch.
• Not great for all NLP tasks: Not the greatest option for all NLP activities: It might not be the ideal option for tasks requiring a lot of domain-specific expertise or specialized models.

Tools for evaluating,packaging,saving and deploying models:

Now that we have trained our model , it important to test their performance  which is generally done using the train-test-validate feature which is available in libraries like Scikit Learn,Pytorch etc, after testing for the metrics The practicioner can reiterate the model training with adjusted paramaters untill satisfactory condition is met, and the models can be saved for deploying:

1.UltraAnalytics:

Benefits:

• End to End Framework: UltraAnalytics provides a thorough framework for preparing data that addresses several facets of feature engineering, data loading, cleaning, and visualization, among other areas of data preparation.
• Versatile: Users can tailor their workflows to meet their unique requirements with UltraAnalytics' versatile data pretreatment pipelines.
• Encourages Multiple Data Formats: The repository is compatible with a wide range of data sources because it supports multiple data formats, such as CSV, JSON, and Avro.
• Developers and users at UltraAnalytics are part of an active community that keeps the repository updated and enhanced.

Disadvantages:

• Steep Learning Curve: While the repository provides a user-friendly API, it may still require a significant amount of time and effort to learn and master its features and tools.
• Limited Documentation: The documentation for UltraAnalytics may be limited, making it challenging for new users to get started with the repository.
• Dependent on Other Libraries: UltraAnalytics may depend on other libraries and frameworks, which can add complexity to the overall workflow.
• Limited Support for Specific Use Cases: While the repository provides a comprehensive framework for data preprocessing, it may not cover all specific use cases or edge cases vulnerable to bias.
• Performance Issues: UltraAnalytics may experience performance issues when handling large datasets or complex data preprocessing pipelines.

2.RoboFlow:

Also a pipeline tool, tailor made for CV based applications for preparing ,processing and annotating datasets guided with ml models of its won to assist the user, often used as a supplementary tool for Ultra-Analytics

Benefits:

• Roboflow offers a thorough framework for developing computer vision models that addresses several computer vision-related topics, such as data loading, data augmentation, model training, and model evaluation.
• Simple to Use: Roboflow is accessible to developers and data scientists of all experience levels because to its user-friendly libraries and API.
• Roboflow facilitates the creation of computer vision pipelines that are both adaptable and adjustable, allowing users to customize their processes to meet particular needs and use cases.
• Encourages Multiple Data Formats: Roboflow is compatible with a wide range of data sources since it supports multiple data formats, such as photos, videos, and 3D point clouds.
• Enterprise-Grade Infrastructure: Roboflow has enterprise-grade infrastructure and compliance with SOC2 Type 1 certification and PCI compliance
• Transfer Learning: Roboflow Train allows models to learn iteratively by starting from the previous model checkpoint, jumpstarting its learning with knowledge generalized from other datasets‍

‍Drawbacks:

• It takes a lot of time and effort to master features due to the steep learning curve.
• Inadequate Documentation: Complicated for Novice Users.
• Reliance on External Libraries: Complicates the process.
• Latency Problems: May have trouble handling complicated jobs or big datasets.
• Weak Support for Certain Use Cases: Doesn't address edge caching or all specific use situations.

3.NLTK Library:

Benefits:

• Maturity: With a lengthy development history and a sizable user and contributor community, NLTK is a mature library.
• All-inclusive: NLTK offers a full range of resources and tools for NLP tasks, such as tokenization, text processing, and semantic reasoning.
• Simple to utilize: Even people without any prior NLP experience can easily utilize NLTK thanks to its user-friendly API. for tensorflow users.
• Comprehensive Documentation: NLTK provides a wealth of information in the form of tutorials, guides, and API references.

Drawbacks:

• Slow: NLTK can be slow for large datasets and complex NLP tasks.
• Limited Support for Deep Learning: NLTK does not have built-in support for deep learning models, which can be a limitation for some users.
• Outdated: NLTK's architecture and design may be outdated, which can make it less efficient and less effective for certain NLP tasks.

4.Google Mediapipe:

Pipeline conceptualized by Google to train,evaluate and integrate ML models like object detection and other CV based models in apps across all commonly used platforms.

Advantages:

• Versatility: MediaPipe can be utilized across different platforms, including Android, iOS, and desktop devices, making it a versatile tool for developers.
• Real-Time Capabilities: With MediaPipe, developers can create applications capable of real-time media data processing and analysis.
• Minimal Latency: Designed for quick processing, MediaPipe is an excellent choice for applications that demand swift and responsive media processing.
• Efficient Performance: MediaPipe is efficient and reliable for handling intricate media processing tasks, ensuring high performance.
• Adaptability: The framework's flexibility allows developers to tailor and expand its functionality according to their specific requirements.
• Open-Source: As an open-source framework, MediaPipe is free to use and distribute, and it fosters community participation in its development.

Disadvantages:

• Steep Learning Curve: MediaPipe has a steep learning curve, requiring developers to have a good understanding of media processing, computer vision, and machine learning concepts.
• Not Suitable for Simple Applications: MediaPipe is not suitable for simple applications that require basic media processing capabilities, as it is a complex and powerful framework that requires a significant amount of resources and expertise.

5.LangChain :

Benefits of LangChain:

• Scalability: Able to manage massive volumes of data.
• Adaptability: Fits well with a range of applications, including chatbots and AI systems that respond to queries.
• Extensibility: Permits developers to add custom functionality.
• User-Friendly: Offers a high-level API for combining data sources and language models.
• Open Source: May be used and modified without charge.
• Dynamic Community: sizable, vibrant developer and user base.
• Extensive and Simple to Understand Documentation: thorough and simple to read.
• Integrations: Flask and TensorFlow frameworks are among those with which it is compatible.
• Timely Maintenance and Enhancement: Crucial for efficient communication with linguistic models.

Disadvantages of LangChain:

• Requires substantial data and computational resources for effective language model utilization.
• Limited support for non-Python languages limiting its use inlarge scale enterprise applications
• Lacks comprehensive documentation, making it challenging for beginners.
• Not suitable for all real-world applications, especially those requiring complex data processing or multiple systems integration.
• May not handle errors as robustly as other frameworks.
• Lacks control over output parsing, potentially leading to formatting and accuracy issues.
• Requires more testing and debugging than other frameworks.
• May not integrate seamlessly with other tools and frameworks.
• Steep learning curve, especially for developers without prior experience.

Packaging the Model:

Model Packaging Overview

• Model Deployment: Enables easy deployment and management of models on cloud platforms, containerization, and orchestration.
• Model Sharing and Collaboration: Facilitates collaboration between data scientists, engineers, and stakeholders.
• Model Reproducibility: Ensures consistent results with the same inputs.
• Model Versioning: Facilitates tracking of changes and updates.
• Model Security: Protects sensitive data and intellectual property.
• Model Explainability: Allows understanding of prediction and decision-making processes.
• Model Scalability: Handles large datasets and high traffic.
• Model Flexibility: Allows easy integration with different data sources and systems.
• Model Maintenance: Ensures model accuracy and relevance over time.
• Model Governance: Ensures compliance with regulations and standards.

1.TFjs:

Advantages:

• Enables seamless deployment of TensorFlow models in web applications.
• Supports a wide range of devices and browsers.
• Provides a lightweight, compact package.

Disadvantages:

• May require additional processing power for complex models.
• Limited support for advanced features and customizations.

2.TF - savedmodel()

Benefits: 

• Facilitates quick and precise inference by allowing for the effective and scalable deployment of TensorFlow models.
• Ensures interoperability with a wide range of contexts by supporting a wide range of platforms and devices. offers a package that is adaptable and configurable, allowing users to customize the model to meet their own requirements.

Drawbacks: 

• To handle sophisticated models, more processing power might be needed, which could affect performance. 
• Limited streaming and real-time processing functionality, which may limit its applicability in some applications

3.TensorFlow Lite:

TensorFlow Lite is used to translate TensorFlow models for use on devices with limited resources, like robots and Internet of Things gadgets. Because of its efficient and lightweight design, it can be used for streaming and real-time processing applications. 

Benefits:

• Flexibility and Customization: A great deal of flexibility and customization is possible with TensorFlow Lite to satisfy particular or changing needs in robotics and Internet of Things applications. 
• Cost-Effectiveness: TensorFlow Lite is available to all sizes of enterprises and is free and open-source. 
• Integration Capabilities: TensorFlow Lite offers smooth integration with robotics and Internet of Things systems by integrating seamlessly with a variety of data sources, ML frameworks, and deployment settings. Learning and Skill
• Development: Teams can gain valuable hands-on experience by using TensorFlow Lite, which exposes them to the newest technologies and techniques in the sector.

Drawbacks:

• Absence of Vendor Support Around-the-Clock: TensorFlow Lite lacks specialized vendor support, therefore community members might be your best bet for help.
• TensorFlow Lite is free in terms of itself, but there are ongoing expenses associated with hosting and supporting the program.
• Restricted Filtering and Specialized Features: TensorFlow Lite might not provide sophisticated filtering choices or features tailored to particular applications, such picture identification or natural language processing.
• Learning Curve: It could take a lot of time and effort to become proficient with TensorFlow Lite due to its high learning curve.
• Security and Compliance Issues: TensorFlow Lite's open-source design may give rise to security and compliance issues, particularly for businesses in regulated sectors.

4.pickle:

Python Pickle Module Overview

• Implements binary protocols for serializing and de-serializing Python object structures.
• "Pickling" converts Python object hierarchy into a byte stream.
• Also known as "serialization", "marshalling", or "flattening."
• "Unpickling" reverses this process, converting a byte stream back into an object hierarchy.

5:Dockers + Kubernetes

Benefits: 

• Makes it possible for machine learning models to be deployed effectively and scalable, which facilitates quick and precise inference.
• Ensures interoperability with a wide range of contexts by supporting a wide range of platforms and devices.
• Offers a package that is adaptable and configurable, allowing users to customize the model to meet their own requirements.
• Its the most preffered method of deploying pipelines in commercial applications.

Drawbacks:

• To handle sophisticated models, more processing power might be needed, which could affect performance.
• Limited streaming and real-time processing functionality, which may limit its applicability in some applications.
• Requires more resources and infrastructure, which raises the cost and complexity.

Deployment: Letting the ship sail the tides

Most low-hanging fruits have been picked. What is left takes more effort to build, hence fewer people can build them.People have realized that it’s hard to be competitive in the generative AI space, so the excitement has calmed down.

In 2023, the layers that saw the highest increases were the applications and application development layers. The infrastructure layer saw a little bit of growth, but it was far from the level of growth seen in other layers

                                                                                                                  -Chip Huyen

                                       Author of Designing ML Systems.

A model is practically useless if its implementation and effectiveness is only limited to small scale simulations, in real world application models often have to work with thousands or even millions of concurrent users, these users dont just expect accuracy of the model but also a good user experience , hence to deploy the models in real life applications  frameworks which preserve the utility of the model in terms of performance and computational latency, but also provide good graphical user interface to interact with the model and gain insights.

Given below  are some commonly  used libraries involved in  deployment:

1.TensorFlow Extended (TFX):

Benefits of TensorFlow Extended (TFX): 

• All-inclusive tool for overseeing the complete machine learning lifecycle, including model deployment.
• Allows for a variety of deployment strategies, including batch inference, real-time streaming, and REST API providing.
• Connects to well-known machine learning frameworks such as scikit-learn, PyTorch, and TensorFlow.

Drwabacks: 

• Limited built-in capabilities for data versioning and administration.
• For advanced model serving scenarios, integration with external tools can be necessary.
• Steep learning curve for teams that are unfamiliar with TFX principles and APIs.
  2.Kubeflow:

Benefits of Kubeflow:

• All-inclusive platform for handling the model deployment phase of the machine learning lifecycle.
• Allows for a variety of deployment strategies, including batch inference, real-time streaming, and REST API providing.
• Connects to well-known machine learning frameworks such as scikit-learn, PyTorch, and TensorFlow.

Drawbacks:

• Limited built-in capabilities for data versioning and administration.
• For advanced model serving scenarios, integration with external tools like can be necessary.

3.ML Flow:

Advantages:

• Convenient Experiment Monitoring: MLflow makes it simple to monitor experiment parameters, metrics, and artifacts, which facilitates model comparison and replication.
• Standardized Model Packaging: MLflow offers a uniform format for ML model packaging, which simplifies the deployment of models in various contexts.
• Centralized Model Registry: Tracking and deploying models is made simpler by MLflow's model registry, which offers a centralized repository for maintaining ML models.
• Reproducible Pipelines: Complex ML workflows are easier to organize and carry out when reproducible pipelines are enabled by MLflow.
• Open Source and Expandable: MLflow is an open-source project that may be expanded and customized to suit certain requirements.
• Integration with Well-Known products: MLflow is easier to use in current workflows since it interfaces with well-known products like Databricks, Neptune, and DAGsHub.

Drwabaks:

• Limited Multi-User Support: Working together on experiments is challenging because MLflow lacks a multi-user environment.
• Restricted Role-Based Access Control: The lack of role-based access control in MLflow makes it challenging to regulate who has access to experiments and models.
• Limited Advanced Security capabilities: Because MLflow has few advanced security capabilities, security problems can arise.
• Restricted allowance  for Real-Time Model Endpoints: It is challenging to deploy models in real-time using MLflow since it does not allow real-time model endpoints.
• Restricted Support for Online and Offline Store Auto Sync: MLflow does not support online and offline store auto synchronization, which makes managing models across various endpoints with roboflow a  challenging task.

4.Vortex AI:

Vortex AI is deployment tool framework devloped by google and has the optionality of runining  on the servers of Google Cloud. Some advantages and disadvantages are listed below:

Benefits

• Machine Learning Algorithm: Vertex AI offers a machine learning algorithm that facilitates work automation, increases productivity, and sharpens judgment.
• Google Cloud integration: Vertex AI offers a comprehensive platform for managing machine learning models and data, integrating with Google Cloud with ease.
• Advanced Features: Support for different frameworks and languages, auto-synching online and offline stores, real-time model endpoints, and other advanced features are provided by Vertex AI.
• Scalability: Vertex AI has a high degree of scalability, making it possible to analyze massive amounts of data and models effectively.
• Cost-Effective: Vertex AI is reasonably priced, offering a pay-as-you-go pricing structure that permits flexible spending plans.

Drawbacks:

• Limited Multi-User Support: Working together on experiments and models is challenging because Vertex AI does not support multi-user setups.
• Restricted Role-Based Access Control: The absence of role-based access control in Vertex AI makes it challenging to govern who has access to data and models.
• Limited Advanced Security Features: Vertex AI is susceptible to security threats since it lacks advanced security features.
• Restricted Support for Real-Time Model Endpoints: It is challenging to deploy models in real-time since Vertex AI does not support real-time model endpoints.
• Limited Assistance for Stores Both Online and Offline  Sync :Vertex AI's lack of support for online and offline store auto sync makes it challenging to manage models in various situations.

5.AWS Sagemaker:

ML deployment and inference service provided by Amazon Web Services and managed using their cloud infrastructure

Benefits:

• Faster Time-to-Market: SageMaker makes machine learning projects more productive by enabling developers to create, train, and deploy models more quickly.
• Built-in Frameworks and Algorithms: TensorFlow, PyTorch, and MXNet are just a few of the many built-in machine learning frameworks and algorithms that SageMaker offers, making it simpler to get started.
• Automated model tuning function:SageMaker's automated model tuning function improves hyperparameters to enhance model performance while requiring less time and effort.
• Ground Truth Labeling Service: SageMaker's Ground Truth service expedites data preparation by assisting customers in accurately and rapidly labeling data.
• Support for Reinforcement Learning: SageMaker comes with built-in support for reinforcement learning, making it simple for users to create and train reinforcement learning models.
• Elastic Inference: SageMaker's Elastic Inference feature allows users to attach GPU acceleration only when needed, reducing the overall cost of GPU usage.
• Built-in Model Monitoring: SageMaker continuously monitors models in production and alerts users to any performance issues, helping ensure optimal model performance.

Drawbacks:

• Complexity: Despite having an easy-to-use interface, machine learning is still a complicated science, and using SageMaker efficiently may need a high level of expertise in the subject.
• Vendor Lock-In: SageMaker's close integration with other AWS services makes it challenging to move to another cloud provider, which can lead to vendor lock-in with AWS.
• Cost: Even with SageMaker's pay-as-you-go pricing structure, machine learning workloads on the platform can still be expensive, particularly for large-scale initiatives.
• Limited Customization: Although SageMaker comes with a large number of pre-built frameworks and algorithms, it might not be able to satisfy every particular project's needs, necessitating the creation of unique solutions.

6.Hugging Face inference API+ Gradio frontend GUI :

• Hugging Face: Provides pre-trained language models and NLP tools for customer service, marketing, and healthcare.
• Gradio: A tool for creating interactive demos and interfaces for machine learning models. • Both tools offer a simple, intuitive deployment process and user engagement.
• Together, they enable developers to create engaging applications showcasing machine learning capabilities.

Benefits of Hugging Face Inference API with Gradio Integration:

• Simplicity of Use: Only a few lines of code are needed to quickly deploy machine learning models.
• Seamless Integration: Models from the Hugging Face Model Hub can be easily deployed without requiring complex infrastructure
• Quick Deployment: Gradio demos don't require you to set up your own hosting infrastructure
• Collaborative Development: Enables several people to share models and demos while working on the same workspace
• Customization: Complex, interactive demos are made possible by a high degree of customization
• Support for Several Models: Asteroid, SpeechBrain, spaCy, and Transformers are all supported
• Support for Various Model Types: Includes support for text-to-speech, speech-to-text, and image-to-text models
• Support for Custom Model Checkpoints: In the event that your model is not supported by the Inference API, it still provides custom model checkpoints.

Disadvantages of using Gradio with HF

• Limited Customization of Spaces: Gradio's flexibility may not be fully utilized in Hugging Face Spaces
• Potential Performance Limitations: The complexity of the model and traffic to the Space may lead to performance or scalability issues
• Vendor Lock-In: The platform may introduce vendor lock-in, making migration difficult
• Learning Curve: New users may face a learning curve
• Limited Support for Advanced Features: Gradio may not support all advanced features or customizations.

Implementations of Research Papers and ML Books to refer:

Paper implementations are a good way to practice our skills as they help us in validating what what we learned in theory and gain an intuitive sense of it , and offer the real unedited views of the author and thought process that lead them to arrive at that conclusion ,so that we understand these models  better to resolve issues that arise in real life deployments. Some of the great papers which can be implemented without significant investment in hardware or servers are:

1. Implementation of Singular Value Decomposition in Python from Scratch:some text
   1. Code
   2. Reference Book (Excerpt of the book , chp-4 of CS-theory-infoage,CMU)
2. Implementing XG Boost form scratchsome text
   1. Paper
   2. Code Implementation
   3. Reference Book
3. Implementing Neural Networks from Scratch:some text
   1. Paper
   2. Code Implementation
   3. Reference Book (only available on author's website,free youtube playlist of the same)
4. Implementation of Neural Net Classifiers , AlexNet (paper):some text
   1. Using TensorFlow and Keras
   2. Using Pytorch
   3. Implementing from Scratch
5. Implementation of Segment wise Multi-Step parallel calculation based RNN (SegRNN): Applications in forecasting of time series data.(paper)(code)
6. Attention is all you need (paper) : The paper that conceptualized Transformerssome text
   1. Code Implementation -(from scratch in python) (using Pytorch)

Next Station: Implemetatation,Iteration and Creation

We know less than we think
The replication crisis is not an aberration,many of the things we believe are wrong, we are often not even asking the right questions,
But we can do more than we think.
We are tied down by invisible orthodoxy of our self-doubts and limits, but in reality, the laws of physics are the only limit.
It's important to do things fast, as we get to  learn more per unit time because we make contact with reality more frequently.

- Nat Friedman

(VC, Entrepreneur, Former CEO of Github)

Every developer, enthusiast faces the fear of joining a competition or creating an actual project on their own, and might get stuck in the loop of jumping from endless cycles of course videos and courses and put up certificates  which would bring no joy, we do have to learn certain basic frameworks before jumping on to it, but at a certain point we have to get ourselves out of the comfort zone to do actual coding, it could be through contributing in ML based repos Implementing, iterating, and creating are key steps in the journey of becoming proficient in machine learning. While it is important to learn the fundamentals and gain knowledge through courses and tutorials, true mastery often comes from hands-on experience. Happy coding and exploring!

Author

This article was written by Sahil Shenoy, and edited by our writers team.