How to build an AI system?

AI...AI...AI, heard that term somewhere? Well, ofcourse, the hype around artificial intelligence has entered every intellect. From voice-activated smart speakers in our living rooms to sophisticated algorithms managing global financial markets, AI(including Artificial Narrow Intelligence) is seamlessly integrating into every aspect of our lives. With the hype also comes the curse of mystification around the subject, is AI just a black box or can we understand its true potential to build amazing systems? In this article we will deal with just this, understanding AI and how to build one.

Artificial Intelligence

The simplest definition of AI is "making machines do what human intelligence does", as simple as that. Everything we humans do by instinct or learn to do as we age can be done by a machine. Well how does a machine learn? Either by observation - (terminology geeks call it "Reinforcement Learning") - or by finding patterns in data - (again it is called supervised or unsupervised).

For the aforsesaid geek, the formal definition of AI is the field of computer science that focuses on the creation of intelligent machines that can perform tasks that require human intelligence. These tasks include problem-solving, speech recognition, decision-making, and learning.

With these techniques what has been done? Well everything from ChatGPT to Tesla, has these very techniques at its basis with varying depths of knowledge. AI includes many interdisciplinary fields, Natural Language Processing, Computer Vision and so on, but for this article we will focus on AI in general.

Types of AI

AI can be categorized based on capability and functionality.

Types of AI based on capability

Artificial Narrow Intelligence
Artificial Narrow Intelligence is designed for a specific task or a narrow set of tasks. It can’t perform tasks beyond its programmed capabilities. Examples include virtual personal assistants like Siri or Alexa, chatbots, and function.

General AI or Artificial General Intelligence
AGI, often called strong AI, can understand, learn, and apply knowledge across a wide range of tasks, similar to human intelligence. AGI doesn’t exist yet, and creating it is a long-term goal in AI development.

Artificial superintelligence (ASI)
This is a hypothetical form of AI that surpasses human intelligence in every aspect. ASI would be capable of solving complex problems, learning instantly, and achieving tasks that humans can’t even comprehend. It’s purely theoretical and a topic of debate among experts.

Types of AI based on functionality

Reactive Machines
These AI systems are designed to perform specific tasks and do not possess memory or the ability to learn from experience. They follow predefined rules. Examples include chess-playing programs.

Limited Memory
‍AI systems in this category can consider past experiences to some extent. They use historical data to make decisions but are not true learning systems. Self-driving cars use this type of AI to navigate.

Theory of Mind
These are hypothetical AI systems that can understand and model human emotions, beliefs, intentions, and consciousness. They don’t exist yet but are a topic of research.

Self-aware AI
Also being a hypothetical concept, self-aware AI would have human-like consciousness and self-awareness. It’s more of a philosophical idea and not a practical AI type at the moment.

The traditional AI Landscape: Steps of building an AI system

Now that we have a solid understanding of AI and the key techniques involved, let's dive into how to build an AI system. Creating an AI solution usually involves these main steps:

1. Problem Definition and Data Collection
   First, clearly define the problem you want the AI system to solve. Understand the project's specific needs, constraints, and goals. Once the problem is well-defined, gather or create the data needed to train and test the AI model. This data should be relevant, diverse, and truly reflect the problem area.

2. Data Preprocessing
   The collected data often needs some preparation before it’s ready for machine learning algorithms. This includes cleaning the data to remove any errors and inconsistencies, dealing with missing values, and performing feature engineering—creating or extracting important features from the raw data to enhance the model’s performance.

3. Model Selection and Training
   Next, choose the right machine learning algorithm or model for the problem. You might select an existing model or design a new one. Then, train the model using the preprocessed data. During this phase, the model learns patterns and relationships in the data to make accurate predictions or decisions.This is a critical part of model training.

4. Model Evaluation and Optimization
   After training, evaluate how well the model performs using various metrics and techniques. This helps you spot areas for improvement and potential issues, like overfitting (when the model memorizes the training data instead of learning general patterns). Based on this evaluation, tweak the model and its parameters to boost performance.

5. Deployment and Monitoring
   Once the model meets your performance standards, deploy it by integrating it into a production environment or application. Even after deployment, keep an eye on the model's performance and set up feedback loops to collect new data. This ongoing process helps continually improve the model over time.

The Modern AI Landscape: API Requests, Not Development

Today, when people talk about building AI, they often mean utilizing existing AI models and adapting them for specific tasks through API requests, rather than developing AI models from scratch. This is because developing a completely novel AI model is an extensive process, requiring significant expertise and computational resources.

The current AI landscape is dominated by large language models, such as ChatGPT or Cohere's Command model, which are trained on massive amounts of text data and can generate human-like responses. These models are made accessible to developers and businesses through APIs, allowing for easy integration into various applications.

For example, a business may want to create a chatbot for its website. Instead of developing an AI model from the ground up, they can utilize an existing large language model through an API. They would provide the model with specific training data related to their business, fine-tuning the model to respond accurately to customer inquiries. This way, they can quickly deploy a sophisticated chatbot without needing to develop AI from scratch.

This shift towards API requests has lowered the barrier to entry for businesses and developers looking to incorporate AI into their products. It has also led to a focus on data collection and model fine-tuning, as the quality of the data used to train and adapt these large language models is critical to their performance.

Traditional AI v/s Modern AI

Example- traditional programming

lets build an AI-Powered Customer Service Chatbot from Scratch( a generative ai example), an intelligent system, we will be using the customer support ticket dataset from kaggle-

• Load the datase
  import pandas as pd
Load the dataset
file_path = 'path_to_your_dataset/twcs.csv'
print(df.head())

• Use NLTK for text preprocessingimport re
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download('stopwords')
nltk.download('punkt')

def clean_text(text):
    text = re.sub(r'http\S+', '', text)
    text = re.sub(r'[^a-zA-Z\s]', '', text)  
    text = text.lower()  
    tokens = word_tokenize(text)  
    stop_words = set(stopwords.words('english'))  
    cleaned_tokens = [word for word in tokens if word not in stop_words]
    return ' '.join(cleaned_tokens)
df['cleaned_text'] = df['text'].apply(clean_text)
print(df[['text', 'cleaned_text']].head())

• Split the Dataset
  
  from sklearn.model_selection import train_test_split
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
train_df, val_df = train_test_split(train_df, test_size=0.2, random_state=42)
print(f"Training set size: {len(train_df)}")
print(f"Validation set size: {len(val_df)}")
print(f"Test set size: {len(test_df)}")

• Build the model- Lets keep it simple a 4 layer model with 2 hidden layers
  
  import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=128, input_length=max_length))
model.add(LSTM(128, return_sequences=True))
model.add(LSTM(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=10, validation_data=(X_val, y_val))

• Evaluate the model
  
  loss, accuracy = model.evaluate(X_test, y_test)
print(f'Test Accuracy: {accuracy * 100:.2f}%')

• Deploy it
  
  from flask import Flask, request, jsonify
‍
app = Flask(name)

@app.route('/predict', methods=['POST'])
def predict():
    data = request.get_json(force=True)    
    prediction = model.predict(data['input'])
    return jsonify({'prediction': prediction})
if name == 'main': app.run()

Example the modern way

phew..build the model, train it, fine-tuning hyperparameters, then do it again and again and again! And still the accuracy isn't what we need! Plus the amount of resources it consumes!

This is what a modern developer and mid level organizations with less resources would think, why not then use someone else's model! The development process and the training process is very resource consuming! (Just look at the cost of one Graphics Processing Unit)

It requires vast amounts of hardware resources, which is often not the business goal and hampers the business operations. APIs on the other hand provide models at an hourly rate! This brings down the development costs of AI-integrated systems, with just an additional training of these pre-built models companies can have tremendous cost savings.

Lets see how to do it the modern way

• Load the dataset
  import pandas as pd
file_path = 'path_to_your_dataset/twcs.csv'
df = pd.read_csv(file_path)
# Display the first few rows of the dataset
print(df.head())

• Preprocess the dataset
  import re
def clean_text(text):
    text = re.sub(r'http\S+', '', text)
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    text = text.lower()
    return text
df['cleaned_text'] = df['text'].apply(clean_text)
print(df[['text', 'cleaned_text']].head())

• Use a pretrained model - we will use a small DialoGPT by MICROSOFT using the transformers library
  
  from transformers import AutoModelForCausalLM, AutoTokenizer import torch
‍
Load pre-trained model and tokenizer

model_name = "microsoft/DialoGPT-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
def generate_response(input_text):
inputs = tokenizer.encode(input_text + tokenizer.eos_token, return_tensors='pt')      
outputs = model.generate(inputs, max_length=1000,
pad_token_id=tokenizer.eos_token_id) response = tokenizer.decode(outputs[0],
skip_special_tokens=True)
return response

input_text = "Hello, I need help with my order."
response = generate_response(input_text) print(f"Input: {input_text}") print(f"Response: {response}")

‍

Congratulations, with just a few lines of code the models ready!!

Applications of AI

Computer vision

Computer vision enables machines to interpret and understand visual information from images or videos. It’s used in facial recognition, object detection, and even self-driving cars.This is a prime example of image recognition.

API Calls/Libraries:

• OpenCV: An open-source library for computer vision and image processing.
• TensorFlow/Keras: For building and training custom models for image recognition.
• PyTorch: Another popular deep learning library for computer vision tasks.
• Google Vision API: Offers powerful image analysis capabilities, including object detection, image labeling, and text recognition.
• AWS Rekognition: Provides facial analysis, object detection, and scene detection.

Customer service

AI-powered customer service uses chatbots and virtual agents to assist customers with inquiries, troubleshooting, and support. It’s available 24/7 and can handle routine customer interactions efficiently.

Take LTS Group’s Personal Document Recognition and Information Extraction System for instance. The system utilizes identification rules and image processing technology through Google Vision to automatically recognize and extract content from personal documents. This ensures high accuracy and the ability to process a large volume of documents efficiently.

API Calls/Libraries:

• Dialogflow (Google): A platform for building conversational interfaces and chatbots.
• Amazon Lex: For creating conversational interfaces with voice and text.
• Microsoft Bot Framework: Provides tools and services to build and deploy chatbots.
• Hugging Face Transformers: For integrating pre-trained language models in chatbot applications.

Virtual assistant

Virtual assistants like Siri, Alexa, and Google Assistant use NLP to perform tasks, answer questions, set reminders, and control smart devices.

For example, LTS Group has developed and deployed an AI-powered Travel Chatbot for our business client in the tourism industry. It provides details on popular destinations, itineraries, activities, and tailored travel recommendations. It also offers information on hotels, restaurants, shopping, and allows for online reservations and pricing comparisons.

API Calls/Libraries:

• Amazon Alexa Skills Kit: For developing voice-driven capabilities for Alexa.
• Google Assistant SDK: For integrating Google Assistant into devices and applications.
• Microsoft Azure Bot Service: For building intelligent bots.
• Rasa: An open-source framework for building conversational AI.

Predictive analysis

Predictive analysis uses historical data and AI algorithms to forecast future trends or outcomes. It’s used in various fields, including finance for predicting stock prices and in healthcare for disease outbreak predictions.

API Calls/Libraries:

• scikit-learn: For implementing machine learning algorithms and predictive models.
• TensorFlow: For building deep learning models.
• AWS Forecast: A service for time-series forecasting.
• Azure Machine Learning: For building and deploying predictive models.

Recommendation

Recommendation systems use AI to suggest products, content, or services to users based on their preferences and behavior. They help users discover new things they might like, such as movie recommendations on streaming platforms or product suggestions on e-commerce websites.

API Calls/Libraries:

• TensorFlow Recommenders: A library for building recommendation systems.
• Surprise: A Python scikit for building and analyzing recommender systems.
• Hugging Face Transformers: For implementing NLP-based recommendation systems.
• AWS Personalize: A service for creating individualized recommendations.

Personalization

Personalization tailors content, services, or product recommendations to individual user preferences. It enhances user experiences, like personalized newsfeeds and product offerings.

API Calls/Libraries:

• TensorFlow: For custom models that personalize content.
• scikit-learn: For implementing machine learning algorithms for personalization.
• Google Recommendations AI: For providing personalized recommendations.
• Azure Personalizer: A service for creating rich personalized user experiences.

Fraud prevention

AI is employed to detect and prevent fraudulent activities, such as credit card fraud and identity theft, by analyzing transaction patterns and user behavior to identify anomalies.

API Calls/Libraries:

• H2O.ai: For building machine learning models for fraud detection.
• TensorFlow: For deep learning-based anomaly detection models.
• Amazon Fraud Detector: A service for identifying fraudulent online activities.
• Azure Fraud Detection: For building and deploying fraud detection models.

Data mining

Data mining uses AI and ML models to discover patterns and insights in large datasets, or big data. AI and ML’s scalability, pattern recognition, automation, speed, and generalization capabilities make them essential for extracting insights and making data-driven decisions from big data, where manual analysis might be impractical or inefficient.

API Calls/Libraries:

• RapidMiner: An open-source platform for data science and machine learning.
• KNIME: For data analytics, reporting, and integration platforms.
• PySpark: A Python API for Spark, used for big data processing.
• scikit-learn: For implementing various data mining algorithms.

List of Libraries/Tools - Emerging Technologies

Let us look at some of the tools for building AI systems,

Hugging Face

Hugging Face has become a central hub for modern AI development, offering a plethora of tools and libraries designed to make working with large language models and other AI technologies straightforward and efficient.

• Transformers: A library for state-of-the-art NLP models like BERT, GPT-3, T5, and more. It simplifies the process of using these models with pre-trained weights and allows for easy fine-tuning.
• Datasets: A toolkit for accessing and processing large datasets, enabling efficient handling of massive amounts of data necessary for training and evaluation.
• Tokenizers: Fast and customizable tokenization library that supports various tokenization algorithms, essential for preparing text data for model training.
• Accelerate: A library to simplify and speed up training and inference on various hardware accelerators.
• Hub: A platform for sharing and accessing pre-trained models, datasets, and tokenizers, fostering a collaborative environment for the AI community.

LangChain

LangChain is a framework specifically designed for building applications with language models. It focuses on creating chains of calls to language models and other computational modules, facilitating the development of complex applications using these models.

• Integration: Seamless integration with various language models, including those from Hugging Face, OpenAI, and Cohere.
• Modularity: Provides modular components to build and manage language model pipelines, making it easy to customize and extend functionalities.
• Data Augmentation: Tools for augmenting training data to improve model performance in specific tasks.

Cloud-Based AI Solutions- Cloud Computing and Cloud Platforms

1. Azure AI StudiAzure Machine Learning: Comprehensive service for building, training, and deploying machine learning models.

1. Cognitive Services: Pre-built APIs for computer vision, speech recognition, language understanding, and decision-making.
2. Bot Service: Tools to build, test, and deploy chatbots across multiple channels.
3. AWS AI and Machine Learning
4. • SageMaker: Fully managed service to build, train, and deploy machine learning models at scale.
   • Rekognition: Image and video analysis service for object detection, facial recognition, and more.
   • Comprehend: NLP service to uncover insights and relationships in text.
   • Polly: Text-to-speech service for creating lifelike speech.
   • Lex: Service for building conversational interfaces using voice and text.
5. Google Cloud AI
6. • Vertex AI: Unified AI platform for building, deploying, and scaling ML models.
   • AutoML: Suite of products for training high-quality custom machine learning models with minimal effort.
   • Cloud Natural Language: Tools for analyzing and understanding text, including sentiment analysis and entity recognition.
   • Cloud Vision: Image analysis service that provides powerful insights into images.

Traditional Libraries

1. NLTK (Natural Language Toolkit)
2. • A comprehensive library for working with human language data (text). It provides easy-to-use interfaces to over 50 corpora and lexical resources, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.
3. OpenCV (Open Source Computer Vision Library)
4. • An open-source computer vision and machine learning software library. It provides a common infrastructure for computer vision applications and accelerates the use of machine perception in commercial products. Includes modules for image processing, video capture, and analysis (e.g., object detection, face recognition).

These libraries and tools represent a modern approach to AI and machine learning, emphasizing ease of use, integration capabilities, and access to pre-trained models and datasets. They cater to both newcomers and experienced developers, allowing for rapid development and deployment of sophisticated AI applications.

Building an AI

AI systems are generally created by a blend of machine learning, deep learning, and natural language processing techniques. Learning these methods and where they are used, are fundamental to understanding the power and restriction of AI.

Machine Learning (ML)

ML algorithms automatically process the dataset and find out patterns and insights to build the prediction by themselves, that means the system need not to be told detail instructions of any prediction or decision.

Types of Machine Learning Techniques:

Supervised Learning : The algorithm is given a set of labeled examples, meaning the answer to each question is already present, the algorithm tries to find a function for which the error is the minimum, meaning the function which gets most of the answers right.

Unsupervised Leaning: The data is not labelled, meaning the task of the algorithm is to find patterns in the data, it could be correlations between different features.

Reinforcement Learning: As mentioned earlier, RL is like a child learning by observing the environment, an "agent" is left free in an environment with the sole purpose of maximizing a function, and it learns eventually the best way to do it.

Deep Learning

Deep Learning is a set of machine learning approaches based on learning data representations, in contrast to task-specific algorithms....Deep learning also involves neural networks, a very popular kind of machine learning technique, but really it just refers to the use of a lot of computational power to learn at a higher level. Deep learning neural networks are those neural networks that mimic the human brain and consist of interconnected 'neurons' arranged in layers that learn and extract different levels of abstraction from the data. It tries to function like a human brain

Natural Language Processing

Natural language processing is the field of AI that focuses on the interaction between computers and human language. NLP enables machines to understand, interpret, and generate human language, including text and speech. This involves tasks such as language translation, sentiment analysis (determining the emotional tone of text), named entity recognition (identifying important entities in text), and question-answering systems.

Computer Vision

Computer Vision is a fascinating branch of Artificial Intelligence that allows machines to see and understand the world around them. Imagine your smartphone recognizing your face to unlock, or a car driving itself by identifying roads and obstacles—these are real-world examples of computer vision at work. This technology is revolutionizing how we live and work, making everyday tasks smarter and more efficient. In this article, we'll dive into the basics of computer vision, explore its various applications, and guide you through creating a simple computer vision system yourself.

Programming Languages For AI Development

let’s learn some basics the technology stack used in building AI-based systems

Python

Python is the go-to language for AI due to its readability and rich ecosystem of AI tools. For instance, you can use libraries like scikit-learn for machine learning or TensorFlow for deep learning. Its simplicity makes it accessible for beginners, and it’s widely used in fields like natural language processing (NLP), computer vision, and reinforcement learning.

Julia

Julia is a rising star in the AI world. It’s designed for data science and offers a balance between speed and ease of use. An example is Flux.jl, a Julia library for building neural networks. Julia’s speed makes it ideal for high-performance computing, which is essential in tasks like simulating complex systems or conducting large-scale data analysis.

R

R, although less user-friendly than Python, has a strong presence in the statistical and data analysis community. It’s excellent for tasks like data visualization and statistical modeling. Packages like ggplot2 and dplyr are widely used in data science for creating insightful visualizations and data manipulation.

Artificial Intelligence Job Profiles

According to the job site Indeed, the demand for AI skills has more than doubled over the past three years, and the number of job postings is up by 119 %. This Artificial Intelligence Tutorial will be incomplete without the different Jobs Profiles. So, if Artificial Intelligence appeals to you and you want a Job in the AI field, then here are the different Job Profiles you can apply for if you have AI Skills.

1. Machine Learning Engineer

Machine learning engineers are sophisticated programmers who develop machines and systems that can learn and apply knowledge without specific direction. Artificial intelligence is the goal of a machine learning engineer, particularly in the domain of Artificial General Intelligence.

They are computer programmers, but their focus goes beyond specifically programming machines to perform specific tasks. They create programs that will enable machines to take actions without being specifically directed to perform those tasks and earn a whopping $111,490 per annum.

2. Data Scientist

Data scientists are those who crack complex data problems with their strong expertise in certain scientific disciplines. They work with several elements related to mathematics, statistics, computer science, etc.

The data scientist role is a position for specialists. You can specialize in different types of skills like speech-analytics, text analytics (NLP), image processing, video processing, medicine simulations, material simulation, etc. Each of these specialist roles is very limited in number and hence the value of such a specialist is immense with an average Salary of $91,470.

3. Artificial Intelligence Engineer

An artificial intelligence engineer works with algorithms, neural networks, and other tools to advance the field of artificial intelligence in some way. Engineers may also choose between projects involving weak or strong artificial intelligence, where different setups focus on different capabilities. the salary for an AI Engineer is around $105,244.

4. Business Intelligence Developer

A Business Intelligence developer spends a lot of time researching and planning solutions for existing problems within the company. The Business Intelligence Developer is responsible for aggregating data from multiple sources in an efficient data warehouse and designing enterprise-level solutions for very large multidimensional databases.

Business intelligence developers play a key role in improving the efficiency and profitability of a business. It’s a career that’s in high demand and commands an annual median salary of $92,278.

5. Research Scientist

Research scientists are responsible for designing, undertaking and analyzing information from controlled laboratory-based investigations, experiments, and trials. You could work for government laboratories, environmental organizations, specialist research organizations or universities and earn an average salary of $105,666.

6. Big Data Engineer/Architect

Big data engineers and architects have among the best paying jobs in artificial intelligence. In fact, they command an annual median salary of $151,307.

The Big Data solutions architect is responsible for managing the full life-cycle of a Hadoop solution. This includes creating the requirements analysis, the platform selection, design of the technical architecture, the design of the application design and development, testing, and deployment of the proposed solution.

Author

This article was written by Zohair Badshah, a former member of our software team, and edited by our writers team.