Generative AI — The future of Engineering
On June 22, 2024, I had the privilege of presenting a master class at the Congrès National des Junior-Entreprises de la Confédération des Junior-Entreprises Marocaines. This event brought together some of the brightest minds and most enthusiastic learners eager to explore the potential of Generative AI and its impact on the future of engineering.
Master class at CJEM — Hamza BARBOUCH
Introduction to Generative AI
Generative AI represents a significant leap forward in artificial intelligence, designed to understand and generate human language. These advanced AI systems, such as GPT-3 and GPT-4, are capable of performing a wide range of tasks, from generating text and answering questions to translating languages and creating code. The foundation of these models lies in their training on vast amounts of data, allowing them to learn patterns and produce human-like responses.
The example below illustrates the remarkable progression of an LLM model, evolving from a novice akin to a toddler to a highly sophisticated entity comparable to Einstein.
Illustration of how LLMs gets better in the learning process
GPT-3 was trained on an extensive corpus of 45TB. To put that into perspective, a 1GB text file contains approximately 178 million words. By this calculation, the training data for GPT-3 encompassed an astounding number of words, highlighting the sheer scale and depth of its training process.
Mindsets Towards AI
The integration of AI into various aspects of our lives and work has elicited a range of mindsets. These can generally be categorized into three types:
Denial: Some believe that AI cannot perform their job, underestimating the capabilities of modern AI systems.Panic: Others fear that AI will take over their jobs, leading to anxiety and resistance towards adopting new technologies.Positive: The most constructive mindset is viewing AI as a tool to enhance and improve one’s skills and productivity. Embracing AI can lead to new opportunities and career growth.Negative & Positive mindsets
The Foundation: How It All Started
The journey of Generative AI began with foundational research, particularly the groundbreaking paper “Attention is All You Need.” This paper introduced the Transformer architecture, which has become the cornerstone of modern AI models. The evolution of Generative AI has been fueled by the diversity, volume, and quality of training data, encompassing books, research papers, web pages, code repositories, public datasets, social media, and Wikipedia.
Attention is All You Need paper — Transformers Architecture
Hardware Evolution
The advancements in Generative AI would not have been possible without significant progress in hardware. The development of powerful GPUs and TPUs has enabled the training of large models on enormous datasets. The cost of training these models, such as the $100 million spent on training GPT-4, reflects the immense computational resources required.
Hardware evolution players
The Architecture of Transformers
At the heart of Generative AI is the Transformer architecture, which comprises several key components that work together to process and generate text. This architecture allows for parallel processing of data, making it more efficient than previous models. Transformers have paved the way for the development of various AI models, including BERT, GPT-2, T5, and the latest versions like GPT-3 and GPT-4.
Transformers architecture components — Encoder & DecoderEncoder-Decoder Structure: The transformer model uses an encoder to read input data (such as text) and a decoder to produce the output. The encoder and decoder each consist of multiple layers.Self-Attention Mechanism: This mechanism allows the model to weigh the importance of different words in a sentence. It helps in understanding context by considering the relationships between words, regardless of their position in the sequence.Multi-Head Attention: By using multiple attention heads, the transformer can focus on different parts of the input sequence simultaneously. This enhances the model’s ability to capture various aspects of the data.Feed-Forward Neural Networks: Each layer in the encoder and decoder contains a feed-forward neural network that processes the output of the attention mechanisms. These networks help in further transforming the data.Positional Encoding: Since transformers do not have a built-in sense of word order, positional encoding is added to the input embeddings to give the model information about the position of each word in the sequence.Layer Normalization and Residual Connections: These techniques help in stabilizing and speeding up the training process. Layer normalization ensures that the output of each layer has a consistent scale, while residual connections allow gradients to flow more easily through the network.
The Emergence of LLMs
Large Language Models (LLMs) have evolved from small models to the behemoths we see today. The progression from BERT to GPT-4 showcases the rapid advancements in model size and capabilities. These models differ in their architecture and primary use cases, ranging from natural language processing tasks to text generation and summarization.
Model LLMs — Open source & Paid
Tools for Leveraging Generative AI: Ollama and Hugging Face
In addition to understanding the architecture and training of Generative AI models, it is essential to be familiar with the tools that facilitate their application. Ollama is a versatile platform that allows you to run your favorite Large Language Models (LLMs) locally, offering flexibility and control over model usage. On the other hand, Hugging Face is a renowned hub for AI models and machine learning tools. It provides an extensive library of pre-trained models and APIs, making it easier for developers to implement and experiment with cutting-edge AI technologies. Both platforms are invaluable resources for those looking to harness the power of Generative AI effectively.
Ollame tool to run LLMs locallyHuggingface — The GitHub like community to download models & datasets
Effective Use of Generative AI
To harness the full potential of Generative AI, one must understand the principles of prompt engineering. Designing and refining prompts to guide the outputs of large language models is crucial. Here are some best practices for effective prompt engineering:
1 — Know the Model’s Limitations: Understand what the model can and cannot do. This helps in setting realistic expectations and crafting prompts that play to the model’s strengths.
2 — Aim for Maximum Clarity: Clear and unambiguous prompts lead to better results. Avoid vague language and be explicit about what you want the model to do.
3 — Be Very Specific and Explicit: The more detail you provide in your prompt, the more accurately the model can generate the desired output. Specify the format, style, and any other relevant details.
4 — Balance Simplicity and Complexity: Simple prompts may not capture all nuances, while overly complex prompts can confuse the model. Strive for a balance that provides enough detail without being overwhelming.
5 — Give the Model Examples: Providing examples of the desired output can help guide the model. This technique, known as few-shot learning, involves giving the model a few examples to mimic.
6 — Iterate and Experiment: Prompt engineering is an iterative process. Experiment with different phrasings and structures to see what works best. Learn from each iteration to refine your prompts.
7 — Use Contextual Information: Including context in your prompts can help the model generate more relevant responses. This could involve providing background information or specifying the scenario in which the model is to generate text.
8 — Incorporate Feedback: Use feedback from the model’s responses to improve your prompts. This could involve adjusting the level of detail, changing the phrasing, or providing additional examples.
By mastering these techniques, users can effectively leverage Generative AI to produce high-quality, relevant outputs tailored to their specific needs.
The Future of Generative AI
Looking ahead, the future of Generative AI is incredibly promising. The continuous advancements in data availability, algorithm development, and hardware capabilities will further enhance the performance and applications of these models. Generative AI is poised to revolutionize various industries, including customer service, human resources, education, entrepreneurship, and scientific research.
Future of Generative AI
Conclusion
The master class at the Congrès National des Junior-Entreprises was an enriching experience, filled with insightful discussions and enthusiastic participation. I am grateful to the dream team at Maltem Africa —Hamza BARBOUCH, Omar BENMOUSSA, Zineb RAZMI, Badr Belomaria, Sara EL KARMOUDI and Aya Hameda-Benchekroun — for their invaluable support in making this event a success.
Generative AI is not just a technological advancement; it represents a paradigm shift in how we approach problem-solving and innovation. As we continue to explore its potential, it is essential to stay informed and engaged with the latest developments in this exciting field.
Thank you to everyone who attended and contributed to the vibrant discussions. Together, we are shaping the future of engineering.
Generative AI — The future of Engineering was originally published in Chatbots Life on Medium, where people are continuing the conversation by highlighting and responding to this story.
