Imagine you're taking an exam and you encounter a question you're not sure about. Do you leave it blank, or take your best guess? For most people, the answer depends on the stakes—but for AI language models, the choice isn't that simple. A groundbreaking new research paper reveals that hallucinations in AI aren't mysterious glitches—they're the predictable result of how we train and test these systems.

The Student Analogy That Changes Everything

The researchers behind this study use a compelling analogy: language models are like students who always guess on tests rather than saying "I don't know." Just as students facing difficult exam questions might fabricate plausible-sounding answers instead of admitting uncertainty, AI models produce confident but incorrect responses when they should express doubt.

This isn't a bug—it's a feature baked into how these systems learn and are evaluated.

What Are Hallucinations, Really?

AI hallucinations occur when language models generate plausible yet incorrect statements instead of acknowledging uncertainty. These aren't random errors but systematic problems that arise from two key stages in AI development:

During Training (Pretraining): Even with perfect training data, the statistical methods used to train language models naturally lead to errors. The research shows that generating valid outputs is mathematically harder than simply classifying whether something is correct.

During Testing (Post-training): Current evaluation methods actually reward guessing over honesty. Models that say "I don't know" score lower than those that confidently guess wrong answers.

The Hidden Mathematics of AI Mistakes

The paper reveals a surprising mathematical relationship: hallucination rates are directly connected to classification errors. This means the same statistical factors that cause mistakes in simple yes/no questions also drive more complex AI hallucinations.

Three main factors contribute to these errors:

• Statistical Complexity
  When there's no learnable pattern in the data—like trying to memorize everyone's birthday—models inevitably hallucinate. The research shows that base models should hallucinate on at least 20% of facts that appear only once in training data.
• Poor Models
  Even when patterns exist, inadequate model architectures can't capture them effectively. For example, older trigram models that only look at two previous words must have error rates of at least 50% for certain language tasks.
• Computational Limits
  Some problems are simply too hard to solve, even for superintelligent AI. No algorithm can violate computational complexity theory, meaning certain queries will always produce uncertain responses.

The Evaluation Problem That Makes Everything Worse

Here's the shocking revelation: most AI evaluation methods actively encourage hallucinations. The research analyzed popular benchmarks and found that they predominantly use binary scoring—you're either right or wrong, with no credit for saying "I don't know."

This creates what researchers call an "epidemic of penalizing uncertainty". When faced with two models—one that honestly admits uncertainty and another that always guesses—the guessing model will score higher on most current tests.

Current evaluation statistics show this problem clearly:

• GPQA: Multiple-choice accuracy with no "I don't know" option
• MMLU-Pro: Binary scoring with zero credit for abstention
• SWE-bench: Pass/fail grading that treats uncertainty as failure
• IFEval: Instruction-following tasks with no abstention mechanism

Only WildBench offers partial credit for expressing uncertainty, but even then, responses that say "I don't know" typically score lower than responses with factual errors.

Current Detection Methods Show Promise

Despite these systemic issues, researchers have developed several approaches to detect hallucinations:

Uncertainty-Based Detection: Methods that analyze how confident a model is in its predictions, achieving up to 97.8% accuracy in detecting certain types of hallucinations.

Semantic Analysis: Comparing generated content with retrieved information or known facts to identify inconsistencies.

Ensemble Approaches: Using multiple models to cross-validate responses, with disagreement indicating potential hallucinations.

RAG-Based Validation: Retrieval-Augmented Generation systems that fact-check responses against external knowledge sources, showing accuracy rates above 75% for prompt-based detection methods.

The Path Forward: Rewarding Honesty

The solution isn't building better hallucination detectors—it's fundamentally changing how we evaluate AI systems. The researchers propose explicit confidence targets in evaluation:

Instead of: "Answer this question correctly"

Use: "Answer only if you are >75% confident, since mistakes are penalized 3 points while correct answers receive 1 point, and 'I don't know' receives 0 points"

This approach would make saying "I don't know" the optimal strategy when models are genuinely uncertain, aligning their behavior with what we actually want from trustworthy AI systems.

Why This Matters for Everyone

Understanding hallucinations isn't just an academic exercise—it has real-world implications:

For Users: Knowing that AI models are systematically trained to guess rather than express uncertainty helps you evaluate their responses more critically.

For Developers: The research shows that smaller models can actually be more honest than larger ones, since it's easier to know your limits when you have fewer capabilities.

For Society: As AI systems become more integrated into critical decisions, we need evaluation methods that reward honesty over confident guessing.

The Bottom Line

AI hallucinations aren't mysterious or inevitable—they're the predictable result of reward systems that prioritize appearing confident over being honest. The breakthrough insight is that this problem can be solved not by building better models, but by changing how we measure their success.

Language models can learn to say "I don't know"—we just need to start rewarding them for it. This shift from accuracy-obsessed evaluation to uncertainty-aware assessment could be the key to building truly trustworthy AI systems.

The research concludes with a clear message: we understand why language models hallucinate, and we know how to fix it. The question now is whether the AI community will embrace evaluation methods that reward intellectual humility over confident guessing.

As AI continues to evolve, this fundamental shift in how we think about and measure AI performance could determine whether these powerful systems become reliable partners or perpetual fabricators in our information ecosystem.

References

1. Kalai, Adam Tauman; Nachum, Ofir; Vempala, Santosh S.; and Zhang, Edwin. "Why Language Models Hallucinate" arXiv:2509.04664 (2025).
2. Alber, Daniel Alexander; Yang, Zihao; Alyakin, Anton; Yang, Eunice; Rai, Sumedha; Valliani, Aly A.; et al. "Medical Large Language Models Are Vulnerable to Data-Poisoning Attacks" Nature Medicine 31, 2 (2025), 618–626.
3. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv:2305.13534 [cs.CL]

5. Retrieval-Augmented Generation (RAG)

Where most fine-tuning approaches focus on adjusting the model’s internal parameters, Retrieval-Augmented Generation (RAG) [5] takes a different path. Instead of expecting the model to memorize every bit of specialized knowledge, RAG equips the model with the ability to look up relevant information from an external source whenever it needs it. This is like giving our well-read “student” (the LLM) a library card that allows them to borrow exactly the right books at the right time.

RAG generally works in three phases:

1. Retrieve: When the model receives a query (e.g., “Translate this product catalog into Italian”), it first consults an external database or “knowledge store” for relevant information—like industry glossaries, product names, or domain-specific style guidelines.
2. Augment: The retrieved information is then combined with the model’s existing understanding of language.
3. Generate: Finally, the LLM uses this combined context to produce its output. Hence the name.

This approach keeps the LLM “lighter” on details that might be too specialized or constantly changing.

For translation services, this approach can greatly improve consistency and accuracy. If a client insists on specific product names or industry-approved terminology, these can be stored externally. Whenever the model needs those details, it can retrieve them on the fly, ensuring the translations remain perfectly aligned with the client’s preferences—without requiring the model to be re-trained every time the glossary is updated. It’s also a safer bet for frequently evolving content: a translator model can quickly adapt to new product features or a recent corporate rebranding simply by referencing an updated knowledge base.

6. Mixture of Experts (MOE)

Mixture of Experts [6] is another advanced approach that focuses on scaling LLMs by dividing the model into multiple specialized “expert” modules. Each expert is trained on a particular domain or set of skills, such as legal text, medical terminology, or marketing copy. Then a gating network decides which expert (or combination of experts) should handle a given input.

Think of MoE as a team of tutors, each specializing in a different subject. The gating mechanism is like a guidance counselor, listening to the question at hand and assigning the right tutor to provide the answer. This setup lets a large model cover many domains without forcing any single part of the model to learn everything.

Hence, MoE is more efficient, as not all experts need to be active at any one time. This means you can scale up the number of experts without linearly scaling up the compute costs. However, MoE can be more complex to manage and train because each expert must be kept up to date, and the gating network needs robust data to learn accurate routing decisions.

7. Choosing the Right Approach

When deciding how to fine-tune a Large Language Model (LLM), consider your resource budget, task complexity, and maintenance needs. Below is a high-level comparison of the methods we’ve covered:

Final Thoughts

Finetuning an LLM is a bit like teaching an already well-read student to excel in a specific career path. From comprehensive retraining (Full-Model Fine-tuning) to quick, targeted lessons (PEFT), from careful instruction on etiquette and style (Instruction Fine-tuning) to granting instant access to an ever-growing reference library (RAG), each approach offers unique strengths.

By mixing and matching these methods—based on task complexity, budget, and how often your knowledge base changes—your AI models can remain both highly accurate and agile. For translation services, specifically, these techniques ensure that you deliver precise, context-aware content that meets evolving industry and client needs.

As AI continues to evolve, new hybrid strategies will emerge. EZ stays on top of these developments and we deploy them thoughtfully to keep our language solutions robust, efficient, and ready to adapt to tomorrow’s challenges.

References:

1. Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.
2. Jiang, Albert Q., Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot et al. "Mixtral of experts." arXiv preprint arXiv:2401.04088 (2024).

Large Language Models (LLMs), like ChatGPT or Gemini, might seem magical, but their capabilities come from meticulous training and tuning. They are trained on an almost unfathomably large dataset. This imparts in them a broad understanding of language—like someone who’s read an entire library. However, this is just the first step in their creation, as knowing a lot doesn’t make one an expert. Just because someone has read about baking doesn’t mean they can perfectly whip up a four-tier wedding cake. They might still need specific lessons or guidance. That’s where finetuning comes in.

At our company, we specialize in fine-tuning LLMs to make them domain experts. Whether it’s translating complex legal documents or helping businesses communicate seamlessly across languages, we tailor these brilliant models to excel in specific tasks. Let’s dive into the fascinating process behind teaching AI to deliver precision and professionalism.

1. Pre-Training: Laying the Groundwork

Pre-training is the initial phase where an LLM ingests massive amounts of text (think: books, websites, articles) to acquire general language capabilities. It is sort of like letting a curious kid loose in a gigantic library. Eventually, they will have a broad understanding of language, and how the world works. At least, theoretically speaking.

Pre-training isn’t something most companies do themselves—this phase typically requires enormous datasets and compute resources. Instead, organizations use publicly available pre-trained models (e.g., from open-source communities or AI providers) as a starting point.

2. Full-Model Tuning: Overwriting existing Knowledge

Full model finetuning updates all of an LLM’s parameters, by retraining it on a curated domain-specific dataset,to improve performance on a specific task or domain. While this method often yields top performance for a narrow domain, and deeply tailors the model to very specific tasks or jargon, this can lead to the model losing some of its general capabilities if the new data is too narrow.

Full-Model Tuning is also computationally expensive, and requires significant expertise to attain training stability.

To continue our analogy, this is sort of like putting the studious kid in a library consisting entirely of bakery literature. They might become proficient at baking, but it will come at the cost of them forgetting other stuff they learnt in the general library, and performing poorly on non-baking related tasks.

3. Parameter-Efficient Finetuning (PEFT) : Tweaking a Few Knobs

Full-model tuning is often too heavy-handed. Enter PEFT -Full-model tuning is often too heavy-handed. Enter PEFT—a set of methods that updates only a small portion of the model’s parameters, keeping the rest frozen [1].

If full-model finetuning is like re-schooling a child from the ground up, PEFT is more like giving them evening tutoring sessions specifically on how to bake a cake or fix a bike—without overhauling their general education.

3.1. Low-Rank Adaptation (LoRA)

In this method, we insert small, trainable matrices into the model’s layers while the core parameters remain frozen. This reduces the number of trainable parameters, speeding up the process and lowering costs. This method is particularly useful in scenarios where multiple clients require fine-tuned models for different applications, allowing for the creation of specific weights for each use case without the need for separate models [2].

There’s also Quantized LoRA (QLoRA) [3], which is an extended version of LoRA that gives us greater memory efficiency by quantising weight parameters. Typically, LLM parameters are stored in a 32-bit format, but QLoRA compresses them to 4-bit or lower, significantly reducing the memory footprint.

3.2. Prompt Tuning

This technique involves learning a small set of continuous task-specific vectors called “soft prompts” that are prepended to the input embeddings. These learnt prompt tokens shape how the model generates text, leaving most model weights, and its existing knowledge base, untouched.

Prompt tuning allows efficient task switching and can be more interpretable than other fine-tuning methods.

4. Instruction Finetuning

Instruction finetuning is all about training an LLM on examples where it’s explicitly shown how to respond to different instructions or queries. Instead of merely digesting vast swaths of text, the model is guided in a “question-and-answer” or “command-and-response” style. By training the model on these pairs of prompts and responses, the LLM learns a direct mapping between a user instruction and the desired output.

Think of this as giving our well-read “student” a step-by-step guide on proper etiquette and how to respond to specific cues. They already have the knowledge (from the massive “library” they’ve read during pre-training), but now we’re teaching them precisely how to apply that knowledge when someone asks a question or issues a command.

4.1. Reinforcement Learning from Human Feedback (RLHF)

An especially powerful form of instruction fine-tuning is Reinforcement Learning from Human Feedback (RLHF). In RLHF, the model’s outputs are continually rated by humans, and these ratings are used to guide the training process. Think of it as a teacher who reviews the student’s assignments and offers real-time praise or corrections. The model learns which types of responses are preferred (e.g., polite, concise, accurate) and which are not (e.g., rude, incorrect, or irrelevant) [4].

Over multiple rounds of feedback and adjustment, Reinforcement Learning from Human Feedback (RLHF) can produce a model that’s more aligned with human values, brand guidelines, or professional standards. For translation tasks, this means ensuring the output is not only culturally sensitive and regulation-compliant but also tailored to client-specific style preferences.

At EZ, this aligns directly with us — where human-first intelligence meets tech-enabled precision. Our learning algorithms don’t just adapt to feedback — they evolve with it. Every interaction, every edit, and every cultural nuance helps refine the system — making each output smarter, sharper, and more aligned with the standards our clients expect.

There’s more to this story. Stay tuned for Part 2 — where we dive deeper into how RLHF powers localization at scale.

References:

1. Xu, Lingling, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. "Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment." arXiv preprint arXiv:2312.12148 (2023).
2. Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).
3. Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).
4. Li, Zihao, Zhuoran Yang, and Mengdi Wang. "Reinforcement learning with human feedback: Learning dynamic choices via pessimism." arXiv preprint arXiv:2305.18438 (2023).

Artificial intelligence is reshaping industries, and Retrieval Augmented Generation (RAG) stands at the forefront of this transformation. By blending retrieval systems with generative AI, RAG bridges the gap between static models and dynamic, context-aware solutions. This article explores how RAG redefines accuracy, efficiency, and adaptability in AI applications.

In recent years, artificial intelligence has advanced significantly particularly in the field of natural language processing. Among these innovations, Retrieval Augmented Generation (RAG) has revolutionized the field. Using the benefits of generative models and retrieval systems RAG addresses significant problems in accuracy, flexibility and efficiency making it a useful tool for developers and businesses alike.

But what makes RAG unique and different from traditional AI methods? Let's look at its components, advantages and transformational potential.

What is Retrieval Augmented Generation?

RAG is an AI framework that generates factual, contextually relevant, and coherent responses. It combines the three essential processes of retrieval, augmentation and generation. This is how each process operates:

• Retrieval: The retrieval module fetches relevant data from a large database or knowledge base. It transforms the user query into an embedding or numerical vector and then performs similarity searches to find the most important documents or data points. This ensures the system is equipped with up-to-date domain-specific knowledge.
• Augmentation: The retrieved data is organized and enhanced by merging it with the original user query. This step guarantees that the input prompt given to the generative model is clear, meaningful and contextual. Augmentation is used to close the gap between the final generated response and the raw retrieved information.
• Generation: After processing the enhanced user input, the generative module (typically a Large Language Model) generally creates a thorough, well-organized and coherent answer. Hallucinations (erroneous or nonsensical outputs) are reduced by this step which grounds the model output in retrieved knowledge.

Advantages of RAG

Compared to more traditional AI methods like training separate LLMs or relying solely on retrieval or generative models, RAG's modular design offers numerous benefits. Here are a few of the primary advantages:

1. Avoids the need for training massive LLMs

It takes a tremendous amount of data, computing power, and time to train a large language model from scratch or improve an existing one. RAG, on the other hand, uses pre-trained LLMs and improves their performance by dynamically obtaining relevant information.

• Lower Costs: erabytes of domain-specific data are not required for model training.
• Faster Implementation: By plugging in pre-existing retrieval systems and generative models, you can bypass time-consuming training procedures.
• Adaptability: The knowledge base can be readily updated or expanded without requiring a complete system retraining.

2. Works with Custom Data Without Sharing Full Confidential Datasets

RAG provides a privacy-preserving solution for businesses handling proprietary or sensitive data:

• Local Hosting: Only pertinent query-specific data is retrieved and shared with the model if the LLM is hosted locally or on your own server, protecting the larger dataset.
• Selective Exposure: RAG minimizes the danger of data leakage or unauthorized access by retrieving only the necessary slices of data, in contrast to standard training, which necessitates full dataset exposure to the model.
• Confidentiality: Because data privacy is crucial in sectors including healthcare, finance, and legal services, RAG is especially appealing.

3. Dynamic and Real-Time Knowledge Integration

RAG is perfect for applications that require current insights since it pulls in real-time data, unlike static generative models that depend on out-of-date training data:

• Examples include summarizing news or analyzing the financial market, where data is constantly changing.
• Real-Time Updates: The system's replies are automatically updated as the database is updated, eliminating the requirement to retrain the model.

4. Domain-Specific Adaptability

RAG offers businesses unmatched flexibility across industries by enabling them to customize the retrieval database to meet their unique needs:

• Healthcare: Look up and produce answers using patient records, guidelines, or medical journals.
• Education: Give students specialized study guides or background information.
• Customer service: To accurately answer user questions, consult the most recent company policies or product manuals.

5. Cost-Efficiency

By shifting knowledge storage to the retrieval module instead of integrating everything into the generative model, RAG dramatically lowers operating costs:

• Smaller Model Sizes: Because the retrieval module takes care of the laborious task of finding and filtering relevant information, RAG can operate with comparatively light LLMs.
• Optimized Resources: Similar to vector databases, retrieval systems are built for effective data search and scale with ease.

6. Higher Accuracy and Reduced Hallucinations

Conventional generative models frequently create answers when they come across new questions. However, RAG grounds its outputs in retrieved data, which increases their reliability and factualness.

• Reliable Outcomes: RAG minimizes errors and guarantees consistency by securing the response to an external knowledge base.
• Use in Critical Fields: RAG provides accurate and verifiable results for applications in the legal, medical, and financial domains.

Real World Applications of RAG

RAG's broad range of applications across industries demonstrates its adaptability:

• Customer Service: RAG-powered chatbots can provide precise and individualized support by dynamically retrieving answers from product manuals, policy documents, and frequently asked questions.
• Medical Support: RAG can help physicians diagnose conditions or recommend treatments by retrieving data from patient histories or medical literature.
• Legal Investigation: RAG can be used by legal professionals to find useful statutes or case laws quickly, improving accuracy and saving time.
• Production of Content: By fusing fact-based retrieval with creative language generation, RAG can be used by writers and marketers to produce articles, product descriptions, or summaries.

Challenges and Future Scope

Despite being a game-changing technology, RAG has drawbacks.

• Latency: There may be delays when retrieving and processing massive amounts of data in real-time.
• Data Quality: The quality and applicability of the retrieval database determine how accurate RAG's outputs are.
• System Complexity: Complex engineering is needed to integrate retrieval, augmentation, and generation.

In spite of these obstacles, RAG systems are still being improved by developments in generative AI, embedding models, and vector search. RAG frameworks may become even faster, more precise, and domain-adaptable in the future.

Conclusion

In terms of AI-powered solutions, retrieval-augmented generation is the next big thing. It gets around the drawbacks of conventional models by incorporating retrieval, augmentation, and generation to provide accurate, scalable, and economical solutions that are suited to particular use cases.

RAG is an essential tool because it can work with custom data and avoid lengthy training cycles, whether you're a developer creating a real-time assistant or a company protecting sensitive data. RAG's applications will develop along with industries, solidifying its position as a key component of AI innovation.

RAG is transforming AI with its ability to deliver smarter, faster, and consistently accurate solutions—aligning perfectly with EZ’s ethos of high quality, speed, and reliability. As businesses continue to innovate, RAG’s efficiency and adaptability ensure it remains a cornerstone for future-ready solutions, much like EZ’s unwavering commitment to empowering professionals worldwide

As the digital landscape evolves and data becomes more crucial than ever, the need to protect sensitive information has never been more pressing. In the words of former FBI Director Robert Mueller, “There are only two types of companies: those that have been hacked and those that will be.” This reality underscores the importance of proactive measures in cybersecurity.

In an era where data is often referred to as the new oil, the importance of safeguarding it has reached unprecedented heights. As cyber threats become increasingly sophisticated, traditional security measures like firewalls, once the frontline defence against cyberattacks, are now just one piece of a much larger puzzle.. Welcome to the world of next-generation data protection, where artificial intelligence (AI) is not just an enhancement but a necessity.

This blog explores how AI is reshaping the landscape of cybersecurity. Imagine a security system that learns and adapts in real-time, identifying potential threats before they can cause harm. With its ability to analyse vast amounts of data, detect anomalies, and automate responses, AI is revolutionising how organisations defend themselves against cyberattacks.

The Cybersecurity Crisis: A Reality Check

Imagine waking up to find your bank account drained overnight because a cybercriminal exploited a vulnerability in your security system. Scary, right? This scenario is becoming all too common as cybercriminals develop increasingly sophisticated tactics. Traditional security measures, such as firewalls, are like putting a band-aid on a bullet wound—they might help, but they won’t stop the bleeding.

Why Firewalls Are No Longer Enough

Firewalls act as the first line of defence, controlling incoming and outgoing traffic. However, they are limited in their ability to detect and respond to complex threats. Think of them as a bouncer at a club who only checks IDs but doesn’t notice if someone sneaks in through the back door. With AI, we can turn that bouncer into a highly trained security team that not only checks IDs but also analyses behaviour patterns and identifies potential threats before they enter.

AI: The New Defender of Data

• Intelligent Threat Detection: AI systems are capable of processing vast amounts of data in real-time, allowing them to identify patterns and anomalies that may indicate a security breach. By leveraging machine learning algorithms, AI can learn from historical data and improve its threat detection capabilities over time. This means that organisations can detect potential threats much earlier than with traditional methods, minimising the risk of data breaches.
• Automated Incident Response: When a cyberattack occurs, speed is of the essence. AI can automate incident response processes, allowing organisations to react swiftly. For instance, when an anomaly is detected, AI systems can automatically isolate affected systems, block malicious traffic, and initiate recovery protocols—all without human intervention. It’s like having a fire alarm that not only alerts you to danger but also calls the fire department and extinguishes the flames!
• Automation of Security Protocols: Automation is another significant advantage of AI in data security. By automating routine security tasks, organisations can reduce the risk of human error, which is a common vulnerability in data protection. AI-driven systems can monitor network traffic, analyse user behaviour, and respond to incidents without human intervention, allowing cybersecurity teams to focus on more complex challenges.
• Prescriptive Analysis: By offering practical suggestions based on data analysis, prescriptive analysis can aid in data security. It does more than just recognise and anticipate dangers; it also makes recommendations for particular actions to reduce risks and improve security.

Why did the computer go to therapy?

Because it had too many bytes of emotional baggage! In the world of cybersecurity, AI helps lighten the load by managing the heavy lifting of data protection, allowing human teams to focus on strategy and innovation rather than constantly putting out fires.

Real-World Applications of AI in Data Protection

The integration of AI into data security strategies is already yielding significant results across various sectors:

• Financial Services: Banks and financial institutions are using AI to monitor transactions for signs of fraud. By analysing transaction patterns in real-time, AI systems can flag suspicious activities and prevent fraudulent transactions before they occur.
• Healthcare: In the healthcare sector, AI is being employed to protect patient data. AI systems can monitor access to sensitive records, detecting unauthorised access and ensuring compliance with regulations such as HIPAA.
• Retail: Retailers are leveraging AI to enhance customer data protection. By analysing purchase patterns and customer interactions, AI can identify potential data breaches and protect sensitive customer information.

Machine learning:

Data security is improved by machine learning (ML), which automates the detection and reaction to cyberattacks. By recognising typical system behaviour and highlighting variations, it can detect anomalies. By examining past data, machine learning (ML) forecasts possible hazards and enables proactive mitigation. Additionally, it speeds up response times by automating answers to threats. Furthermore, machine learning (ML) filters spam and phishing efforts by identifying characteristics of dangerous content and detects fraud by finding patterns in real-time transactions. Data protection becomes more effective and efficient as a result.

Natural Language Processing:

NLP, or natural language processing, improves data security by deciphering and interpreting spoken language. By looking for questionable linguistic patterns in email content, it can identify phishing efforts. NLP is also used to extract pertinent threat intelligence from social media, forums, and security reports. Through text analysis, NLP can also assist in detecting leaks of sensitive information and tracking adherence to security guidelines. Better threat detection and more knowledgeable security judgments are the results of this.

Challenges and Ethical Considerations

While AI offers numerous advantages, it also presents challenges. The reliance on AI systems raises questions about data privacy, particularly regarding how personal information is collected, stored, and used. Organisations must ensure compliance with regulations like DPDPA, GDPR while leveraging AI's capabilities.

As AI continues to become more advanced, it is essential to focus on its responsible development and deployment. We must make sure that AI systems are developed with ethical concerns, transparency, and responsibility in view. Responsible AI can assist in reducing such risks and make sure that AI technologies have positive impacts towards all people. So, protecting personal information has a story the threats and methods of fighting them always change

Moreover, the quality of data used to train AI models is crucial. Biassed or inaccurate data can lead to flawed decision-making. As we embrace AI in data protection, we must also prioritise ethical considerations and transparency to build trust with users.

Conclusion: The Future of Data Protection

As we move beyond firewalls, the role of AI in next-generation data protection becomes increasingly vital. By harnessing the power of AI, organisations can enhance their cybersecurity posture, proactively defend against threats, and ensure compliance with data privacy regulations.

The future of data security lies in the integration of AI technologies that not only protect sensitive information but also foster trust and accountability in an ever-evolving digital landscape. So, let’s embrace AI as our trusty sidekick in the quest for robust data protection—because in the battle against cyber threats, every superhero needs a powerful ally!

As we face the complexities of digital threats, we at EZ build our approach towards enhancing data protection through innovative AI-driven solutions. Our methods strengthen your defenses while empowering organizations to adapt and thrive in a fast-changing environment. Together, we can build a future where data integrity and trust are at the forefront.

References:

• AI in data privacy and security.
• Research on Data Privacy Protection Strategies Based on Artificial Intelligence.
• Insights Into Privacy Protection Research in AI.
• Rethinking Privacy in the AI Era.

AI is undergoing a paradigm shift in which models are being pre-trained on a broad set of extensive training data using self-supervised learning with deep neural networks with further fine-tuning on a wide range of downstream tasks. Usually, these pre-trained models (BERT, BART, DALL-E, or GPT's) are called foundation models. These foundation models provide a strong basis for solving downstream tasks such as text classification, text entailment, image classification, etc.

Due to the emergence of foundation models, significant interest has developed in Artificial Intelligence to utilize these models in revolutionizing various fields, from natural language processing to computer vision.

Due to the emergence of foundation models, significant interest has developed in Artificial Intelligence to utilize these models in revolutionizing various fields, from natural language processing to computer vision utilizing new emergent capabilities. However, alongside their promises and incentives, foundation models also bring forth a host of challenges and risks that necessitate careful consideration. Emergence and homogenization are two key concepts that play a significant role in the development and impact of foundation models and AI systems in general. Emergence is a source of both excitement and concern in AI research. It can lead to unexpected and novel capabilities in AI systems that were not explicitly programmed or anticipated during the design phase. For instance, the ability of GPT-3 to perform a wide range of tasks through natural language prompts is an emergent property that was not explicitly trained for. **Homogenization** refers to the process of consolidating methodologies or approaches for building AI systems across different applications or domains. It involves standardizing practices, architectures, or techniques to create a unified foundation that can be leveraged for various tasks. However, having Homogenization poses a risk of a single point of failure for AI systems.

This blog will outline some of the incentives, Opportunities, and Risks of utilizing Foundation models.

Key Incentives of Using Foundational Models

Task and Domain Adaptation: Foundation models are like building blocks that can be easily adjusted to solve different problems, making AI more versatile and useful in many situations. Unlike "one-size-fits-all" solutions, these models can be adapted to fit the specific needs of different industries and tasks.

Transfer Learning: Building AI solutions just got easier and faster! Thanks to foundation models, companies can jump-start their AI projects without needing massive resources or time investments. Forget starting from scratch. Foundation models act as pre-trained building blocks, allowing businesses to quickly customize AI solutions for their specific needs, saving both time and money.

Cost and Time Savings: Building AI solutions just got easier and faster! Thanks to foundation models, companies can jump-start their AI projects without needing massive resources or time investments. Forget starting from scratch! Foundation models are like pre-built Lego towers for AI, saving valuable time for developers. Say goodbye to months of AI development. Foundation models come pre-trained, like learning to ride a bike with training wheels, accelerating your journey.

Data Annotation: Traditional AI is a data hog, needing mountains of labeled examples to learn. Foundation models, however, are like efficient students, requiring far fewer examples thanks to their pre-trained knowledge. This is a game-changer when data is scarce or expensive. Forget drowning in data! Foundation models learn like champions, needing less labeled data to excel, making them perfect when data collection is a bottleneck.

Versatility: Instead of building separate AI solutions for each data type, foundation models offer a multimodal powerhouse. They can handle text analysis, image recognition, and even combined tasks, streamlining your AI efforts.

Key Opportunities of a Foundation Models

Enhanced Adaptation Performance: Foundation models signify a paradigm shift where massive amounts of data are utilized to enhance adaptation performance significantly. The overarching principle of "the more data, the better" underscores the potential for improved model adaptation and performance.

Multimodal Integration: Foundation models enable data integration across new modalities, such as robotics and healthcare, expanding the scope of applications and capabilities in diverse domains.

Language Understanding and Generation: Foundation models exhibit exceptional proficiency in understanding and generating human language. This capability empowers applications in translation, summarization, conversational interfaces, and various language-related tasks.

Comprehensive Vision Capabilities: In the realm of computer vision, foundation models have shown promise in leveraging RGB-3D data to comprehend indoor environments, paving the way for advancements in visual understanding and scene analysis.

Improved Content Creation: Foundation models have the potential to generate content that looks like it has been created by humans, enabling the creation of high-quality text and images across a wide range of languages. This capability can be harnessed for various creative and communicative purposes.

Empowering AI Applications: The capabilities of foundation models extend to diverse fields such as law, healthcare, and education, offering opportunities to enhance existing applications and develop innovative solutions in these domains.

Scalability and Efficiency: Foundation models, with their large-scale architecture and training procedures, provide scalability and efficiency in handling complex tasks and datasets, making them suitable for a wide range of applications.

Risks in using Foundational Models

Bias and Fairness Concerns: Foundation models can adopt biases embedded in the training data, resulting in biased outcomes and perpetuating societal inequalities. Addressing bias and ensuring fairness in model predictions is crucial to prevent discriminatory practices.

Lack of Transparency: Foundation models are often complex and difficult to interpret, making it challenging to make specific decisions or predictions. This lack of transparency,can impede trust in the model's outputs and raise concerns about accountability.

Data Privacy and Security: Foundation models require vast amounts of data for training, raising concerns about data privacy and security. Unauthorized access to sensitive data used in training foundation models can lead to privacy breaches and data misuse.

Environmental Impact: The training and deployment of foundation models consume significant computational resources, contributing to environmental concerns related to energy consumption and carbon emissions. Addressing the environmental impact of foundation models is essential for sustainable AI development.

Robustness and Generalization: Foundation models, due to their complexity and scale, may exhibit vulnerabilities and lack robustness in certain scenarios. Ensuring the robustness and generalization of foundation models across diverse use cases is crucial to prevent unexpected failures.

Ethical Considerations: The widespread deployment of foundation models raises ethical dilemmas related to the responsible use of AI technologies. Ethical considerations such as transparency, accountability, and fairness must be prioritized to mitigate potential harm and ensure ethical AI practices.

Misuse and Misalignment: Foundation models can be susceptible to misuse or optimization for misaligned goals, leading to unintended consequences or ethical dilemmas. Safeguarding against the misuse of foundation models and aligning their objectives with societal values is essential for ethical AI development.

As we navigate the Incentives, opportunities, and risks inherent in foundation models, it becomes clear that their development requires a nuanced understanding of training, data, and evaluation methodologies.

• "[1810.04805] BERT: Pre-training of Deep Bidirectional Transformers ...." 11 Oct. 2018, https://arxiv.org/abs/1810.04805. Accessed 7 Feb. 2024.
• "BART: Denoising Sequence-to-Sequence Pre-training for Natural ...." 29 Oct. 2019, https://arxiv.org/abs/1910.13461. Accessed 7 Feb. 2024.
• "DALL·E 3 - OpenAI." https://openai.com/dall-e-3. Accessed 7 Feb. 2024.
• "GPT-4 - OpenAI." 13 Mar. 2023, https://openai.com/gpt-4. Accessed 7 Feb. 2024.
• "On the Opportunities and Risks of Foundation Models - arXiv." 16 Aug. 2021, https://arxiv.org/abs/2108.07258. Accessed 7 Feb. 2024.
• "Are We Modeling the Task or the Annotator? An Investigation ... - arXiv." 21 Aug. 2019, https://arxiv.org/abs/1908.07898. Accessed 7 Feb. 2024.
• "On the Opportunities and Risks of Foundation Models - arXiv." https://arxiv.org/abs/2108.07258. Accessed 7 Feb. 2024.
• "Foundation models: Opportunities, risks and mitigations - IBM." https://www.ibm.com/downloads/cas/E5KE5KRZ. Accessed 7 Feb. 2024.

Introduction

In recent years, there has been a growing trend of people deleting their social media accounts and online profiles in an effort to regain control of their data and privacy. This "delete culture" movement has seen users leaving platforms like Facebook, Reddit, and others due to data leaks, privacy concerns, and unwanted policy changes. For example, after the Cambridge Analytica scandal revealed that up to 87 million Facebook users had their data improperly shared, many decided to #DeleteFacebook. Additionally, changes to Reddit's API and policies around third-party apps is leading their users to abandon the platform.

As people become more aware of the misuse of their personal data in the virtual world, the desire to delete one's digital footprint is understandable. However, deleting an account is not as simple as pressing a button. Our data persists in many forms, from cached versions on other servers to machine learning models that have analyzed our profiles. There is a need to not just let people control their data, but also provide them with control over how it is used to train algorithms. But the most important thing in the whole discussion is the “trust” that needs to be established by responsible use of AI systems.

A growing field of study in AI that focuses on the moral and ethical implications of the design, development and use of artificial intelligence systems is termed as ethical and responsible AI. More empirical and qualitative study is required when considering the impact of AI and its application on individuals, society and the environment with respect to open ended issues such as biasness, fairness, transparency, accountability and privacy. Machine unlearning and responsible design patterns in AI applications can be one potential solution for maintaining privacy with respect to AI and its applications.

Machine Unlearning

Machine unlearning refers to the process of removing data from AI and machine learning models. The goal is to induce "selective amnesia" so that the models forget specific people or types of information, without compromising the model's overall performance. Apart from enforcing data privacy on a deeper, more meaningful level, there are other benefits of this procedure too:

• Improving data security by eliminating vulnerabilities from machine learning models
• Increasing trust in AI systems by providing users with more transparency, and control over their data.
• Reducing bias in these systems by addressing data imbalances.
• Supporting privacy regulations like GDPR's data deletion requirements.

But wait, how can a machine ‘unlearn’? Just like, it’s much easier to teach a child something than to make them forget it.

There are actually some intuitive ways to make a model ‘unlearn’ the information it learned from a user’s data. Some of these are:

• Model retraining: This involves retraining a machine learning model from scratch with the deleted data. This is computationally expensive but ensures the data is fully removed from the model.
• Data poisoning:This modifies or removes the deleted data in a way that the model's predictions for that data become meaningless, essentially corrupting the knowledge it gained from the data.
• Differential privacy: Noise is added to data before training a model so that no individual's data can be identified, allowing for data deletion at a later point.

However, these relatively simple strategies may not be effective for the gigantically complex models of today. There are several approaches to help these massive machines forget what they've learned from our data:

• SISA Training: This method strategically limits the influence of a data point in the training procedure, expediting the unlearning process. SISA training reduces the computational overhead associated with unlearning, even in the worst-case scenarios where unlearning requests are made uniformly across the training set.
• Data Partitioning and Ordering:By taking into account the distribution of unlearning requests, data can be partitioned and ordered accordingly to further decrease the overhead from unlearning. It's like organizing a messy room, making it easier to find and remove specific items when needed.
• Transfer Learning:This technique involves using pre-trained models to speed up the retraining process after unlearning.

Responsible Design Patterns

In addition to machine unlearning, responsible design patterns machine learning pipelines play a pivotal role in ensuring ethical and fair use of AI. These patterns involve designing the pipeline in a way that promotes transparency, accountability, and fairness.

For example,

• Using diverse and representative datasets can help reduce bias in the models.
• Pushing towards explainable AI allows us to understand how the model makes decisions and identify any potential biases.
• Regular audits and monitoring of the pipeline can detect and address any unintended consequences or biases that may arise.

By implementing responsible design patterns, we can create machine learning systems that are more reliable, unbiased, and respectful of user privacy and data.

Future Work

It is worth noting that machine unlearning is an active area of research, and there is no one-size-fits-all solution. Different approaches may be more suitable depending on the specific context and requirements. As we look into the future, there's still much work to be done in the field of machine unlearning. Researchers and developers must continue to explore new techniques and refine existing ones to make unlearning more efficient and effective. Additionally, the adoption of Responsible Design Patterns will be crucial in ensuring that AI systems are built with machine unlearning and data privacy in mind from the ground up.

Conclusion

In a world where "delete culture" is on the rise, machine unlearning should be considered an essential skill for AI. The AI revolution is unstoppable, and it’s here to stay. Machine unlearning is the closest thing we have to a delete button on the memory of an intelligent system, allowing them to adapt and forget the data they've learned from us.

But machine unlearning is not a perfect solution to this problem. In an ideal world, there will be no need for a model to “unlearn”, as it wouldn’t learn what it is not supposed to in the first place. The only way to ensure this is to standardize and integrate responsible design patterns in our existing machine learning pipelines.

So, let's praise the undying contribution of machine unlearning – the unsung hero of the delete culture, helping us keep our digital skeletons safely locked away in the closet.

• "[1912.03817] Machine Unlearning - arXiv." 9 Dec. 2019, https://arxiv.org/abs/1912.03817. Accessed 15 Jun. 2023.
• "[2209.02299] A Survey of Machine Unlearning - arXiv." 6 Sep. 2022, https://arxiv.org/abs/2209.02299. Accessed 15 Jun. 2023.
• "Responsible Design Patterns for Machine Learning Pipelines - arXiv." 31 May. 2023, https://arxiv.org/abs/2306.01788. Accessed 15 Jun. 2023.
• "a Pattern Collection for Designing Responsible AI Systems - arXiv." 2 Mar. 2022, https://arxiv.org/abs/2203.00905. Accessed 15 Jun. 2023.

Once upon a time, in the not-so-distant past, language models (LMs) were just learning to crawl. Today, they're sprinting at breakneck speed, leaving the human beings behind. Enter the world of Large Language Models (LLMs), massive AI systems that can understand the context and meaning behind words, generate human-like text, answer complex questions, and even write entire articles (unlike this one, which is almost fully written by a human).

The “BIG”

They're like the cool colleagues who can effortlessly write complex code, crack jokes (some of which can even be funny!) , and even help with your homework and essays (wink wink)!. But what sets LLMs apart from their smaller counterparts? Size matters, my friends. The “Large” in LLMs is not just hyperbole. GPT-3, the model supporting OpenAI’s chatGPT, has 175 billion parameters. Huawei Researchers have recently developed LLMs with over a trillion Parameters. In fact, GPT-4 is so large, they won’t even tell us!

These parameters basically represent the building blocks of their knowledge. Knowledge that allows them to perform tasks like sentiment analysis, language translation, and even personal assistant-like chatbots. But the hype around LLMs is not just due to what they can do, but also what they might be able to do. LLMs can be used in fields like healthcare, finance, education, litigation etc, where they can help with tasks like medical diagnosis, financial analysis, language learning, and even simplifying complex legal documents into layman-speak. They can also be used to create chatbots,virtual assistants, highly personalized educators and even video game characters.

The “BRAINY”

Let's dive into the science behind these behemoths (or rather just dip toes; as what follows is a hyper simplification of the inner workings of the most complex machines yet created by humans). Imagine LLMs as giant sponges, soaking up vast amounts of text from the internet, and breaking down the said text into smaller units called tokens; much like how sea sponges filter large amounts of water for plankton and break it down to simple sugar. These tokens are then fed into a neural network, which learns to predict the probability of the next word in a sequence based on the previous words.

There is an ongoing discourse which argues that LLMs are not that impressive, since all they are doing is predicting the next phrases based on their previous phrases. That being said, introspect about the fact that aren’t we humans doing the exact same thing?

The “BOLD”

AI philosophy aside, the evolution of LLMs has been nothing short of breathtaking. In just a few years, it has evolved from basic language models that could barely string a sentence together to sophisticated LLMs like GPT-4, which can write entire articles (like this one!). Following the rapidfire breakthroughs in Language models has been akin to watching a baby go from babbling to reciting Shakespeare in a day.

So, what's next for these linguistic titans? Honestly, it’s hard to predict. The capabilities of GPT-4, for example, shocked even the researchers to the point that they decided to fine tune existing technologies before moving on to developing GPT-5.

Once Spiderman Said…

“With great power comes great responsibility” and therein lies the problem with LLMs. The potential for misuse of LLMs can be scary to think about. They can be used to spread misinformation, generate fake news, or even create deepfake content. To address these concerns, researchers and developers are working on ways to make LLMs more transparent, accountable, and ethical. It's like teaching these AI prodigies not just to be smart, but also to be good citizens.

In a nutshell, the rise of Large Language Models has been a thrilling roller coaster ride, filled with jaw-dropping advancements and mind-boggling potential. As we continue to explore the possibilities and address the challenges, one thing is certain: LLMs are here to stay, and they're ready to save the world, one word at a time!

• OpenAI Presents GPT-3, a 175 Billion Parameters Language Model." Accessed May 4, 2023.
• Huawei has created the world's largest Chinese language model." Accessed May 4, 2023.
• OpenAI's GPT-4 Is Closed Source and Shrouded in Secrecy - VICE." Accessed May 4, 2023.
• NEXT-LEVEL NARRATIVES, GAMES AND EXPERIENCES." 13 Apr. 2023.

In the new changing dynamics of the world economy, data and information have become priceless possessions. According to one of the articles by The Economist^[1], the world’s most valuable resource is no longer oil, but data. With data becoming a valuable resource, securing it and ensuring that it is not misused, has become a matter of grave concern. Hence, it is imperative to take a step ahead of our adversaries and look for security problems associated with storing and handling data.

Cyber Security is the convergence of people, processes, and technology, to protect organizations, individuals, or networks, from digital attacks. It is comparatively easier to prevent cyber attacks, like phishing and malware, but stopping an insider attack is an incredibly daunting task. Insider attacks originate within the organization, and the attackers are generally closely associated with the workplace, directly, indirectly, physically, or logically. Interestingly, insider attacks are the most underestimated attacks in cybersecurity, but preventing them is an extremely challenging task. Training a model that can help prevent insider attacks is extremely difficult, due to the imbalanced nature of the dataset. Moreover, insider attacks are rare anomalies, so we do not have enough data that can be used to train a model.

Application of Machine Learning, in cybersecurity and data security, has always been a challenge, and scarcity of available annotated data resources aggravates this challenge further. Moreover, the availability of a balanced dataset makes machine learning all the more difficult. In the past, techniques, such as random oversampling, undersampling, SMOTE, and more, were used to make the dataset balanced. Synthetic data was created to handle skewed data, too. However, none of those techniques were effective.

We, at EZ, work relentlessly to improve and devise new techniques, such that our clients rest assured about the security of the valuable information they entrust us with. Recently, while reading a paper on Cybersecurity and Deep Learning^[2], we found a new way to detect and prevent insider attacks. The proposed solution is split into three parts, namely, behavior extraction, conditional GAN-based data augmentation, and anomaly detection.

In behavior extraction, feature extraction is done from the dataset. Context-based behavior profiling is used, in which each user is identified as an insider, based on the entire activity log, where all the features contribute to the user behavior. Then, a Conditional Generative Adversarial Network (CGAN) is used to generate data and reduce the negative effect of skewed data. GAN models consist of two parts, namely, generator and discriminator. In the network, the discriminator (D) tries to distinguish whether the data is from the real distribution, and the generator (G) generates synthetic data and tries to fool the discriminator. The research paper uses a fully connected neural network in the generator and discriminator.

The final part of the proposed solution is to use multiclass classification, instead of binary classification. Anomaly detection based on multiclass classification considers labeled samples of training data as multiple normal and non-malicious classes. The multinomial classifier tries to discriminate the anomalous samples from the rest of the classes, which helps in building a more robust classifier. One additional feature of using multiclass classification is that in case a new insider activity emerges, there would be no need to make any changes to the existing framework. We should use t-distributed Stochastic Neighbor Embedding (t-SNE), a manifold-learning-based visualization method, to perform a qualitative analysis of the generated data. XGBoost, MLP, and 1-d CNN models were used in the research paper, XGBoost performed better for all sorts of datasets.

Intrigued to know more about Cyber Security and the unconventional ways to prevent insider attacks? Read the Reference articles provided below -

• Mayra Macas, & Chunming Wu. (2020). Review: Deep Learning Methods for Cybersecurity and Intrusion Detection Systems.
• Gautam Raj Mode, & Khaza Anuarul Hoque. (2020). Crafting Adversarial Examples for Deep Learning-Based Prognostics (Extended Version).
• Ihai Rosenberg, Asaf Shabtai, Yuval Elovici, & Lior Rokach. (2021). Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain.
• Li, D., & Li, Q. (2020). Adversarial Deep Ensemble: Evasion Attacks and Defenses for Malware Detection. IEEE Transactions on Information Forensics and Security, 15, 3886–3900.
• Simran K, Prathiksha Balakrishna, Vinayakumar Ravi, & Soman KP. (2020). Deep Learning-based Frameworks for Handling Imbalance in DGA, Email, and URL Data Analysis.

Mankind has witnessed three industrial revolutions, starting with the development of the Steam Engine, followed by electricity and digital computing. We are on the verge of a 4th industrial revolution that will be primarily driven by Artificial Intelligence and Big Data. Artificial Intelligence heavily relies on the data for the development of algorithms that can reason about the decision-making done by the intelligent systems or computer systems only.

Face Recognition: Modern Day Biometric Security Solution

The advent of these advanced technologies has provided us with various techniques for security solutions that will prevent unauthorized access to precious data, providing a sense of security to our clients. However, selecting the appropriate biometric security solution has become a major decision-making process for businesses and enterprises, across wide industries. A new biometric security system that has arrived under the umbrella of Artificial Intelligence is a Face recognition system.

With the ease of implementation and widespread adoption - face recognition is rapidly becoming the go-to choice for the modern implementation of Biometric Solutions. Facial recognition is a modern-day biometric solution developed for the purpose of recognizing a human face without any physical contact required. Facial recognition algorithms are designed to match the facial features of a person to the images or the facial data available in the database saved.

Facial Recognition The Next Big Thing?

Research and studies on Facial Recognition have been conducted for many years now, but there has been an unprecedented growth when we talk about the actual implementation of Facial Recognition. Technology has become so efficient that now we can unlock our phones using facial recognition. Countries have also started using facial recognition for surveillance purposes to track down criminals and use it to prevent crime. Tracking down criminals has become too easy with the help of facial recognition. All we need to do is set up a camera in public spaces and check if any criminal/suspicious person shows up.

Recent Studies Suggest Otherwise

Recent studies and research have suggested that the leading facial recognition software packages are biased. Yes! You read it right. Leading facial recognition packages tend to be more accurate for white, male faces than for people of color or for women.

In a 2019 study, it was found out that many commercial algorithms currently being used for surveillance show a high False Positive Rate for the minority community. There have been some cases around the world where someone innocent got arrested due to false positives shown by these surveillance devices. One such incident happened in January 2020, in Detroit, when police used facial recognition technology on surveillance footage of theft to falsely arrest a Black man.

Let us try and identify what lies at the core of this based nature of face recognition software programs. Facial recognition application is broadly divided into two parts; Verification and Identification.

• Verification confirms that the faceprint matches with the stored faceprint.
• This is usually used at airports and to unlock your smartphone.
• The verification part of Facial recognition is not biased, in fact extremely accurate; here, artificial intelligence is as skillful as the sharpest-eyed humans.
• The real issue is the Identification part of Facial Recognition, which is used for surveillance.

Disparate False Positive Rates

The false Positive Rate of 60 per 10,000 samples for minority groups might not seem much, but when you compare it with the Positive rate of <5 per 10,000 samples for white people, you can clearly see the difference. We need to make sure that the false-positive rate in the identification model should be minimal since this is usually used on crowd surveillance. If you are using facial recognition for crowd surveillance, and you are monitoring around 5000 people in a day, you could easily end up with hundreds of people being falsely accused^[1].

Once the issue was identified, AI researchers started working on finding a solution to the biases available to these facial recognition models. In June 2020, IBM announced it would no longer offer a facial recognition service, while other service providers have acknowledged the issue and started working on finding a solution ^[2]. There has also been a public backlash against crowd surveillance.

The reason why there is such a high false-positive rate in facial recognition for a minority group is that the data on which these models were built had an uneven distribution of racial faces.

To avoid such errors, new databases and techniques have been used:

• Techniques of augmentation of feature space of underrepresented classes were to make the dataset more balanced.
• Recently, Generative Adversarial Networks (GAN) were also trained to generate face features to augment classes with fewer samples.
• People have also started shifting to more balanced datasets like Racial Faces in the Wild (RFW) and Balanced Faces In the Wild (BFW) to reduce the bias^[3].

There has been a great improvement in accuracy for facial recognition in the past few years. Researchers have better models and constructed better datasets to provide highly accurate and low bias models. Big service providers have acknowledged the problem, constantly researching to create accurate surveillance models. The future of facial Recognition seems bright now as the awareness among other service providers and clients has increased about the drawbacks of such technology.

We, at EZ, believe that the purpose of technology is to assist us, and not replace us. Therefore, before becoming dependent on any programming language, we understand its flaws and make conscious efforts to overcome them. As a programming language, Python provides us with innumerable Python libraries and frameworks, a mature and supportive Python Community, versatility, efficiency, reliability, speed, and more. We work with Python so extensively that its security flaws often get ignored. Read the blog below to know about the security loopholes found in the PyPI ecosystem, and how we can overcome them.

What is PIP?

• PIP, or Python Package Installer for Python, is a default python package manager that provides a platform for developers to share and reuse the codes written by third-party developers.
• PIP supports downloading packages from PyPI.org, a repository for the Python programming language. PyPI helps in finding and installing packages or software for python programming languages.
• By design, the PyPI ecosystem allows any arbitrary user to share and reuse python software packages, which along with their dependencies, are downloaded recursively with the help of PIP.

Security risk while the installation of Python Packages

Bagmar et al. had provided a detailed study on the security threats in the python ecosystem, which is largely based on the PyPI repository database.

• Every time, while PIP installs invocation, two python files are executed, namely, setup.py and __init__.py.
• Along with these executions, some arbitrary Python codes, which may contain exploits, also get executed at varying points.
• Exploits come in two modes, which are given below:
  1. Directly from the source, using editable mode installation, and importing the malicious package.
  2. Installation using sudo(administrator) privileges.

Factors that help us determine the impact of exploiting python packages

There are four main factors that can help us understand the impact of exploiting Python packages, which are given below:

• Package Reach: It is defined as the number of other packages that explicitly require it transitively or directly. Packages with high package reach are liable to higher attack vectors, making them malicious.
• Maintainer Reach: It is the combined reach of all the Maintainer packages. Influential Maintainers are the potential targets for security attacks.
• Implicitly Trusted Packages: It is the number of distinct nodes traversed while searching for the longest path from a given starting node. An increase in implicitly trusted packages increases security risk attacks.
• Implicitly Trusted Maintainers: This metric gives the vulnerability score based on other package maintainer's accounts.

Most common Python Package Impersonation Attacks

Package impersonation attacks are user-centric attacks, which aim at tricking users to download a malicious package.

There are various ways of fooling the users, and making them download malicious packages, some of which are given below:

• TypoSquating: Intentionally making minor spelling mistakes.
• Altering Word Order: Changing the order in which packages name themselves.
• Python3 vs Python2: Adding number “3” in the package, imitating the original package, with support to python3.
• Removing Hyphenation: Removing hyphen in the genuine packages.
• Built-In Packages: There are multiple instances of packages being uploaded to PyPI.
• Jellyfish Attack: In this attack, a TypoSquat package is imported somewhere.

License Violation in PyPI ecosystem

PyPI does not perform any automated checks for OSS license violations. Any violation can be considered when a package imports another package having a less permissible license.

Suggested Preventive Measures

• There should be strict enforcement and compulsion to specify dependencies in the metadata of uploaded packages.
• A permission model, similar to mobile phones, can be implemented while installing packages.
• Having a trusted or maintainer package badge on a popular package might be helpful.
• Showing statistics while installing packages.
• License fields must not be free text.