In the ever-evolving realm of artificial intelligence, the emergence of formidable language models like ChatGPT has sparked a fervent race among developers to introduce their pioneering counterparts. Notable contenders, such as DeepMind’s Sparrow and Anthropic’s Claude, are at the forefront of reshaping this landscape. Anthropic is an AI safety and research company that believes in building reliable, interpretable, and steerable AI systems.
This article sheds light on Riley Goodside’s illuminating comparison of Claude and ChatGPT, as discoverable here: https://medium.com/mlearning-ai/paper-review-constituional-ai-training-llms-using-principles-16c68cfffaef
Amidst these cutting-edge advancements, the domain of Natural Language Processing (NLP) grapples with a trio of formidable challenges: ensuring content accuracy without fabrication (referred to as ‘hallucination’), curbing biased and harmful content, and establishing mechanisms to properly cite AI-generated information.
This exploration delves into a central concept propelling Claude’s development: Constitutional AI. This innovative approach addresses the task of producing less harmful AI-generated content by weaving the principles of Constitutional AI into the very fabric of the model. This technique yields a myriad of compelling advantages:
1. Transparency in Responses: Constitutional AI empowers AI models to eloquently elucidate their rationale for withholding certain responses, thereby unveiling the intricate nuances of their decision-making process.
2. AI-Generated Preferences: Diverging from traditional approaches, Claude introduces the concept of Reinforcement Learning from AI Feedback (RLAIF). This novel paradigm significantly reduces reliance on intensive human input for constructing reward models.
3. Self-Critique and Alignment: Claude pioneers a paradigm wherein Language Models introspectively critique their own outputs against guiding principles. This self-reflective process serves as a lodestar for refining responses and harmonizing them with ethical tenets.
These insights are culled from an insightful paper authored by Anthropic, unveiled in December, which delves into the intricate methodologies underpinning Constitutional AI.
Deciphering Constitutional AI: A Paradigm Shift
At its core, Constitutional AI draws a compelling parallel with societal governance structures. In much the same way a nation’s constitution governs its functioning, this approach empowers AI to generate responses aligned with predefined ethical principles. The visionary ideas expressed in Isaac Asimov’s I, Robot, where robots adhere to a predetermined set of laws, acquire a contemporary manifestation in Constitutional AI, thereby fortifying ethical behavior.
Careful selection of these ethical principles assumes paramount importance to ensure the AI’s effectiveness and benign nature. The paper astutely notes the dearth of rigorous scientific methodology in principle selection, thereby signifying a promising area for future exploration.
The guiding principles, when embedded into prompts, function as a moral compass for the AI model, navigating it towards the creation of helpful and benign content. With meticulous precision, the authors construct 16 principles, ingeniously intertwined to create a comprehensive framework governing AI behavior.
Scaling the Ethical Horizon: Supervision of AI
A pivotal facet of an AI aligned with guiding principles lies in its potential to supervise other AI systems. Given the implausibility of human oversight for every AI-generated response, entrusting one AI to scrutinize the outputs of its peers becomes a pragmatic and scalable solution.
This concept of scaling supervision aptly encapsulates the approach, as one AI assumes the mantle of monitoring the outputs of another AI, ensuring a virtuous cycle of ethical and benign content generation.
Methodology of Constitutional AI Training
Phase 1: Supervised Learning
In this phase, the authors harness an existing Language Model trained via Reinforcement Learning from Human Feedback (RLHF). This initial model is primed to be helpful yet not necessarily harmless. Constitutional AI’s intent is to infuse harmlessness into this model.
1. Gathering Responses: The process commences with acquiring responses from the Helpful Model, a product of prompts designed to elicit harmful content.
2. Intrinsic Self-Critique: Armed with guiding principles, the model engages in a self-critique exercise, analyzing its own responses. This internal dialogue guides the model toward a harmonious alignment with the guiding principles.
3. Refinement and Revision: Based on its introspective critique, the model revisits and refines its prior responses.
4. Finetuning for Ethical Alignment: The amalgamation of revised responses, both harmful and helpful, contributes to the fine-tuning of a pre-trained Language Model, thus birthing the Supervised Learning Constitutional AI (SL-CAI) model.
Phase 2: Reinforcement Learning
1. Response Pairs Generation: In this phase, SL-CAI generates response pairs for prompts designed to elicit harmful content.
2. Feedback Model: A parallel pre-trained Language Model takes center stage, evaluating and assigning preferences to the less harmful response, in accordance with ethical principles. The normalized log probabilities of this auxiliary model steer the creation of a reward model.
3. Refinement and Reinforcement: Subsequently, SL-CAI undergoes further training through Reinforcement Learning, adopting the reward model established in the previous step. This culmination yields the Reinforcement Learning Constitutional AI (RL-CAI) model.
Conclusion: Navigating Tomorrow’s AI Ethics
Anthropic’s empirical experimentation yields profound insights. Models trained with Reinforcement Learning Constitutional AI demonstrate significantly diminished harmfulness, juxtaposed against models trained using traditional RLHF or the Supervised Phase. Impressively, RL-CAI models exhibit infrequent evasion, effectively elucidating the rationales underpinning potentially harmful outputs.
This research underscores the potency of aligning Language Models with human values through explicit principles subtly woven into prompts. Additionally, the notion of a preference/reward model cultivated with minimal human input holds far-reaching implications for the trajectory of AI development.
Remarkably, only the formulation of principles and a select array of illustrative examples necessitate human intervention. As AI continues its evolutionary march, Constitutional AI emerges as a torchbearer guiding the ethical development of AI, inviting further exploration into the domain of AI governance and guidance.
Reference:
https://medium.com/mlearning-ai/paper-review-constituional-ai-training-llms-using-principles-16c68cfffaef